序列化－pickle

序列化－pickle，json

1、序列化

将对象的状态信息转换为可以存储或者可以传输的形式过程

内存中有一些结构数据，希望保存下来冲用，或者发送给别人使用

很多游戏允许你在退出的时候保存进度，然后你再次启动的时候回到退出的地方。

2、常用的一些序列化

pickle，cPickle

JSON

Shelve

YAML

3、序列化对象到磁盘，所有的python支持的类型都可以用pickle做序列化

序列化到磁盘：pickle.dump(obj, file)，从磁盘反序列化：pickle.load(file)

序列化到内存：pickle.dumps(obj)，从内存反序列化：pickle.loads(str)

＃pickle序列化到磁盘,pickle.dump(参数1，参数2)，fd是问价句炳In [1]: import pickleIn [2]: entry = {'a':11, 'b':22}In [3]: with open('/tmp/1.pickle','wb') as fd:  #w表示写方式，b表示二进制   ...:     pickle.dump(entry, fd)[root@133 ~]# file /tmp/1.pickle /tmp/1.pickle: ASCII text   [root@133 ~]cat /tmp/1.pickle (dp0S'a'p1I11sS'b'p2I22   #pickle.load(fd)从硬盘反序列化，In [7]: with open('/tmp/1.pickle') as fd:   ...:     a = pickle.load(fd)      ...:     In [8]: aOut[8]: {'a': 11, 'b': 22}#pickle序列化数据到内存中，In [14]: entry = {'a':11, 'b':22}In [15]: b = pickle.dumps(entry)In [16]: bOut[16]: "(dp0\nS'a'\np1\nI11\nsS'b'\np2\nI22\ns."#pickle.loads(fd)从内存反序列化In [17]: entry1=pickle.loads(b)In [19]: entry1Out[19]: {'a': 11, 'b': 22}

pickle简单应用

[root@133 tmp]# cd /opt/python/django/[root@133 django]# cp -pr simplecmdb simplecmdbbak[root@133 simplecmdb]# cd  /opt/python/django/simplecmdbbak/simplecmdb[root@133 simplecmdb]# vim settings.py#注释掉默认的sqlite3数据库#DATABASES = {#    'default': {#        'ENGINE': 'django.db.backends.sqlite3',#        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),#    }#}#指定使用mysql数据库DATABASES = {    'default':{        'ENGINE':'django.db.backends.mysql',        'NAME':'cmdb',        'USER':'root',        'PASSWORD':'Amos!@#$',        'HOST':'127.0.0.1',        'PORT':'3306',    }}

登录mysql，创建数据库cmdb，在数据库cmdb中创建数据表

[root@133 simplecmdbbak]# cd /opt/python/django/simplecmdbbak[root@133 simplecmdbbak]# rm -rf db.sqlite3 #删除sqlite3的数据库[root@133 simplecmdbbak]# cd hostinfo/[root@133 hostinfo]# vim models.py #models.py已经定义了数据表，只要同步即可在mysql中建表from django.db import models# Create your models here.class Host(models.Model):    hostname = models.CharField(max_length = 50)    ip = models.IPAddressField()    vendor = models.CharField(max_length = 50)    product = models.CharField(max_length = 50)    sn = models.CharField(max_length = 50)    cpu_model = models.CharField(max_length = 50)    cpu_num = models.IntegerField(max_length = 50)    memory = models.CharField(max_length = 50)    osver = models.CharField(max_length = 50)    #查看数据库配置文件有没有错误[root@133 simplecmdbbak]# python manage.py validate0 errors foundmysql> create database cmdb;Query OK, 1 row affected (0.00 sec)[root@133 simplecmdbbak]# python manage.py syncdbCreating tables ...Creating table django_admin_logCreating table auth_permissionCreating table auth_group_permissionsCreating table auth_groupCreating table auth_user_groupsCreating table auth_user_user_permissionsCreating table auth_userCreating table django_content_typeCreating table django_sessionCreating table hostinfo_hostCreating table hostinfo_hostgroup_membersCreating table hostinfo_hostgroupYou just installed Django's auth system, which means you don't have any superusers defined.Would you like to create one now? (yes/no): yesUsername (leave blank to use 'root'): rootEmail address: 1350368559@qq.comPassword: Password (again): Superuser created successfully.Installing custom SQL ...Installing indexes ...Installed 0 object(s) from 0 fixture(s)[root@133 simplecmdbbak]# mysql -uroot -pEnter password: mysql> use cmdb;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql> show tables;+----------------------------+| Tables_in_cmdb             |+----------------------------+| auth_group                 || auth_group_permissions     || auth_permission            || auth_user                  || auth_user_groups           || auth_user_user_permissions || django_admin_log           || django_content_type        || django_session             || hostinfo_host              || hostinfo_hostgroup         || hostinfo_hostgroup_members |+----------------------------+12 rows in set (0.00 sec)mysql> exitBye

修改系统收集信息

[root@133 django]# vim /opt/python/django/sysinformation-pickle.pyimport pickle#   d = urllib.urlencode(dic)    d = pickle.dumps(dic) #使用pickle将收集到的信息保存到内存中    #修改视图文件，得到属性文件[root@133 django]# vim /opt/python/django/simplecmdbbak/hostinfo/views.py   print req.body

启动django服务器

[root@133 simplecmdbbak]# python manage.py  runserver 112.65.140.133:8080

执行系统收集脚本：python sysinformation-pickle.py，报错500，排查过程如下：

[root@133 django]# python sysinformation-pickle.py #由于后边的变量值没有收到具体的信息报错Traceback (most recent call last):  File "sysinformation-pickle.py", line 118, in     req = urllib2.urlopen('http://112.65.140.133:8080/hostinfo/collect/',d)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 126, in urlopen    return _opener.open(url, data, timeout)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 406, in open    response = meth(req, response)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 519, in http_response    'http', request, response, code, msg, hdrs)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 444, in error    return self._call_chain(*args)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 378, in _call_chain    result = func(*args)  File "/opt/amos/python2.7/lib/python2.7/urllib2.py", line 527, in http_error_default    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR[root@133 simplecmdbbak]# python manage.py  runserver 112.65.140.133:8080Validating models...0 errors foundJanuary 15, 2017 - 18:00:56Django version 1.6.5, using settings 'simplecmdb.settings'Starting development server at http://112.65.140.133:8080/Quit the server with CONTROL-C.#以下是pickle发送到服务器的信息，格式是pickle格式，由simplecmdbbak/hostinfo/views.py处理得到的(dp0S'osver'p1S'CentOS release 6.7 (Final)'p2sS'product'p3S'PowerEdge R710'p4sS'vendor'p5S'Dell Inc.'p6sS'sn'p7S'4HBDT2X'p8sS'memory'p9S'19976M'p10sS'cpu_num'p11I8sS'ip'p12(S'br1'p13S'112.65.140.133'p14S'A4:BA:DB:20:93:23'p15tp16sS'hostname'p17S'133'p18sS'cpu_model'p19S'Intel(R) 2.00GHz'p20s.NoneNoneNoneNoneNoneNoneNoneNoneNone[15/Jan/2017 18:01:04] "POST /hostinfo/collect/ HTTP/1.1" 500 118260#测试，如果将：sysinformation-pickle.py修改为：    d = urllib.urlencode(dic)#    d = pickle.dumps(dic)结果就是：osver=CentOS+release+6.7+%28Final%29&product=PowerEdge+R710&vendor=Dell+Inc.&sn=4HBDT2X&memory=19976M&cpu_num=8&ip=%28%27br1%27%2C+%27112.65.140.133%27%2C+%27A4%3ABA%3ADB%3A20%3A93%3A23%27%29&hostname=133&cpu_model=Intel%28R%29+2.00GHz133('br1', '112.65.140.133', 'A4:BA:DB:20:93:23')CentOS release 6.7 (Final)Dell Inc.PowerEdge R710Intel(R) 2.00GHz819976M4HBDT2X数据已经发送到djago这里，如何将数据发序列化出来成为字典，即可正常使用。[root@133 django]# vim /opt/python/django/simplecmdbbak/hostinfo/views.pyfrom django.shortcuts import renderfrom django.http import HttpResponsefrom hostinfo.models import Hostimport pickle# Create your views here.def collect(req):    if req.POST:        print pickle.loads(req.body) [root@133 django]# python sysinformation-pickle.py#结果显示已经去得了pickle loads出来的字典了{'product': 'PowerEdge R710', 'vendor': 'Dell Inc.', 'cpu_num': 8, 'ip': ('br1', '112.65.140.133', 'A4:BA:DB:20:93:23'), 'hostname': '133', 'cpu_model': 'Intel(R) 2.00GHz', 'osver': 'CentOS release 6.7 (Final)', 'sn': '4HBDT2X', 'memory': '19976M'}[root@133 django]# vim /opt/python/django/simplecmdbbak/hostinfo/views.py import pickledef collect(req):    if req.POST:        obj =  pickle.loads(req.body)        hostname = obj['hostname']        print hostname        ip = obj['ip']        print ip        osver = obj['osver']        print osver        vendor = obj['vendor']        print vendor        product = obj['product']        print product         cpu_model = obj['cpu_model']        print cpu_model        cpu_num = obj['cpu_num']        print cpu_num        memory = obj['memory']        print memory        sn = obj['sn']        print sn        :10,27s/req.POST.get/obj/:10,27s/(/[/:10,27s/)/]/[root@133 simplecmdb]# vim hostinfo/models.pyfrom django.db import models# Create your models here.class Host(models.Model):    hostname = models.CharField(max_length = 200)    ip = models.IPAddressField()    vendor = models.CharField(max_length = 200)    product = models.CharField(max_length = 200)    sn = models.CharField(max_length = 200)    cpu_model = models.CharField(max_length = 200)    cpu_num = models.IntegerField(max_length = 200)    memory = models.CharField(max_length = 200)    osver = models.CharField(max_length = 200)    def __unicode__(self):        return self.hostname数据库字段的长度改为200，然后数据库cmdb删除，重新同步的，结果还是一样。500搞不清为什么会这样？另外一个问题：    #d = urllib.urlencode(dic)    d = pickle.dumps(dic)    req = urllib2.urlopen('http://112.65.140.133:8080/hostinfo/collect/',d)    print req.read()    #d = urllib.urlencode(dic)  这里的d难道不用urlencode吗？不转换成urlencode格式，怎么能用urllib2.urlopen发送处理呢？host是实例，不是字典，所以不能这样host['hostname']，只能host.hostname这样来调用属性。可以在实例化（ host = Host()）之前来打印hostname、ip等信息，来进行拍错。这里的d是序列化后的对象，这里可以使用pickle序列化，跟使用json一样。视频里有三种方式1. 使用urlencode2. 使用pickle3. 使用json我的问题找到了，原因是：ip = models.IPAddressField() 无法保存元祖。def collect(req):    if req.POST:        print pickle.loads(req.body)        obj =  pickle.loads(req.body)        hostname = obj['hostname']        ip = obj['ip']        osver = obj['osver']        vendor = obj['vendor']        product = obj['product']        cpu_model = obj['cpu_model']        cpu_num = obj['cpu_num']        memory = obj['memory']        sn = obj['sn']        print sn        print 'pickle load ok'        host = Host()        host.hostname = hostname        host.ip = ip        host.osver = osver        host.vendor = vendor        host.product = product        host.cpu_model = cpu_model        host.cpu_num = cpu_num        host.memory = memory        host.sn = sn        print host.sn        host.save()        print "host save ok"  ＃没有打印        print host.sn               ＃没有打印        return HttpResponse('OK')    else:        return HttpResponse('no data')结果是：[16/Jan/2017 11:34:20] "POST /hostinfo/collect/ HTTP/1.1" 500 72226{'product': 'PowerEdge R710', 'vendor': 'Dell Inc.', 'cpu_num': 8, 'ip': ('br1', '112.65.140.133', 'A4:BA:DB:20:93:23'), 'hostname': '133', 'cpu_model': 'Intel(R) 2.00GHz', 'osver': 'CentOS release 6.7 (Final)', 'sn': '4HBDT2X', 'memory': '19976M'}4HBDT2Xpickle load ok4HBDT2X很明显是host.save()没有成功，但是我却不知道为什么没有成功？最后发现是：我打印出来的IP被我设置为三个参数：def parseIfconfig(parsed_data):    dic = {}    tuple_addr= ('lo','vir','vnet','em3','em4')    parsed_data = [i for i in parsed_data if i and not i.startswith(tuple_addr)]    for lines in parsed_data:        line_list = lines.split('\n')        devname = line_list[0].split()[0]        macaddr = line_list[0].split()[-1]        ipaddr = line_list[1].split()[1].split(':')[1]        break    dic['ip'] = devname,ipaddr,macaddr    return dicip如果是：'ip': '112.65.140.133' 就可以保存了而实际结果是：'ip': ('br1', '112.65.140.133', 'A4:BA:DB:20:93:23')所以无法保存到数据库中，所以会报错改为：dic['ip'] = ipaddr就ok了