In this Python Pickle tutorial, we will study what a Pickle is in Python and how Python Serialization deals with the \u2018pickle\u2019 module of Python for serialization.<\/p>\n
At last, we will discuss some Python Pickle Examples. So, let’s start the Python Pickle Tutorial.<\/p>\n
In Python, when we want to serialize and deserialize a Python object, we use functions and methods from the Python pickle module. Pickling turns objects into byte streams via pickle.dumps(); load them back with pickle.loads().<\/p>\n
We also call this \u2018serialization\u2019, \u2018marshalling\u2019, or \u2018flattening\u2019. Unpickling is its inverse, ie., converting a byte stream from a binary file or bytes-like object into an object.<\/p>\n
Advantages of Serialization in Python:<\/strong><\/p>\n \u2018marshal\u2019 is a more primitive module for serialization in Python, and its purpose is to support .pyc files. However, we prefer Python pickle. The two differ in the following ways:<\/p>\n JSON is a standard library module for serialization and deserialization with Python.<\/p>\n Python pickle uses a Python-specific data format. So, external standards like JSON or XDR impose no restrictions. But this makes it impossible for non-Python programs to reconstruct pickled Python objects.<\/p>\n As we said above, Python pickle uses a data format with a relatively compact binary representation. We can efficiently compress it.<\/p>\n Complementary to Python pickle is the module \u2018pickletools\u2019 for analyzing data streams that it generates.<\/p>\n We have five different protocols for pickling:<\/strong><\/p>\n 1. Protocol version 0:<\/strong>\u00a0Original, human-readable protocol; backwards-compatible with earlier versions of Python.<\/p>\n 2. Protocol version 1:<\/strong>\u00a0Old, binary format; compatible with earlier versions of Python.<\/p>\n 3. Protocol version 2:<\/strong>\u00a0Added in Python 2.3; provides more efficient pickling of new-style classes.<\/p>\n 4. Protocol version 3:<\/strong>\u00a0Introduced in Python 3.0; default; supports bytes objects; cannot be unpickled by Python 2.x. It is recommended when we need compatibility with other Python 3 versions.<\/p>\n 5. Protocol version 4:<\/strong>\u00a0Introduced in Python 3.4; supports very large objects, more kinds of objects, and certain data format optimizations.<\/p>\n To serialize and deserialize, we use functions dumps() and loads(), respectively. Alternatively, we can create our own Pickler and Unpickler objects for more control over this.<\/p>\n Python pickle has two constants:<\/p>\n This is an integer, and it holds the highest protocol version that is available. We can pass this as a protocol<\/i> value to dump() and dumps(), and to the Pickler constructor.<\/p>\n Also an integer, this holds the default protocol version for pickling. The default is currently Protocol 3.<\/p>\n It also has the following functions:<\/p>\n a. dump(obj, file, protocol=None, *, fix_imports=True)<\/strong><\/p>\n This writes a pickled representation of object obj<\/i> to file<\/i>, an open file object. Consider this equivalent to Pickler(file, protocol).dump(obj)<\/p>\n When we checked the file abcde.txt, we found this: So, it can be a file you opened in binary mode, an io.BytesIO instance, or a custom object meeting this interface. Protocol<\/i> lets us choose which protocol to use.<\/p>\n When fix_imports<\/i> is true, and we use a protocol less than 3, pickle maps new Python 3 names to old module names in Python 2. This lets Python 2 read the pickle data stream.<\/p>\n b. dumps(obj, protocol=None, *, fix_imports=True)<\/strong><\/p>\n This returns the pickled representation of obj<\/i> as a bytes object. This does not write it to a file.<\/p>\n Output<\/strong><\/p>\n b’\\x80\\x03K\\x07.’<\/p>\n<\/div>\n c. load(file, *, fix_imports=True, encoding=”ASCII”, errors=”strict”)<\/strong><\/p>\n load() takes in a file<\/i>, an open file object, reads a pickled representation from it, and returns the reconstructed object hierarchy. Consider this equivalent to Unpickler(file).load()<\/p>\n file<\/i> can be a file object opened in the binary reading mode, an io.BytesIO object, or an object that meets its interface. This is because it must have two methods- \u00a0read(), which takes one integer argument, and readline(), which takes no arguments.<\/p>\n Both of these methods must return bytes.<\/p>\n fix_imports, encoding, and errors help control compatibility support for pickle streams by Python 2. When fix_imports is true, pickle maps old Python 2 names to new Python 3 names.<\/p>\n The other two guide pickle with decoding 8-bit string instances pickled by Python 2. The default encoding is \u2018ASCII\u2019, and the default value for errors is \u2018strict\u2019.<\/p>\n To read such 8-bit string instances as byte objects, we can set the encoding to \u2018bytes\u2019.<\/p>\n Let\u2019s try doing this.<\/p>\n Output<\/strong><\/p>\n Traceback (most recent call last):File “<pyshell#63>”, line 1, in <module><\/p>\n pickle.load(f)<\/p>\n EOFError: Ran out of input<\/p>\n<\/div>\n Uh-oh. Let\u2019s get to the beginning of the file.<\/p>\n Output<\/strong><\/p>\n 0<\/p>\n<\/div>\n Now, we can successfully load it.<\/p>\n Output<\/strong><\/p>\n d. loads(bytes_object, *, fix_imports=True, encoding=”ASCII”, errors=”strict”)<\/strong><\/p>\n This function takes in a bytes object, reads a pickled object hierarchy, and returns the reconstructed object hierarchy.<\/p>\n fix_imports, encoding, and errors help control compatibility support for pickle streams that Python 2 generates.<\/p>\n When it is true, pickle maps old Python 2 names to new Python 3 names. encoding guides pickle with decoding 8-bit string instances pickled by Python 2.<\/p>\n The default for encoding is \u2018ASCII\u2019, and that for errors is \u2018strict\u2019. To read such 8-byte instances as byte objects, we can set the encoding to \u2018bytes\u2019.<\/p>\n Output<\/strong><\/p>\n The Python pickle module also defines three kinds of exceptions:<\/p>\n This is the common parent class for all other pickling exceptions. It, in turn, inherits from Exception.<\/p>\n When the Pickler encounters an unpicklable object, it raises a PicklingError. This class inherits from PickleError.<\/p>\n When Python pickle cannot unpickle an object due to data corruption or a security violation, it raises an UnpicklingError. This inherits from PickleError.<\/p>\n Some other exceptions we observe when pickling, as we did above, include: Python pickle imports two classes- Pickler and Unpickler:<\/p>\n Pickler takes in a binary file and writes a pickle data stream. file<\/i> must have a write() method accepting a single byte argument. This can be a file object for a file opened for writing in binary mode, an io.BytesIO instance, or a custom object meeting this interface. protocol<\/i> is an integer, and informs the pickler about which protocol to use (0 to HIGHEST_PROTOCOL). Otherwise, it uses DEFAULT_PROTOCOL. On providing a negative number, it selects HIGHEST_PROTOCOL.<\/p>\n When fix_imports<\/i> is true, and the protocol version is less than 3, pickle maps new Python 3 names to old Python 2 names. This makes the Python data stream readable by Python 2.<\/p>\n Python Pickler has the following members:<\/p>\n a. dump(obj)<\/strong><\/p>\n This takes in obj<\/i> and writes a pickled representation of it to the open file object specified in the constructor of Pickler.<\/p>\n b. persistent_id(obj)<\/strong><\/p>\n By default, it does nothing. It only exists to let subclasses override it. If it returns none, pickle pickles obj<\/i> as usual.<\/p>\n Otherwise, Pickler emits the returned value as a persistent ID for obj<\/i>. Unpickler.persistent_load() defines this context.<\/p>\n c. dispatch_table<\/strong><\/p>\n For an object of Pickler, a dispatch table is a registry holding reduction functions that we can declare with copyreg.pickle(). This mapping has classes as its keys, and reduction functions as its values.<\/p>\n A reduction function takes one argument of the class, and conforms to this interface as a __reduce__() method.<\/p>\n But pickler objects don\u2019t have dispatch_tables by default. Instead, it makes use of the global dispatch table that the copyreg module manages.<\/p>\n To customize pickling for an object of a specific object of Pickler, we can set dispatch_table to a dict-like object.<\/p>\n Or, if one of the subclasses of Pickler has dispatch_table, then this serves as the default dispatch table for instances of that class.<\/p>\n d. fast<\/strong><\/p>\n Although this is deprecated(no longer advised), it enables fast mode when set to true. This mode disables memo, thereby speeding pickling as it doesn\u2019t generate extra PUT opcodes.<\/p>\n However, do not use it with self-referential objects, as it can set Pickler off into infinite recursion.<\/p>\n For more compact pickles, we can use pickletools.optimize().<\/p>\n The Unpickler takes in a binary file and reads a pickle data stream. file<\/i> must have the methods read()- that takes an integer argument, and readline()- that needs no arguments. Both of these methods must return bytes.<\/p>\n This can be a file object opened for reading in binary mode, an io.BytesIO object, or a custom object meeting this interface.<\/p>\n pickle automatically detects the version of protocol used; we don\u2019t need an argument for that. fix_imports, encoding, <\/i>and errors <\/i>help control compatibility support for pickle streams generated by Python 2.<\/p>\n When fix_imports<\/i> is true, pickle maps old Python 2 names to new Python 3 names. encoding <\/i>and errors <\/i>guide pickle with decoding 8-bit string instances pickled by Python 2.<\/p>\n The default for encoding <\/i>is \u2018ASCII\u2019, and that for errors<\/i> is \u2018strict\u2019. When we want to read such 8-bit string instances as bytes objects, we can set the encoding<\/i> to \u2018bytes\u2019.<\/p>\n Unpickler has the following members:<\/p>\n 1. load()<\/strong><\/p>\n This takes in an open file object, reads a pickled object representation, and returns the reconstructed object hierarchy.<\/p>\n 2. persistent_load(pid)<\/strong><\/p>\n By default, this raises an UnpicklingError. When we define it, however, it must return the object about the persistent ID pid<\/i>. If we pass an invalid persistent ID, it raises an UnpicklingError.<\/p>\n 3. find_class(module,name)<\/strong><\/p>\n If necessary, it imports the module<\/i>\u00a0and returns the object name<\/i> from it. Here, module<\/i> and name<\/i> are str objects. We can also use find_class() to find functions.<\/p>\n A subclass can override this to control what kind of objects it can take and how we can load them. This alleviates security risks.<\/p>\n Any doubt yet in Python Pickle? Please Comment.<\/p>\n We can pickle the following types:<\/p>\n When we try to pickle an unpicklable object, pickle raises the PicklingError exception. In this process, it is possible that an unspecified number of bytes have already been written to the file.<\/p>\n In trying to pickle a highly recursive data structure, we may exceed the maximum recursion depth. Such a case raises a RecursionError. However, we can raise that limit with sys.setrecursionlimit().<\/p>\n Let\u2019s take a quick look.<\/p>\n We pickle functions by their \u2018fully-qualified\u2019 name references, not by their values. This way, we only pickle the function name and the module it resides in.<\/p>\n We do not pickle the function\u2019s code or attributes. So, we should be able to import the defining module in the unpickling environment.<\/p>\n This module must hold the named object. Otherwise, it raises an exception.<\/p>\n We pickle classes by name reference. This way, the same restrictions apply in the unpickling environment. We do not pickle the class\u2019s code or data. We only pickle instance data.<\/p>\n Such restrictions mandate that we define picklable classes and functions in a module\u2019s top-level.<\/p>\n This was all on Python Pickle and Python Serialization. Hope you now understand Python Serialization.<\/p>\n 1. What is a pickle in Python?<\/p>\n 2. What is Python pickle used for?<\/p>\n 3. How do you pickle data in Python?<\/p>\n 4. What is serialization in Python?<\/p>\n 5. Is Pickle built in Python?<\/p>\n Simply speaking, Python serialization is the act of converting a Python object into a byte stream. In Python, we use the module \u2018pickle\u2019, which has a binary serializable format.<\/p>\n We can also serialize classes and functions. We have also studied in detail Python Pickle and its comparison with other modules.<\/p>\n","protected":false},"excerpt":{"rendered":" In this Python Pickle tutorial, we will study what a Pickle is in Python and how Python Serialization deals with the \u2018pickle\u2019 module of Python for serialization. At last, we will discuss some Python...<\/p>\n","protected":false},"author":5,"featured_media":31031,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[46],"tags":[3975,4088,4542,4712,8356,9000,9479,9485,9486,9487,9489,10758,10831,10832,15162],"class_list":["post-11819","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-dispatch_table","tag-dumpobj","tag-fast","tag-find_classmodule","tag-load","tag-name","tag-persistent_loadpid","tag-pickle-dump","tag-pickle-module-interface","tag-pickleerror","tag-picklingerror","tag-python-pickle","tag-python-serialization","tag-python-serialize","tag-unpicklingerror"],"yoast_head":"\n\n
Comparing Python Pickle to Other Python Modules<\/h3>\n
1. Comparing Python pickle to marshal<\/h4>\n
\n
2. Comparing Python pickle to JSON<\/h4>\n
\n
Python Pickle Supports Data Stream Format<\/h3>\n
Pickle Module Interface in Python<\/h3>\n
1. HIGHEST_PROTOCOL in Python<\/h4>\n
2. DEFAULT_PROTOCOL in Python<\/h4>\n
>>> x=7\r\n>>> import os\r\n>>> os.chdir('C:\\\\Users\\\\lifei\\\\Desktop')\r\n>>> import pickle\r\n>>> f=open('abcde.txt','r+b') \/\/opened it in binary mode to pickle\r\n>>> pickle.dump(x,f)<\/pre>\n
\n\u20acK. \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u20acK.\u20acK.
\n<\/i>file<\/i>, here, must have a write() method accepting a single byte argument.<\/p>\n>>> pickle.dumps(x)<\/pre>\n
>>> pickle.load(f)<\/pre>\n
>>> f.seek(0)<\/pre>\n
>>> pickle.load(f)<\/pre>\n
>>> f.seek(0)<\/pre>\n
>>> pickle.loads(f.read())<\/pre>\n
Python Pickle Exceptions<\/h3>\n
1. PickleError in Python<\/h4>\n
2. PicklingError in Python<\/h4>\n
3. UnpicklingError in Python<\/h4>\n
\nEOFError
\nTypeError
\nValueError
\nAttributeError
\nImportError
\nIndexError<\/p>\nImported Classes in Python Pickle<\/h3>\n
1. Pickler(file, protocol=None, *, fix_imports=True) in Python<\/h4>\n
\n<\/i><\/p>\n
\n<\/i><\/p>\n2. Unpickler(file, *, fix_imports=True, encoding=”ASCII”, errors=”strict”) in Python<\/h4>\n
\n<\/i><\/p>\n
\n<\/i><\/p>\nWhat Can We Pickle and Unpickle in Python?<\/h3>\n
\n
Python Interview Questions on Pickle & Serialization<\/h3>\n
Conclusion<\/h3>\n