1
.. PyZMQ serialization doc, by Min Ragan-Kelley, 2011
5
Serializing messages with PyZMQ
6
===============================
8
When sending messages over a network, you often need to marshall your data into bytes.
14
PyZMQ is primarily bindings for libzmq, but we do provide three builtin serialization
15
methods for convenience, to help Python developers learn libzmq. Python has two primary
16
packages for serializing objects: :py:mod:`json` and :py:mod:`pickle`, so we provide
17
simple convenience methods for sending and receiving objects serialized with these
18
modules. A socket has the methods :meth:`~.Socket.send_json` and
19
:meth:`~.Socket.send_pyobj`, which correspond to sending an object over the wire after
20
serializing with json and pickle respectively, and any object sent via those
21
methods can be reconstructed with the :meth:`~.Socket.recv_json` and
22
:meth:`~.Socket.recv_pyobj` methods.
25
These methods designed for convenience, not for performance, so developers who do want
26
to emphasize performance should use their own serialized send/recv methods.
28
Using your own serialization
29
----------------------------
31
In general, you will want to provide your own serialization that is optimized for your
32
application or library availability. This may include using your own preferred
33
serialization ([msgpack]_, [protobuf]_), or adding compression via [zlib]_ in the standard
34
library, or the super fast [blosc]_ library.
36
There are two simple models for implementing your own serialization: write a function
37
that takes the socket as an argument, or subclass Socket for use in your own apps.
39
For instance, pickles can often be reduced substantially in size by compressing the data.
40
The following will send *compressed* pickles over the wire:
42
.. sourcecode:: python
44
import zlib, cPickle as pickle
46
def send_zipped_pickle(socket, obj, flags=0, protocol=-1):
47
"""pickle an object, and zip the pickle before sending it"""
48
p = pickle.dumps(obj, protocol)
50
return socket.send(z, flags=flags)
52
def recv_zipped_pickle(socket, flags=0, protocol=-1):
53
"""inverse of send_zipped_pickle"""
54
z = socket.recv(flags)
55
p = zlib.uncompress(z)
56
return pickle.loads(p)
58
A common data structure in Python is the numpy array. PyZMQ supports sending
59
numpy arrays without copying any data, since they provide the Python buffer interface.
60
However just the buffer is not enough information to reconstruct the array on the
61
receiving side. Here is an example of a send/recv that allow non-copying
62
sends/recvs of numpy arrays including the dtype/shape data necessary for reconstructing
65
.. sourcecode:: python
69
def send_array(socket, A, flags=0, copy=True, track=False):
70
"""send a numpy array with metadata"""
75
socket.send_json(md, flags|zmq.SNDMORE)
76
return socket.send(A, flags, copy=copy, track=track)
78
def recv_array(socket, flags=0, copy=True, track=False):
79
"""recv a numpy array"""
80
md = socket.recv_json(flags=flags)
81
msg = socket.recv(flags=flags, copy=copy, track=track)
83
A = numpy.frombuffer(buf, dtype=md['dtype'])
84
return A.reshape(md['shape'])
87
.. [msgpack] Message Pack serialization library http://msgpack.org
88
.. [protobuf] Google Protocol Buffers http://code.google.com/p/protobuf
89
.. [zlib] Python stdlib module for zip compression: :py:mod:`zlib`
90
.. [blosc] Blosc: A blocking, shuffling and loss-less (and crazy-fast) compression library http://blosc.pytables.org/trac
added
removed
docs/source/serialization.rst
