1
This is v0.9.10 of PyECLib. This library provides a simple Python interface for
2
implementing erasure codes and is known to work with Python v2.6, 2.7 and 3.x.
4
To obtain the best possible performance, the library utilizes liberasurecode,
5
which is a C based erasure code library. Please let us know if you have any
6
other issues building or installing (email: kmgreen2@gmail.com or
9
This library makes use of Jesasure for Reed-Solomon as implemented by the
10
liberasurecode library and provides its' own flat XOR-based erasure code
11
encoder and decoder. Currently, it implements a specific class of HD
12
Combination Codes (see "Flat XOR-based erasure codes in storage systems:
13
Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010). These
14
codes are well-suited to archival use-cases, have a simple construction and
15
require a minimum number of participating disks during single-disk
16
reconstruction (think XOR-based LRC code).
18
Examples of using this library are provided in "tools" directory:
20
Command-line encoder::
22
tools/pyeclib_encode.py
24
Command-line decoder::
26
tools/pyeclib_decode.py
28
Utility to determine what is needed to reconstruct missing fragments::
30
tools/pyeclib_fragments_needed.py
33
PyEClib initialization::
35
ec_driver = ECDriver("pyeclib.core.ECPyECLibDriver",
36
k=<num_encoded_data_fragments>,
37
m=<num_encoded_parity_fragments>,
40
Supported ``ec_type`` values:
42
* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding
43
* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes
45
A configuration utility is provided to help compare available EC schemes in
46
terms of performance and redundancy:: tools/pyeclib_conf_tool.py
49
The Python API supports the following functions:
53
Encode N bytes of a data object into k (data) + m (parity) fragments::
55
def encode(self, data_bytes)
57
input: data_bytes - input data object (bytes)
58
returns: list of fragments (bytes)
63
Decode between k and k+m fragments into original object::
65
def decode(self, fragment_payloads)
67
input: list of fragment_payloads (bytes)
68
returns: decoded object (bytes)
71
*Note*: ``bytes`` is a synonym to ``str`` in Python 2.6, 2.7.
72
In Python 3.x, ``bytes`` and ``str`` types are non-interchangeable and care
73
needs to be taken when handling input to and output from the ``encode()`` and
74
``decode()`` routines.
79
Reconstruct "missing_fragment_indexes" using "available_fragment_payloads"::
81
def reconstruct(self, available_fragment_payloads, missing_fragment_indexes)
84
- Fragments needed for EC Reconstruct
86
Return the indexes of fragments needed to reconstruct "missing_fragment_indexes"::
88
def fragments_needed(self, missing_fragment_indexes)
93
Return an opaque buffer known by the underlying library::
95
def get_metadata(self, fragment)
98
- Verify EC Stripe Consistency
100
Use opaque buffers from get_metadata() to verify a the consistency of a stripe::
102
def verify_stripe_metadata(self, fragment_metadata_list)
105
- Get EC Segment Info
107
Return a dict with the keys - segment_size, last_segment_size, fragment_size, last_fragment_size and num_segments::
109
def get_segment_info(self, data_len, segment_size)
114
Standard stuff to install::
116
``Python 2.6``, ``2.7`` or ``3.x`` (including development packages), ``argparse`` and ``liberasurecode``.
119
As mentioned above, PyECLib depends on the installation of the liberasurecde library (liberasurecode
120
can be found at http://bitbucket.org/elambert/liberasurecode).
125
$ sudo python setup.py install
127
Run test suite included::
129
$ sudo python setup.py test && (cd test; ./ec_pyeclib_file_test.sh)
131
If all of this works, then you should be good to go. If not, send us an email!
133
If the test suite fails because it cannot find any of the shared libraries,
134
then you probably need to add /usr/local/lib to the path searched when loading
135
libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to::
added
removed
README
