1
<?xml version="1.0"?> <!-- -*- sgml -*- -->
2
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[
5
<!-- various strings, dates etc. common to all docs -->
6
<!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents;
9
<book lang="en" id="userman" xreflabel="bzip2 Manual">
12
<title>bzip2 and libbzip2, version 1.0.6</title>
13
<subtitle>A program and library for data compression</subtitle>
15
<year>&bz-lifespan;</year>
16
<holder>Julian Seward</holder>
18
<releaseinfo>Version &bz-version; of &bz-date;</releaseinfo>
22
<firstname>Julian</firstname>
23
<surname>Seward</surname>
25
<orgname>&bz-url;</orgname>
32
<para>This program, <computeroutput>bzip2</computeroutput>, the
33
associated library <computeroutput>libbzip2</computeroutput>, and
34
all documentation, are copyright © &bz-lifespan; Julian Seward.
35
All rights reserved.</para>
37
<para>Redistribution and use in source and binary forms, with
38
or without modification, are permitted provided that the
39
following conditions are met:</para>
41
<itemizedlist mark='bullet'>
43
<listitem><para>Redistributions of source code must retain the
44
above copyright notice, this list of conditions and the
45
following disclaimer.</para></listitem>
47
<listitem><para>The origin of this software must not be
48
misrepresented; you must not claim that you wrote the original
49
software. If you use this software in a product, an
50
acknowledgment in the product documentation would be
51
appreciated but is not required.</para></listitem>
53
<listitem><para>Altered source versions must be plainly marked
54
as such, and must not be misrepresented as being the original
55
software.</para></listitem>
57
<listitem><para>The name of the author may not be used to
58
endorse or promote products derived from this software without
59
specific prior written permission.</para></listitem>
63
<para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY
64
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
65
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
66
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
67
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
68
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
69
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
70
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
71
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
72
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
73
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
74
THE POSSIBILITY OF SUCH DAMAGE.</para>
76
<para>PATENTS: To the best of my knowledge,
77
<computeroutput>bzip2</computeroutput> and
78
<computeroutput>libbzip2</computeroutput> do not use any patented
79
algorithms. However, I do not have the resources to carry
80
out a patent search. Therefore I cannot give any guarantee of
90
<chapter id="intro" xreflabel="Introduction">
91
<title>Introduction</title>
93
<para><computeroutput>bzip2</computeroutput> compresses files
94
using the Burrows-Wheeler block-sorting text compression
95
algorithm, and Huffman coding. Compression is generally
96
considerably better than that achieved by more conventional
97
LZ77/LZ78-based compressors, and approaches the performance of
98
the PPM family of statistical compressors.</para>
100
<para><computeroutput>bzip2</computeroutput> is built on top of
101
<computeroutput>libbzip2</computeroutput>, a flexible library for
102
handling compressed data in the
103
<computeroutput>bzip2</computeroutput> format. This manual
104
describes both how to use the program and how to work with the
105
library interface. Most of the manual is devoted to this
106
library, not the program, which is good news if your interest is
107
only in the program.</para>
109
<itemizedlist mark='bullet'>
111
<listitem><para><xref linkend="using"/> describes how to use
112
<computeroutput>bzip2</computeroutput>; this is the only part
113
you need to read if you just want to know how to operate the
114
program.</para></listitem>
116
<listitem><para><xref linkend="libprog"/> describes the
117
programming interfaces in detail, and</para></listitem>
119
<listitem><para><xref linkend="misc"/> records some
120
miscellaneous notes which I thought ought to be recorded
121
somewhere.</para></listitem>
128
<chapter id="using" xreflabel="How to use bzip2">
129
<title>How to use bzip2</title>
131
<para>This chapter contains a copy of the
132
<computeroutput>bzip2</computeroutput> man page, and nothing
135
<sect1 id="name" xreflabel="NAME">
138
<itemizedlist mark='bullet'>
140
<listitem><para><computeroutput>bzip2</computeroutput>,
141
<computeroutput>bunzip2</computeroutput> - a block-sorting file
142
compressor, v1.0.6</para></listitem>
144
<listitem><para><computeroutput>bzcat</computeroutput> -
145
decompresses files to stdout</para></listitem>
147
<listitem><para><computeroutput>bzip2recover</computeroutput> -
148
recovers data from damaged bzip2 files</para></listitem>
155
<sect1 id="synopsis" xreflabel="SYNOPSIS">
156
<title>SYNOPSIS</title>
158
<itemizedlist mark='bullet'>
160
<listitem><para><computeroutput>bzip2</computeroutput> [
161
-cdfkqstvzVL123456789 ] [ filenames ... ]</para></listitem>
163
<listitem><para><computeroutput>bunzip2</computeroutput> [
164
-fkvsVL ] [ filenames ... ]</para></listitem>
166
<listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [
167
filenames ... ]</para></listitem>
169
<listitem><para><computeroutput>bzip2recover</computeroutput>
170
filename</para></listitem>
177
<sect1 id="description" xreflabel="DESCRIPTION">
178
<title>DESCRIPTION</title>
180
<para><computeroutput>bzip2</computeroutput> compresses files
181
using the Burrows-Wheeler block sorting text compression
182
algorithm, and Huffman coding. Compression is generally
183
considerably better than that achieved by more conventional
184
LZ77/LZ78-based compressors, and approaches the performance of
185
the PPM family of statistical compressors.</para>
187
<para>The command-line options are deliberately very similar to
188
those of GNU <computeroutput>gzip</computeroutput>, but they are
189
not identical.</para>
191
<para><computeroutput>bzip2</computeroutput> expects a list of
192
file names to accompany the command-line flags. Each file is
193
replaced by a compressed version of itself, with the name
194
<computeroutput>original_name.bz2</computeroutput>. Each
195
compressed file has the same modification date, permissions, and,
196
when possible, ownership as the corresponding original, so that
197
these properties can be correctly restored at decompression time.
198
File name handling is naive in the sense that there is no
199
mechanism for preserving original file names, permissions,
200
ownerships or dates in filesystems which lack these concepts, or
201
have serious file name length restrictions, such as
204
<para><computeroutput>bzip2</computeroutput> and
205
<computeroutput>bunzip2</computeroutput> will by default not
206
overwrite existing files. If you want this to happen, specify
207
the <computeroutput>-f</computeroutput> flag.</para>
209
<para>If no file names are specified,
210
<computeroutput>bzip2</computeroutput> compresses from standard
211
input to standard output. In this case,
212
<computeroutput>bzip2</computeroutput> will decline to write
213
compressed output to a terminal, as this would be entirely
214
incomprehensible and therefore pointless.</para>
216
<para><computeroutput>bunzip2</computeroutput> (or
217
<computeroutput>bzip2 -d</computeroutput>) decompresses all
218
specified files. Files which were not created by
219
<computeroutput>bzip2</computeroutput> will be detected and
220
ignored, and a warning issued.
221
<computeroutput>bzip2</computeroutput> attempts to guess the
222
filename for the decompressed file from that of the compressed
223
file as follows:</para>
225
<itemizedlist mark='bullet'>
227
<listitem><para><computeroutput>filename.bz2 </computeroutput>
229
<computeroutput>filename</computeroutput></para></listitem>
231
<listitem><para><computeroutput>filename.bz </computeroutput>
233
<computeroutput>filename</computeroutput></para></listitem>
235
<listitem><para><computeroutput>filename.tbz2</computeroutput>
237
<computeroutput>filename.tar</computeroutput></para></listitem>
239
<listitem><para><computeroutput>filename.tbz </computeroutput>
241
<computeroutput>filename.tar</computeroutput></para></listitem>
243
<listitem><para><computeroutput>anyothername </computeroutput>
245
<computeroutput>anyothername.out</computeroutput></para></listitem>
249
<para>If the file does not end in one of the recognised endings,
250
<computeroutput>.bz2</computeroutput>,
251
<computeroutput>.bz</computeroutput>,
252
<computeroutput>.tbz2</computeroutput> or
253
<computeroutput>.tbz</computeroutput>,
254
<computeroutput>bzip2</computeroutput> complains that it cannot
255
guess the name of the original file, and uses the original name
256
with <computeroutput>.out</computeroutput> appended.</para>
258
<para>As with compression, supplying no filenames causes
259
decompression from standard input to standard output.</para>
261
<para><computeroutput>bunzip2</computeroutput> will correctly
262
decompress a file which is the concatenation of two or more
263
compressed files. The result is the concatenation of the
264
corresponding uncompressed files. Integrity testing
265
(<computeroutput>-t</computeroutput>) of concatenated compressed
266
files is also supported.</para>
268
<para>You can also compress or decompress files to the standard
269
output by giving the <computeroutput>-c</computeroutput> flag.
270
Multiple files may be compressed and decompressed like this. The
271
resulting outputs are fed sequentially to stdout. Compression of
272
multiple files in this manner generates a stream containing
273
multiple compressed file representations. Such a stream can be
274
decompressed correctly only by
275
<computeroutput>bzip2</computeroutput> version 0.9.0 or later.
276
Earlier versions of <computeroutput>bzip2</computeroutput> will
277
stop after decompressing the first file in the stream.</para>
279
<para><computeroutput>bzcat</computeroutput> (or
280
<computeroutput>bzip2 -dc</computeroutput>) decompresses all
281
specified files to the standard output.</para>
283
<para><computeroutput>bzip2</computeroutput> will read arguments
284
from the environment variables
285
<computeroutput>BZIP2</computeroutput> and
286
<computeroutput>BZIP</computeroutput>, in that order, and will
287
process them before any arguments read from the command line.
288
This gives a convenient way to supply default arguments.</para>
290
<para>Compression is always performed, even if the compressed
291
file is slightly larger than the original. Files of less than
292
about one hundred bytes tend to get larger, since the compression
293
mechanism has a constant overhead in the region of 50 bytes.
294
Random data (including the output of most file compressors) is
295
coded at about 8.05 bits per byte, giving an expansion of around
298
<para>As a self-check for your protection,
299
<computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make
300
sure that the decompressed version of a file is identical to the
301
original. This guards against corruption of the compressed data,
302
and against undetected bugs in
303
<computeroutput>bzip2</computeroutput> (hopefully very unlikely).
304
The chances of data corruption going undetected is microscopic,
305
about one chance in four billion for each file processed. Be
306
aware, though, that the check occurs upon decompression, so it
307
can only tell you that something is wrong. It can't help you
308
recover the original uncompressed data. You can use
309
<computeroutput>bzip2recover</computeroutput> to try to recover
310
data from damaged files.</para>
312
<para>Return values: 0 for a normal exit, 1 for environmental
313
problems (file not found, invalid flags, I/O errors, etc.), 2
314
to indicate a corrupt compressed file, 3 for an internal
315
consistency error (eg, bug) which caused
316
<computeroutput>bzip2</computeroutput> to panic.</para>
321
<sect1 id="options" xreflabel="OPTIONS">
322
<title>OPTIONS</title>
327
<term><computeroutput>-c --stdout</computeroutput></term>
328
<listitem><para>Compress or decompress to standard
329
output.</para></listitem>
333
<term><computeroutput>-d --decompress</computeroutput></term>
334
<listitem><para>Force decompression.
335
<computeroutput>bzip2</computeroutput>,
336
<computeroutput>bunzip2</computeroutput> and
337
<computeroutput>bzcat</computeroutput> are really the same
338
program, and the decision about what actions to take is done on
339
the basis of which name is used. This flag overrides that
340
mechanism, and forces bzip2 to decompress.</para></listitem>
344
<term><computeroutput>-z --compress</computeroutput></term>
345
<listitem><para>The complement to
346
<computeroutput>-d</computeroutput>: forces compression,
347
regardless of the invokation name.</para></listitem>
351
<term><computeroutput>-t --test</computeroutput></term>
352
<listitem><para>Check integrity of the specified file(s), but
353
don't decompress them. This really performs a trial
354
decompression and throws away the result.</para></listitem>
358
<term><computeroutput>-f --force</computeroutput></term>
359
<listitem><para>Force overwrite of output files. Normally,
360
<computeroutput>bzip2</computeroutput> will not overwrite
361
existing output files. Also forces
362
<computeroutput>bzip2</computeroutput> to break hard links to
363
files, which it otherwise wouldn't do.</para>
364
<para><computeroutput>bzip2</computeroutput> normally declines
365
to decompress files which don't have the correct magic header
366
bytes. If forced (<computeroutput>-f</computeroutput>),
367
however, it will pass such files through unmodified. This is
368
how GNU <computeroutput>gzip</computeroutput> behaves.</para>
373
<term><computeroutput>-k --keep</computeroutput></term>
374
<listitem><para>Keep (don't delete) input files during
375
compression or decompression.</para></listitem>
379
<term><computeroutput>-s --small</computeroutput></term>
380
<listitem><para>Reduce memory usage, for compression,
381
decompression and testing. Files are decompressed and tested
382
using a modified algorithm which only requires 2.5 bytes per
383
block byte. This means any file can be decompressed in 2300k
384
of memory, albeit at about half the normal speed.</para>
385
<para>During compression, <computeroutput>-s</computeroutput>
386
selects a block size of 200k, which limits memory use to around
387
the same figure, at the expense of your compression ratio. In
388
short, if your machine is low on memory (8 megabytes or less),
389
use <computeroutput>-s</computeroutput> for everything. See
390
<xref linkend="memory-management"/> below.</para></listitem>
394
<term><computeroutput>-q --quiet</computeroutput></term>
395
<listitem><para>Suppress non-essential warning messages.
396
Messages pertaining to I/O errors and other critical events
397
will not be suppressed.</para></listitem>
401
<term><computeroutput>-v --verbose</computeroutput></term>
402
<listitem><para>Verbose mode -- show the compression ratio for
403
each file processed. Further
404
<computeroutput>-v</computeroutput>'s increase the verbosity
405
level, spewing out lots of information which is primarily of
406
interest for diagnostic purposes.</para></listitem>
410
<term><computeroutput>-L --license -V --version</computeroutput></term>
411
<listitem><para>Display the software version, license terms and
412
conditions.</para></listitem>
416
<term><computeroutput>-1</computeroutput> (or
417
<computeroutput>--fast</computeroutput>) to
418
<computeroutput>-9</computeroutput> (or
419
<computeroutput>-best</computeroutput>)</term>
420
<listitem><para>Set the block size to 100 k, 200 k ... 900 k
421
when compressing. Has no effect when decompressing. See <xref
422
linkend="memory-management" /> below. The
423
<computeroutput>--fast</computeroutput> and
424
<computeroutput>--best</computeroutput> aliases are primarily
425
for GNU <computeroutput>gzip</computeroutput> compatibility.
426
In particular, <computeroutput>--fast</computeroutput> doesn't
427
make things significantly faster. And
428
<computeroutput>--best</computeroutput> merely selects the
429
default behaviour.</para></listitem>
433
<term><computeroutput>--</computeroutput></term>
434
<listitem><para>Treats all subsequent arguments as file names,
435
even if they start with a dash. This is so you can handle
436
files with names beginning with a dash, for example:
437
<computeroutput>bzip2 --
438
-myfilename</computeroutput>.</para></listitem>
442
<term><computeroutput>--repetitive-fast</computeroutput></term>
443
<term><computeroutput>--repetitive-best</computeroutput></term>
444
<listitem><para>These flags are redundant in versions 0.9.5 and
445
above. They provided some coarse control over the behaviour of
446
the sorting algorithm in earlier versions, which was sometimes
447
useful. 0.9.5 and above have an improved algorithm which
448
renders these flags irrelevant.</para></listitem>
456
<sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT">
457
<title>MEMORY MANAGEMENT</title>
459
<para><computeroutput>bzip2</computeroutput> compresses large
460
files in blocks. The block size affects both the compression
461
ratio achieved, and the amount of memory needed for compression
462
and decompression. The flags <computeroutput>-1</computeroutput>
463
through <computeroutput>-9</computeroutput> specify the block
464
size to be 100,000 bytes through 900,000 bytes (the default)
465
respectively. At decompression time, the block size used for
466
compression is read from the header of the compressed file, and
467
<computeroutput>bunzip2</computeroutput> then allocates itself
468
just enough memory to decompress the file. Since block sizes are
469
stored in compressed files, it follows that the flags
470
<computeroutput>-1</computeroutput> to
471
<computeroutput>-9</computeroutput> are irrelevant to and so
472
ignored during decompression.</para>
474
<para>Compression and decompression requirements, in bytes, can be
477
Compression: 400k + ( 8 x block size )
479
Decompression: 100k + ( 4 x block size ), or
480
100k + ( 2.5 x block size )
483
<para>Larger block sizes give rapidly diminishing marginal
484
returns. Most of the compression comes from the first two or
485
three hundred k of block size, a fact worth bearing in mind when
486
using <computeroutput>bzip2</computeroutput> on small machines.
487
It is also important to appreciate that the decompression memory
488
requirement is set at compression time by the choice of block
491
<para>For files compressed with the default 900k block size,
492
<computeroutput>bunzip2</computeroutput> will require about 3700
493
kbytes to decompress. To support decompression of any file on a
494
4 megabyte machine, <computeroutput>bunzip2</computeroutput> has
495
an option to decompress using approximately half this amount of
496
memory, about 2300 kbytes. Decompression speed is also halved,
497
so you should use this option only where necessary. The relevant
498
flag is <computeroutput>-s</computeroutput>.</para>
500
<para>In general, try and use the largest block size memory
501
constraints allow, since that maximises the compression achieved.
502
Compression and decompression speed are virtually unaffected by
505
<para>Another significant point applies to files which fit in a
506
single block -- that means most files you'd encounter using a
507
large block size. The amount of real memory touched is
508
proportional to the size of the file, since the file is smaller
509
than a block. For example, compressing a file 20,000 bytes long
510
with the flag <computeroutput>-9</computeroutput> will cause the
511
compressor to allocate around 7600k of memory, but only touch
512
400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor
513
will allocate 3700k but only touch 100k + 20000 * 4 = 180
516
<para>Here is a table which summarises the maximum memory usage
517
for different block sizes. Also recorded is the total compressed
518
size for 14 files of the Calgary Text Compression Corpus
519
totalling 3,141,622 bytes. This column gives some feel for how
520
compression varies with block size. These figures tend to
521
understate the advantage of larger block sizes for larger files,
522
since the Corpus is dominated by smaller files.</para>
525
Compress Decompress Decompress Corpus
526
Flag usage usage -s usage Size
528
-1 1200k 500k 350k 914704
529
-2 2000k 900k 600k 877703
530
-3 2800k 1300k 850k 860338
531
-4 3600k 1700k 1100k 846899
532
-5 4400k 2100k 1350k 845160
533
-6 5200k 2500k 1600k 838626
534
-7 6100k 2900k 1850k 834096
535
-8 6800k 3300k 2100k 828642
536
-9 7600k 3700k 2350k 828642
542
<sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES">
543
<title>RECOVERING DATA FROM DAMAGED FILES</title>
545
<para><computeroutput>bzip2</computeroutput> compresses files in
546
blocks, usually 900kbytes long. Each block is handled
547
independently. If a media or transmission error causes a
548
multi-block <computeroutput>.bz2</computeroutput> file to become
549
damaged, it may be possible to recover data from the undamaged
550
blocks in the file.</para>
552
<para>The compressed representation of each block is delimited by
553
a 48-bit pattern, which makes it possible to find the block
554
boundaries with reasonable certainty. Each block also carries
555
its own 32-bit CRC, so damaged blocks can be distinguished from
556
undamaged ones.</para>
558
<para><computeroutput>bzip2recover</computeroutput> is a simple
559
program whose purpose is to search for blocks in
560
<computeroutput>.bz2</computeroutput> files, and write each block
561
out into its own <computeroutput>.bz2</computeroutput> file. You
562
can then use <computeroutput>bzip2 -t</computeroutput> to test
563
the integrity of the resulting files, and decompress those which
564
are undamaged.</para>
566
<para><computeroutput>bzip2recover</computeroutput> takes a
567
single argument, the name of the damaged file, and writes a
568
number of files <computeroutput>rec0001file.bz2</computeroutput>,
569
<computeroutput>rec0002file.bz2</computeroutput>, etc, containing
570
the extracted blocks. The output filenames are designed so that
571
the use of wildcards in subsequent processing -- for example,
572
<computeroutput>bzip2 -dc rec*file.bz2 >
573
recovered_data</computeroutput> -- lists the files in the correct
576
<para><computeroutput>bzip2recover</computeroutput> should be of
577
most use dealing with large <computeroutput>.bz2</computeroutput>
578
files, as these will contain many blocks. It is clearly futile
579
to use it on damaged single-block files, since a damaged block
580
cannot be recovered. If you wish to minimise any potential data
581
loss through media or transmission errors, you might consider
582
compressing with a smaller block size.</para>
587
<sect1 id="performance" xreflabel="PERFORMANCE NOTES">
588
<title>PERFORMANCE NOTES</title>
590
<para>The sorting phase of compression gathers together similar
591
strings in the file. Because of this, files containing very long
592
runs of repeated symbols, like "aabaabaabaab ..." (repeated
593
several hundred times) may compress more slowly than normal.
594
Versions 0.9.5 and above fare much better than previous versions
595
in this respect. The ratio between worst-case and average-case
596
compression time is in the region of 10:1. For previous
597
versions, this figure was more like 100:1. You can use the
598
<computeroutput>-vvvv</computeroutput> option to monitor progress
599
in great detail, if you want.</para>
601
<para>Decompression speed is unaffected by these
604
<para><computeroutput>bzip2</computeroutput> usually allocates
605
several megabytes of memory to operate in, and then charges all
606
over it in a fairly random fashion. This means that performance,
607
both for compressing and decompressing, is largely determined by
608
the speed at which your machine can service cache misses.
609
Because of this, small changes to the code to reduce the miss
610
rate have been observed to give disproportionately large
611
performance improvements. I imagine
612
<computeroutput>bzip2</computeroutput> will perform best on
613
machines with very large caches.</para>
619
<sect1 id="caveats" xreflabel="CAVEATS">
620
<title>CAVEATS</title>
622
<para>I/O error messages are not as helpful as they could be.
623
<computeroutput>bzip2</computeroutput> tries hard to detect I/O
624
errors and exit cleanly, but the details of what the problem is
625
sometimes seem rather misleading.</para>
627
<para>This manual page pertains to version &bz-version; of
628
<computeroutput>bzip2</computeroutput>. Compressed data created by
629
this version is entirely forwards and backwards compatible with the
630
previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
631
1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
632
above can correctly decompress multiple concatenated compressed files.
633
0.1pl2 cannot do this; it will stop after decompressing just the first
634
file in the stream.</para>
636
<para><computeroutput>bzip2recover</computeroutput> versions
637
prior to 1.0.2 used 32-bit integers to represent bit positions in
638
compressed files, so it could not handle compressed files more
639
than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints
640
on some platforms which support them (GNU supported targets, and
641
Windows). To establish whether or not
642
<computeroutput>bzip2recover</computeroutput> was built with such
643
a limitation, run it without arguments. In any event you can
644
build yourself an unlimited version if you can recompile it with
645
<computeroutput>MaybeUInt64</computeroutput> set to be an
646
unsigned 64-bit integer.</para>
652
<sect1 id="author" xreflabel="AUTHOR">
653
<title>AUTHOR</title>
656
<computeroutput>&bz-email;</computeroutput></para>
658
<para>The ideas embodied in
659
<computeroutput>bzip2</computeroutput> are due to (at least) the
660
following people: Michael Burrows and David Wheeler (for the
661
block sorting transformation), David Wheeler (again, for the
662
Huffman coder), Peter Fenwick (for the structured coding model in
663
the original <computeroutput>bzip</computeroutput>, and many
664
refinements), and Alistair Moffat, Radford Neal and Ian Witten
665
(for the arithmetic coder in the original
666
<computeroutput>bzip</computeroutput>). I am much indebted for
667
their help, support and advice. See the manual in the source
668
distribution for pointers to sources of documentation. Christian
669
von Roques encouraged me to look for faster sorting algorithms,
670
so as to speed up compression. Bela Lubkin encouraged me to
671
improve the worst-case compression performance.
672
Donna Robinson XMLised the documentation.
674
patches, helped with portability problems, lent machines, gave
675
advice and were generally helpful.</para>
683
<chapter id="libprog" xreflabel="Programming with libbzip2">
685
Programming with <computeroutput>libbzip2</computeroutput>
688
<para>This chapter describes the programming interface to
689
<computeroutput>libbzip2</computeroutput>.</para>
691
<para>For general background information, particularly about
692
memory use and performance aspects, you'd be well advised to read
693
<xref linkend="using"/> as well.</para>
696
<sect1 id="top-level" xreflabel="Top-level structure">
697
<title>Top-level structure</title>
699
<para><computeroutput>libbzip2</computeroutput> is a flexible
700
library for compressing and decompressing data in the
701
<computeroutput>bzip2</computeroutput> data format. Although
702
packaged as a single entity, it helps to regard the library as
703
three separate parts: the low level interface, and the high level
704
interface, and some utility functions.</para>
706
<para>The structure of
707
<computeroutput>libbzip2</computeroutput>'s interfaces is similar
708
to that of Jean-loup Gailly's and Mark Adler's excellent
709
<computeroutput>zlib</computeroutput> library.</para>
711
<para>All externally visible symbols have names beginning
712
<computeroutput>BZ2_</computeroutput>. This is new in version
713
1.0. The intention is to minimise pollution of the namespaces of
714
library clients.</para>
716
<para>To use any part of the library, you need to
717
<computeroutput>#include <bzlib.h></computeroutput>
718
into your sources.</para>
722
<sect2 id="ll-summary" xreflabel="Low-level summary">
723
<title>Low-level summary</title>
725
<para>This interface provides services for compressing and
726
decompressing data in memory. There's no provision for dealing
727
with files, streams or any other I/O mechanisms, just straight
728
memory-to-memory work. In fact, this part of the library can be
729
compiled without inclusion of
730
<computeroutput>stdio.h</computeroutput>, which may be helpful
731
for embedded applications.</para>
733
<para>The low-level part of the library has no global variables
734
and is therefore thread-safe.</para>
736
<para>Six routines make up the low level interface:
737
<computeroutput>BZ2_bzCompressInit</computeroutput>,
738
<computeroutput>BZ2_bzCompress</computeroutput>, and
739
<computeroutput>BZ2_bzCompressEnd</computeroutput> for
740
compression, and a corresponding trio
741
<computeroutput>BZ2_bzDecompressInit</computeroutput>,
742
<computeroutput>BZ2_bzDecompress</computeroutput> and
743
<computeroutput>BZ2_bzDecompressEnd</computeroutput> for
744
decompression. The <computeroutput>*Init</computeroutput>
745
functions allocate memory for compression/decompression and do
746
other initialisations, whilst the
747
<computeroutput>*End</computeroutput> functions close down
748
operations and release memory.</para>
750
<para>The real work is done by
751
<computeroutput>BZ2_bzCompress</computeroutput> and
752
<computeroutput>BZ2_bzDecompress</computeroutput>. These
753
compress and decompress data from a user-supplied input buffer to
754
a user-supplied output buffer. These buffers can be any size;
755
arbitrary quantities of data are handled by making repeated calls
756
to these functions. This is a flexible mechanism allowing a
757
consumer-pull style of activity, or producer-push, or a mixture
763
<sect2 id="hl-summary" xreflabel="High-level summary">
764
<title>High-level summary</title>
766
<para>This interface provides some handy wrappers around the
767
low-level interface to facilitate reading and writing
768
<computeroutput>bzip2</computeroutput> format files
769
(<computeroutput>.bz2</computeroutput> files). The routines
770
provide hooks to facilitate reading files in which the
771
<computeroutput>bzip2</computeroutput> data stream is embedded
772
within some larger-scale file structure, or where there are
773
multiple <computeroutput>bzip2</computeroutput> data streams
774
concatenated end-to-end.</para>
776
<para>For reading files,
777
<computeroutput>BZ2_bzReadOpen</computeroutput>,
778
<computeroutput>BZ2_bzRead</computeroutput>,
779
<computeroutput>BZ2_bzReadClose</computeroutput> and
780
<computeroutput>BZ2_bzReadGetUnused</computeroutput> are
781
supplied. For writing files,
782
<computeroutput>BZ2_bzWriteOpen</computeroutput>,
783
<computeroutput>BZ2_bzWrite</computeroutput> and
784
<computeroutput>BZ2_bzWriteFinish</computeroutput> are
787
<para>As with the low-level library, no global variables are used
788
so the library is per se thread-safe. However, if I/O errors
789
occur whilst reading or writing the underlying compressed files,
790
you may have to consult <computeroutput>errno</computeroutput> to
791
determine the cause of the error. In that case, you'd need a C
792
library which correctly supports
793
<computeroutput>errno</computeroutput> in a multithreaded
796
<para>To make the library a little simpler and more portable,
797
<computeroutput>BZ2_bzReadOpen</computeroutput> and
798
<computeroutput>BZ2_bzWriteOpen</computeroutput> require you to
799
pass them file handles (<computeroutput>FILE*</computeroutput>s)
800
which have previously been opened for reading or writing
801
respectively. That avoids portability problems associated with
802
file operations and file attributes, whilst not being much of an
803
imposition on the programmer.</para>
808
<sect2 id="util-fns-summary" xreflabel="Utility functions summary">
809
<title>Utility functions summary</title>
811
<para>For very simple needs,
812
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
813
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are
814
provided. These compress data in memory from one buffer to
815
another buffer in a single function call. You should assess
816
whether these functions fulfill your memory-to-memory
817
compression/decompression requirements before investing effort in
818
understanding the more general but more complex low-level
821
<para>Yoshioka Tsuneo
822
(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has
823
contributed some functions to give better
824
<computeroutput>zlib</computeroutput> compatibility. These
825
functions are <computeroutput>BZ2_bzopen</computeroutput>,
826
<computeroutput>BZ2_bzread</computeroutput>,
827
<computeroutput>BZ2_bzwrite</computeroutput>,
828
<computeroutput>BZ2_bzflush</computeroutput>,
829
<computeroutput>BZ2_bzclose</computeroutput>,
830
<computeroutput>BZ2_bzerror</computeroutput> and
831
<computeroutput>BZ2_bzlibVersion</computeroutput>. You may find
832
these functions more convenient for simple file reading and
833
writing, than those in the high-level interface. These functions
834
are not (yet) officially part of the library, and are minimally
835
documented here. If they break, you get to keep all the pieces.
836
I hope to document them properly when time permits.</para>
838
<para>Yoshioka also contributed modifications to allow the
839
library to be built as a Windows DLL.</para>
846
<sect1 id="err-handling" xreflabel="Error handling">
847
<title>Error handling</title>
849
<para>The library is designed to recover cleanly in all
850
situations, including the worst-case situation of decompressing
851
random data. I'm not 100% sure that it can always do this, so
852
you might want to add a signal handler to catch segmentation
853
violations during decompression if you are feeling especially
854
paranoid. I would be interested in hearing more about the
855
robustness of the library to corrupted compressed data.</para>
857
<para>Version 1.0.3 more robust in this respect than any
858
previous version. Investigations with Valgrind (a tool for detecting
859
problems with memory management) indicate
860
that, at least for the few files I tested, all single-bit errors
861
in the decompressed data are caught properly, with no
862
segmentation faults, no uses of uninitialised data, no out of
863
range reads or writes, and no infinite looping in the decompressor.
864
So it's certainly pretty robust, although
865
I wouldn't claim it to be totally bombproof.</para>
867
<para>The file <computeroutput>bzlib.h</computeroutput> contains
868
all definitions needed to use the library. In particular, you
869
should definitely not include
870
<computeroutput>bzlib_private.h</computeroutput>.</para>
872
<para>In <computeroutput>bzlib.h</computeroutput>, the various
873
return values are defined. The following list is not intended as
874
an exhaustive description of the circumstances in which a given
875
value may be returned -- those descriptions are given later.
876
Rather, it is intended to convey the rough meaning of each return
877
value. The first five actions are normal and not intended to
878
denote an error situation.</para>
883
<term><computeroutput>BZ_OK</computeroutput></term>
884
<listitem><para>The requested action was completed
885
successfully.</para></listitem>
889
<term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK,
890
BZ_FINISH_OK</computeroutput></term>
892
<computeroutput>BZ2_bzCompress</computeroutput>, the requested
893
flush/finish/nothing-special action was completed
894
successfully.</para></listitem>
898
<term><computeroutput>BZ_STREAM_END</computeroutput></term>
899
<listitem><para>Compression of data was completed, or the
900
logical stream end was detected during
901
decompression.</para></listitem>
906
<para>The following return values indicate an error of some
912
<term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term>
913
<listitem><para>Indicates that the library has been improperly
914
compiled on your platform -- a major configuration error.
915
Specifically, it means that
916
<computeroutput>sizeof(char)</computeroutput>,
917
<computeroutput>sizeof(short)</computeroutput> and
918
<computeroutput>sizeof(int)</computeroutput> are not 1, 2 and
919
4 respectively, as they should be. Note that the library
920
should still work properly on 64-bit platforms which follow
921
the LP64 programming model -- that is, where
922
<computeroutput>sizeof(long)</computeroutput> and
923
<computeroutput>sizeof(void*)</computeroutput> are 8. Under
924
LP64, <computeroutput>sizeof(int)</computeroutput> is still 4,
925
so <computeroutput>libbzip2</computeroutput>, which doesn't
926
use the <computeroutput>long</computeroutput> type, is
927
OK.</para></listitem>
931
<term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term>
932
<listitem><para>When using the library, it is important to call
933
the functions in the correct sequence and with data structures
934
(buffers etc) in the correct states.
935
<computeroutput>libbzip2</computeroutput> checks as much as it
936
can to ensure this is happening, and returns
937
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not.
938
Code which complies precisely with the function semantics, as
939
detailed below, should never receive this value; such an event
940
denotes buggy code which you should
941
investigate.</para></listitem>
945
<term><computeroutput>BZ_PARAM_ERROR</computeroutput></term>
946
<listitem><para>Returned when a parameter to a function call is
947
out of range or otherwise manifestly incorrect. As with
948
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this
949
denotes a bug in the client code. The distinction between
950
<computeroutput>BZ_PARAM_ERROR</computeroutput> and
951
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit
952
hazy, but still worth making.</para></listitem>
956
<term><computeroutput>BZ_MEM_ERROR</computeroutput></term>
957
<listitem><para>Returned when a request to allocate memory
958
failed. Note that the quantity of memory needed to decompress
959
a stream cannot be determined until the stream's header has
961
<computeroutput>BZ2_bzDecompress</computeroutput> and
962
<computeroutput>BZ2_bzRead</computeroutput> may return
963
<computeroutput>BZ_MEM_ERROR</computeroutput> even though some
964
of the compressed data has been read. The same is not true
965
for compression; once
966
<computeroutput>BZ2_bzCompressInit</computeroutput> or
967
<computeroutput>BZ2_bzWriteOpen</computeroutput> have
968
successfully completed,
969
<computeroutput>BZ_MEM_ERROR</computeroutput> cannot
970
occur.</para></listitem>
974
<term><computeroutput>BZ_DATA_ERROR</computeroutput></term>
975
<listitem><para>Returned when a data integrity error is
976
detected during decompression. Most importantly, this means
977
when stored and computed CRCs for the data do not match. This
978
value is also returned upon detection of any other anomaly in
979
the compressed data.</para></listitem>
983
<term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term>
984
<listitem><para>As a special case of
985
<computeroutput>BZ_DATA_ERROR</computeroutput>, it is
986
sometimes useful to know when the compressed stream does not
987
start with the correct magic bytes (<computeroutput>'B' 'Z'
988
'h'</computeroutput>).</para></listitem>
992
<term><computeroutput>BZ_IO_ERROR</computeroutput></term>
993
<listitem><para>Returned by
994
<computeroutput>BZ2_bzRead</computeroutput> and
995
<computeroutput>BZ2_bzWrite</computeroutput> when there is an
996
error reading or writing in the compressed file, and by
997
<computeroutput>BZ2_bzReadOpen</computeroutput> and
998
<computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts
999
to use a file for which the error indicator (viz,
1000
<computeroutput>ferror(f)</computeroutput>) is set. On
1001
receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the
1002
caller should consult <computeroutput>errno</computeroutput>
1003
and/or <computeroutput>perror</computeroutput> to acquire
1004
operating-system specific information about the
1005
problem.</para></listitem>
1009
<term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term>
1010
<listitem><para>Returned by
1011
<computeroutput>BZ2_bzRead</computeroutput> when the
1012
compressed file finishes before the logical end of stream is
1013
detected.</para></listitem>
1017
<term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term>
1018
<listitem><para>Returned by
1019
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
1020
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to
1021
indicate that the output data will not fit into the output
1022
buffer provided.</para></listitem>
1031
<sect1 id="low-level" xreflabel=">Low-level interface">
1032
<title>Low-level interface</title>
1035
<sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit">
1036
<title>BZ2_bzCompressInit</title>
1041
unsigned int avail_in;
1042
unsigned int total_in_lo32;
1043
unsigned int total_in_hi32;
1046
unsigned int avail_out;
1047
unsigned int total_out_lo32;
1048
unsigned int total_out_hi32;
1052
void *(*bzalloc)(void *,int,int);
1053
void (*bzfree)(void *,void *);
1057
int BZ2_bzCompressInit ( bz_stream *strm,
1063
<para>Prepares for compression. The
1064
<computeroutput>bz_stream</computeroutput> structure holds all
1065
data pertaining to the compression activity. A
1066
<computeroutput>bz_stream</computeroutput> structure should be
1067
allocated and initialised prior to the call. The fields of
1068
<computeroutput>bz_stream</computeroutput> comprise the entirety
1069
of the user-visible data. <computeroutput>state</computeroutput>
1070
is a pointer to the private data structures required for
1073
<para>Custom memory allocators are supported, via fields
1074
<computeroutput>bzalloc</computeroutput>,
1075
<computeroutput>bzfree</computeroutput>, and
1076
<computeroutput>opaque</computeroutput>. The value
1077
<computeroutput>opaque</computeroutput> is passed to as the first
1078
argument to all calls to <computeroutput>bzalloc</computeroutput>
1079
and <computeroutput>bzfree</computeroutput>, but is otherwise
1080
ignored by the library. The call <computeroutput>bzalloc (
1081
opaque, n, m )</computeroutput> is expected to return a pointer
1082
<computeroutput>p</computeroutput> to <computeroutput>n *
1083
m</computeroutput> bytes of memory, and <computeroutput>bzfree (
1084
opaque, p )</computeroutput> should free that memory.</para>
1086
<para>If you don't want to use a custom memory allocator, set
1087
<computeroutput>bzalloc</computeroutput>,
1088
<computeroutput>bzfree</computeroutput> and
1089
<computeroutput>opaque</computeroutput> to
1090
<computeroutput>NULL</computeroutput>, and the library will then
1091
use the standard <computeroutput>malloc</computeroutput> /
1092
<computeroutput>free</computeroutput> routines.</para>
1094
<para>Before calling
1095
<computeroutput>BZ2_bzCompressInit</computeroutput>, fields
1096
<computeroutput>bzalloc</computeroutput>,
1097
<computeroutput>bzfree</computeroutput> and
1098
<computeroutput>opaque</computeroutput> should be filled
1099
appropriately, as just described. Upon return, the internal
1100
state will have been allocated and initialised, and
1101
<computeroutput>total_in_lo32</computeroutput>,
1102
<computeroutput>total_in_hi32</computeroutput>,
1103
<computeroutput>total_out_lo32</computeroutput> and
1104
<computeroutput>total_out_hi32</computeroutput> will have been
1105
set to zero. These four fields are used by the library to inform
1106
the caller of the total amount of data passed into and out of the
1107
library, respectively. You should not try to change them. As of
1108
version 1.0, 64-bit counts are maintained, even on 32-bit
1109
platforms, using the <computeroutput>_hi32</computeroutput>
1110
fields to store the upper 32 bits of the count. So, for example,
1111
the total amount of data in is <computeroutput>(total_in_hi32
1112
<< 32) + total_in_lo32</computeroutput>.</para>
1114
<para>Parameter <computeroutput>blockSize100k</computeroutput>
1115
specifies the block size to be used for compression. It should
1116
be a value between 1 and 9 inclusive, and the actual block size
1117
used is 100000 x this figure. 9 gives the best compression but
1118
takes most memory.</para>
1120
<para>Parameter <computeroutput>verbosity</computeroutput> should
1121
be set to a number between 0 and 4 inclusive. 0 is silent, and
1122
greater numbers give increasingly verbose monitoring/debugging
1123
output. If the library has been compiled with
1124
<computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output
1125
will appear for any verbosity setting.</para>
1127
<para>Parameter <computeroutput>workFactor</computeroutput>
1128
controls how the compression phase behaves when presented with
1129
worst case, highly repetitive, input data. If compression runs
1130
into difficulties caused by repetitive data, the library switches
1131
from the standard sorting algorithm to a fallback algorithm. The
1132
fallback is slower than the standard algorithm by perhaps a
1133
factor of three, but always behaves reasonably, no matter how bad
1136
<para>Lower values of <computeroutput>workFactor</computeroutput>
1137
reduce the amount of effort the standard algorithm will expend
1138
before resorting to the fallback. You should set this parameter
1139
carefully; too low, and many inputs will be handled by the
1140
fallback algorithm and so compress rather slowly, too high, and
1141
your average-to-worst case compression times can become very
1142
large. The default value of 30 gives reasonable behaviour over a
1143
wide range of circumstances.</para>
1145
<para>Allowable values range from 0 to 250 inclusive. 0 is a
1146
special case, equivalent to using the default value of 30.</para>
1148
<para>Note that the compressed output generated is the same
1149
regardless of whether or not the fallback algorithm is
1152
<para>Be aware also that this parameter may disappear entirely in
1153
future versions of the library. In principle it should be
1154
possible to devise a good way to automatically choose which
1155
algorithm to use. Such a mechanism would render the parameter
1158
<para>Possible return values:</para>
1162
if the library has been mis-compiled
1165
or blockSize < 1 or blockSize > 9
1166
or verbosity < 0 or verbosity > 4
1167
or workFactor < 0 or workFactor > 250
1169
if not enough memory is available
1174
<para>Allowable next actions:</para>
1178
if BZ_OK is returned
1179
no specific action needed in case of error
1185
<sect2 id="bzCompress" xreflabel="BZ2_bzCompress">
1186
<title>BZ2_bzCompress</title>
1189
int BZ2_bzCompress ( bz_stream *strm, int action );
1192
<para>Provides more input and/or output buffer space for the
1193
library. The caller maintains input and output buffers, and
1194
calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer
1195
data between them.</para>
1197
<para>Before each call to
1198
<computeroutput>BZ2_bzCompress</computeroutput>,
1199
<computeroutput>next_in</computeroutput> should point at the data
1200
to be compressed, and <computeroutput>avail_in</computeroutput>
1201
should indicate how many bytes the library may read.
1202
<computeroutput>BZ2_bzCompress</computeroutput> updates
1203
<computeroutput>next_in</computeroutput>,
1204
<computeroutput>avail_in</computeroutput> and
1205
<computeroutput>total_in</computeroutput> to reflect the number
1206
of bytes it has read.</para>
1208
<para>Similarly, <computeroutput>next_out</computeroutput> should
1209
point to a buffer in which the compressed data is to be placed,
1210
with <computeroutput>avail_out</computeroutput> indicating how
1211
much output space is available.
1212
<computeroutput>BZ2_bzCompress</computeroutput> updates
1213
<computeroutput>next_out</computeroutput>,
1214
<computeroutput>avail_out</computeroutput> and
1215
<computeroutput>total_out</computeroutput> to reflect the number
1216
of bytes output.</para>
1218
<para>You may provide and remove as little or as much data as you
1219
like on each call of
1220
<computeroutput>BZ2_bzCompress</computeroutput>. In the limit,
1221
it is acceptable to supply and remove data one byte at a time,
1222
although this would be terribly inefficient. You should always
1223
ensure that at least one byte of output space is available at
1226
<para>A second purpose of
1227
<computeroutput>BZ2_bzCompress</computeroutput> is to request a
1228
change of mode of the compressed stream.</para>
1230
<para>Conceptually, a compressed stream can be in one of four
1231
states: IDLE, RUNNING, FLUSHING and FINISHING. Before
1233
(<computeroutput>BZ2_bzCompressInit</computeroutput>) and after
1234
termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>),
1235
a stream is regarded as IDLE.</para>
1237
<para>Upon initialisation
1238
(<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream
1239
is placed in the RUNNING state. Subsequent calls to
1240
<computeroutput>BZ2_bzCompress</computeroutput> should pass
1241
<computeroutput>BZ_RUN</computeroutput> as the requested action;
1242
other actions are illegal and will result in
1243
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.</para>
1245
<para>At some point, the calling program will have provided all
1246
the input data it wants to. It will then want to finish up -- in
1247
effect, asking the library to process any data it might have
1248
buffered internally. In this state,
1249
<computeroutput>BZ2_bzCompress</computeroutput> will no longer
1250
attempt to read data from
1251
<computeroutput>next_in</computeroutput>, but it will want to
1252
write data to <computeroutput>next_out</computeroutput>. Because
1253
the output buffer supplied by the user can be arbitrarily small,
1254
the finishing-up operation cannot necessarily be done with a
1256
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
1258
<para>Instead, the calling program passes
1259
<computeroutput>BZ_FINISH</computeroutput> as an action to
1260
<computeroutput>BZ2_bzCompress</computeroutput>. This changes
1261
the stream's state to FINISHING. Any remaining input (ie,
1262
<computeroutput>next_in[0 .. avail_in-1]</computeroutput>) is
1263
compressed and transferred to the output buffer. To do this,
1264
<computeroutput>BZ2_bzCompress</computeroutput> must be called
1265
repeatedly until all the output has been consumed. At that
1266
point, <computeroutput>BZ2_bzCompress</computeroutput> returns
1267
<computeroutput>BZ_STREAM_END</computeroutput>, and the stream's
1268
state is set back to IDLE.
1269
<computeroutput>BZ2_bzCompressEnd</computeroutput> should then be
1272
<para>Just to make sure the calling program does not cheat, the
1273
library makes a note of <computeroutput>avail_in</computeroutput>
1274
at the time of the first call to
1275
<computeroutput>BZ2_bzCompress</computeroutput> which has
1276
<computeroutput>BZ_FINISH</computeroutput> as an action (ie, at
1277
the time the program has announced its intention to not supply
1278
any more input). By comparing this value with that of
1279
<computeroutput>avail_in</computeroutput> over subsequent calls
1280
to <computeroutput>BZ2_bzCompress</computeroutput>, the library
1281
can detect any attempts to slip in more data to compress. Any
1282
calls for which this is detected will return
1283
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>. This
1284
indicates a programming mistake which should be corrected.</para>
1286
<para>Instead of asking to finish, the calling program may ask
1287
<computeroutput>BZ2_bzCompress</computeroutput> to take all the
1288
remaining input, compress it and terminate the current
1289
(Burrows-Wheeler) compression block. This could be useful for
1290
error control purposes. The mechanism is analogous to that for
1291
finishing: call <computeroutput>BZ2_bzCompress</computeroutput>
1292
with an action of <computeroutput>BZ_FLUSH</computeroutput>,
1293
remove output data, and persist with the
1294
<computeroutput>BZ_FLUSH</computeroutput> action until the value
1295
<computeroutput>BZ_RUN</computeroutput> is returned. As with
1296
finishing, <computeroutput>BZ2_bzCompress</computeroutput>
1297
detects any attempt to provide more input data once the flush has
1300
<para>Once the flush is complete, the stream returns to the
1301
normal RUNNING state.</para>
1303
<para>This all sounds pretty complex, but isn't really. Here's a
1304
table which shows which actions are allowable in each state, what
1305
action will be taken, what the next state is, and what the
1306
non-error return values are. Note that you can't explicitly ask
1307
what state the stream is in, but nor do you need to -- it can be
1308
inferred from the values returned by
1309
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
1313
Illegal. IDLE state only exists after BZ2_bzCompressEnd or
1314
before BZ2_bzCompressInit.
1315
Return value = BZ_SEQUENCE_ERROR
1318
Compress from next_in to next_out as much as possible.
1319
Next state = RUNNING
1320
Return value = BZ_RUN_OK
1323
Remember current value of next_in. Compress from next_in
1324
to next_out as much as possible, but do not accept any more input.
1325
Next state = FLUSHING
1326
Return value = BZ_FLUSH_OK
1329
Remember current value of next_in. Compress from next_in
1330
to next_out as much as possible, but do not accept any more input.
1331
Next state = FINISHING
1332
Return value = BZ_FINISH_OK
1335
Compress from next_in to next_out as much as possible,
1336
but do not accept any more input.
1337
If all the existing input has been used up and all compressed
1338
output has been removed
1339
Next state = RUNNING; Return value = BZ_RUN_OK
1341
Next state = FLUSHING; Return value = BZ_FLUSH_OK
1345
Return value = BZ_SEQUENCE_ERROR
1348
Compress from next_in to next_out as much as possible,
1349
but to not accept any more input.
1350
If all the existing input has been used up and all compressed
1351
output has been removed
1352
Next state = IDLE; Return value = BZ_STREAM_END
1354
Next state = FINISHING; Return value = BZ_FINISH_OK
1358
Return value = BZ_SEQUENCE_ERROR
1362
<para>That still looks complicated? Well, fair enough. The
1363
usual sequence of calls for compressing a load of data is:</para>
1367
<listitem><para>Get started with
1368
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para></listitem>
1370
<listitem><para>Shovel data in and shlurp out its compressed form
1371
using zero or more calls of
1372
<computeroutput>BZ2_bzCompress</computeroutput> with action =
1373
<computeroutput>BZ_RUN</computeroutput>.</para></listitem>
1375
<listitem><para>Finish up. Repeatedly call
1376
<computeroutput>BZ2_bzCompress</computeroutput> with action =
1377
<computeroutput>BZ_FINISH</computeroutput>, copying out the
1378
compressed output, until
1379
<computeroutput>BZ_STREAM_END</computeroutput> is
1380
returned.</para></listitem> <listitem><para>Close up and go home. Call
1381
<computeroutput>BZ2_bzCompressEnd</computeroutput>.</para></listitem>
1385
<para>If the data you want to compress fits into your input
1386
buffer all at once, you can skip the calls of
1387
<computeroutput>BZ2_bzCompress ( ..., BZ_RUN )</computeroutput>
1388
and just do the <computeroutput>BZ2_bzCompress ( ..., BZ_FINISH
1389
)</computeroutput> calls.</para>
1391
<para>All required memory is allocated by
1392
<computeroutput>BZ2_bzCompressInit</computeroutput>. The
1393
compression library can accept any data at all (obviously). So
1394
you shouldn't get any error return values from the
1395
<computeroutput>BZ2_bzCompress</computeroutput> calls. If you
1397
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, and indicate
1398
a bug in your programming.</para>
1400
<para>Trivial other possible return values:</para>
1404
if strm is NULL, or strm->s is NULL
1410
<sect2 id="bzCompress-end" xreflabel="BZ2_bzCompressEnd">
1411
<title>BZ2_bzCompressEnd</title>
1414
int BZ2_bzCompressEnd ( bz_stream *strm );
1417
<para>Releases all memory associated with a compression
1420
<para>Possible return values:</para>
1423
BZ_PARAM_ERROR if strm is NULL or strm->s is NULL
1430
<sect2 id="bzDecompress-init" xreflabel="BZ2_bzDecompressInit">
1431
<title>BZ2_bzDecompressInit</title>
1434
int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );
1437
<para>Prepares for decompression. As with
1438
<computeroutput>BZ2_bzCompressInit</computeroutput>, a
1439
<computeroutput>bz_stream</computeroutput> record should be
1440
allocated and initialised before the call. Fields
1441
<computeroutput>bzalloc</computeroutput>,
1442
<computeroutput>bzfree</computeroutput> and
1443
<computeroutput>opaque</computeroutput> should be set if a custom
1444
memory allocator is required, or made
1445
<computeroutput>NULL</computeroutput> for the normal
1446
<computeroutput>malloc</computeroutput> /
1447
<computeroutput>free</computeroutput> routines. Upon return, the
1448
internal state will have been initialised, and
1449
<computeroutput>total_in</computeroutput> and
1450
<computeroutput>total_out</computeroutput> will be zero.</para>
1452
<para>For the meaning of parameter
1453
<computeroutput>verbosity</computeroutput>, see
1454
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
1456
<para>If <computeroutput>small</computeroutput> is nonzero, the
1457
library will use an alternative decompression algorithm which
1458
uses less memory but at the cost of decompressing more slowly
1459
(roughly speaking, half the speed, but the maximum memory
1460
requirement drops to around 2300k). See <xref linkend="using"/>
1461
for more information on memory management.</para>
1463
<para>Note that the amount of memory needed to decompress a
1464
stream cannot be determined until the stream's header has been
1466
<computeroutput>BZ2_bzDecompressInit</computeroutput> succeeds, a
1467
subsequent <computeroutput>BZ2_bzDecompress</computeroutput>
1469
<computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
1471
<para>Possible return values:</para>
1475
if the library has been mis-compiled
1477
if ( small != 0 && small != 1 )
1478
or (verbosity <; 0 || verbosity > 4)
1480
if insufficient memory is available
1483
<para>Allowable next actions:</para>
1487
if BZ_OK was returned
1488
no specific action required in case of error
1494
<sect2 id="bzDecompress" xreflabel="BZ2_bzDecompress">
1495
<title>BZ2_bzDecompress</title>
1498
int BZ2_bzDecompress ( bz_stream *strm );
1501
<para>Provides more input and/out output buffer space for the
1502
library. The caller maintains input and output buffers, and uses
1503
<computeroutput>BZ2_bzDecompress</computeroutput> to transfer
1504
data between them.</para>
1506
<para>Before each call to
1507
<computeroutput>BZ2_bzDecompress</computeroutput>,
1508
<computeroutput>next_in</computeroutput> should point at the
1509
compressed data, and <computeroutput>avail_in</computeroutput>
1510
should indicate how many bytes the library may read.
1511
<computeroutput>BZ2_bzDecompress</computeroutput> updates
1512
<computeroutput>next_in</computeroutput>,
1513
<computeroutput>avail_in</computeroutput> and
1514
<computeroutput>total_in</computeroutput> to reflect the number
1515
of bytes it has read.</para>
1517
<para>Similarly, <computeroutput>next_out</computeroutput> should
1518
point to a buffer in which the uncompressed output is to be
1519
placed, with <computeroutput>avail_out</computeroutput>
1520
indicating how much output space is available.
1521
<computeroutput>BZ2_bzCompress</computeroutput> updates
1522
<computeroutput>next_out</computeroutput>,
1523
<computeroutput>avail_out</computeroutput> and
1524
<computeroutput>total_out</computeroutput> to reflect the number
1525
of bytes output.</para>
1527
<para>You may provide and remove as little or as much data as you
1528
like on each call of
1529
<computeroutput>BZ2_bzDecompress</computeroutput>. In the limit,
1530
it is acceptable to supply and remove data one byte at a time,
1531
although this would be terribly inefficient. You should always
1532
ensure that at least one byte of output space is available at
1535
<para>Use of <computeroutput>BZ2_bzDecompress</computeroutput> is
1537
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
1539
<para>You should provide input and remove output as described
1540
above, and repeatedly call
1541
<computeroutput>BZ2_bzDecompress</computeroutput> until
1542
<computeroutput>BZ_STREAM_END</computeroutput> is returned.
1543
Appearance of <computeroutput>BZ_STREAM_END</computeroutput>
1544
denotes that <computeroutput>BZ2_bzDecompress</computeroutput>
1545
has detected the logical end of the compressed stream.
1546
<computeroutput>BZ2_bzDecompress</computeroutput> will not
1547
produce <computeroutput>BZ_STREAM_END</computeroutput> until all
1548
output data has been placed into the output buffer, so once
1549
<computeroutput>BZ_STREAM_END</computeroutput> appears, you are
1550
guaranteed to have available all the decompressed output, and
1551
<computeroutput>BZ2_bzDecompressEnd</computeroutput> can safely
1554
<para>If case of an error return value, you should call
1555
<computeroutput>BZ2_bzDecompressEnd</computeroutput> to clean up
1556
and release memory.</para>
1558
<para>Possible return values:</para>
1562
if strm is NULL or strm->s is NULL
1563
or strm->avail_out < 1
1565
if a data integrity error is detected in the compressed stream
1567
if the compressed stream doesn't begin with the right magic bytes
1569
if there wasn't enough memory available
1571
if the logical end of the data stream was detected and all
1572
output in has been consumed, eg s-->avail_out > 0
1577
<para>Allowable next actions:</para>
1581
if BZ_OK was returned
1589
<sect2 id="bzDecompress-end" xreflabel="BZ2_bzDecompressEnd">
1590
<title>BZ2_bzDecompressEnd</title>
1593
int BZ2_bzDecompressEnd ( bz_stream *strm );
1596
<para>Releases all memory associated with a decompression
1599
<para>Possible return values:</para>
1603
if strm is NULL or strm->s is NULL
1608
<para>Allowable next actions:</para>
1619
<sect1 id="hl-interface" xreflabel="High-level interface">
1620
<title>High-level interface</title>
1622
<para>This interface provides functions for reading and writing
1623
<computeroutput>bzip2</computeroutput> format files. First, some
1624
general points.</para>
1626
<itemizedlist mark='bullet'>
1628
<listitem><para>All of the functions take an
1629
<computeroutput>int*</computeroutput> first argument,
1630
<computeroutput>bzerror</computeroutput>. After each call,
1631
<computeroutput>bzerror</computeroutput> should be consulted
1632
first to determine the outcome of the call. If
1633
<computeroutput>bzerror</computeroutput> is
1634
<computeroutput>BZ_OK</computeroutput>, the call completed
1635
successfully, and only then should the return value of the
1636
function (if any) be consulted. If
1637
<computeroutput>bzerror</computeroutput> is
1638
<computeroutput>BZ_IO_ERROR</computeroutput>, there was an
1639
error reading/writing the underlying compressed file, and you
1640
should then consult <computeroutput>errno</computeroutput> /
1641
<computeroutput>perror</computeroutput> to determine the cause
1642
of the difficulty. <computeroutput>bzerror</computeroutput>
1643
may also be set to various other values; precise details are
1644
given on a per-function basis below.</para></listitem>
1646
<listitem><para>If <computeroutput>bzerror</computeroutput> indicates
1647
an error (ie, anything except
1648
<computeroutput>BZ_OK</computeroutput> and
1649
<computeroutput>BZ_STREAM_END</computeroutput>), you should
1651
<computeroutput>BZ2_bzReadClose</computeroutput> (or
1652
<computeroutput>BZ2_bzWriteClose</computeroutput>, depending on
1653
whether you are attempting to read or to write) to free up all
1654
resources associated with the stream. Once an error has been
1655
indicated, behaviour of all calls except
1656
<computeroutput>BZ2_bzReadClose</computeroutput>
1657
(<computeroutput>BZ2_bzWriteClose</computeroutput>) is
1658
undefined. The implication is that (1)
1659
<computeroutput>bzerror</computeroutput> should be checked
1660
after each call, and (2) if
1661
<computeroutput>bzerror</computeroutput> indicates an error,
1662
<computeroutput>BZ2_bzReadClose</computeroutput>
1663
(<computeroutput>BZ2_bzWriteClose</computeroutput>) should then
1664
be called to clean up.</para></listitem>
1666
<listitem><para>The <computeroutput>FILE*</computeroutput> arguments
1667
passed to <computeroutput>BZ2_bzReadOpen</computeroutput> /
1668
<computeroutput>BZ2_bzWriteOpen</computeroutput> should be set
1669
to binary mode. Most Unix systems will do this by default, but
1670
other platforms, including Windows and Mac, will not. If you
1671
omit this, you may encounter problems when moving code to new
1672
platforms.</para></listitem>
1674
<listitem><para>Memory allocation requests are handled by
1675
<computeroutput>malloc</computeroutput> /
1676
<computeroutput>free</computeroutput>. At present there is no
1677
facility for user-defined memory allocators in the file I/O
1678
functions (could easily be added, though).</para></listitem>
1684
<sect2 id="bzreadopen" xreflabel="BZ2_bzReadOpen">
1685
<title>BZ2_bzReadOpen</title>
1688
typedef void BZFILE;
1690
BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f,
1691
int verbosity, int small,
1692
void *unused, int nUnused );
1695
<para>Prepare to read compressed data from file handle
1696
<computeroutput>f</computeroutput>.
1697
<computeroutput>f</computeroutput> should refer to a file which
1698
has been opened for reading, and for which the error indicator
1699
(<computeroutput>ferror(f)</computeroutput>)is not set. If
1700
<computeroutput>small</computeroutput> is 1, the library will try
1701
to decompress using less memory, at the expense of speed.</para>
1703
<para>For reasons explained below,
1704
<computeroutput>BZ2_bzRead</computeroutput> will decompress the
1705
<computeroutput>nUnused</computeroutput> bytes starting at
1706
<computeroutput>unused</computeroutput>, before starting to read
1707
from the file <computeroutput>f</computeroutput>. At most
1708
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes may be
1709
supplied like this. If this facility is not required, you should
1710
pass <computeroutput>NULL</computeroutput> and
1711
<computeroutput>0</computeroutput> for
1712
<computeroutput>unused</computeroutput> and
1713
n<computeroutput>Unused</computeroutput> respectively.</para>
1715
<para>For the meaning of parameters
1716
<computeroutput>small</computeroutput> and
1717
<computeroutput>verbosity</computeroutput>, see
1718
<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
1720
<para>The amount of memory needed to decompress a file cannot be
1721
determined until the file's header has been read. So it is
1722
possible that <computeroutput>BZ2_bzReadOpen</computeroutput>
1723
returns <computeroutput>BZ_OK</computeroutput> but a subsequent
1724
call of <computeroutput>BZ2_bzRead</computeroutput> will return
1725
<computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
1727
<para>Possible assignments to
1728
<computeroutput>bzerror</computeroutput>:</para>
1732
if the library has been mis-compiled
1735
or small is neither 0 nor 1
1736
or ( unused == NULL && nUnused != 0 )
1737
or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) )
1739
if ferror(f) is nonzero
1741
if insufficient memory is available
1746
<para>Possible return values:</para>
1749
Pointer to an abstract BZFILE
1755
<para>Allowable next actions:</para>
1767
<sect2 id="bzread" xreflabel="BZ2_bzRead">
1768
<title>BZ2_bzRead</title>
1771
int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );
1774
<para>Reads up to <computeroutput>len</computeroutput>
1775
(uncompressed) bytes from the compressed file
1776
<computeroutput>b</computeroutput> into the buffer
1777
<computeroutput>buf</computeroutput>. If the read was
1778
successful, <computeroutput>bzerror</computeroutput> is set to
1779
<computeroutput>BZ_OK</computeroutput> and the number of bytes
1780
read is returned. If the logical end-of-stream was detected,
1781
<computeroutput>bzerror</computeroutput> will be set to
1782
<computeroutput>BZ_STREAM_END</computeroutput>, and the number of
1783
bytes read is returned. All other
1784
<computeroutput>bzerror</computeroutput> values denote an
1787
<para><computeroutput>BZ2_bzRead</computeroutput> will supply
1788
<computeroutput>len</computeroutput> bytes, unless the logical
1789
stream end is detected or an error occurs. Because of this, it
1790
is possible to detect the stream end by observing when the number
1791
of bytes returned is less than the number requested.
1792
Nevertheless, this is regarded as inadvisable; you should instead
1793
check <computeroutput>bzerror</computeroutput> after every call
1795
<computeroutput>BZ_STREAM_END</computeroutput>.</para>
1797
<para>Internally, <computeroutput>BZ2_bzRead</computeroutput>
1798
copies data from the compressed file in chunks of size
1799
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes before
1800
decompressing it. If the file contains more bytes than strictly
1801
needed to reach the logical end-of-stream,
1802
<computeroutput>BZ2_bzRead</computeroutput> will almost certainly
1803
read some of the trailing data before signalling
1804
<computeroutput>BZ_SEQUENCE_END</computeroutput>. To collect the
1805
read but unused data once
1806
<computeroutput>BZ_SEQUENCE_END</computeroutput> has appeared,
1807
call <computeroutput>BZ2_bzReadGetUnused</computeroutput>
1809
<computeroutput>BZ2_bzReadClose</computeroutput>.</para>
1811
<para>Possible assignments to
1812
<computeroutput>bzerror</computeroutput>:</para>
1816
if b is NULL or buf is NULL or len < 0
1818
if b was opened with BZ2_bzWriteOpen
1820
if there is an error reading from the compressed file
1822
if the compressed file ended before
1823
the logical end-of-stream was detected
1825
if a data integrity error was detected in the compressed stream
1827
if the stream does not begin with the requisite header bytes
1828
(ie, is not a bzip2 data file). This is really
1829
a special case of BZ_DATA_ERROR.
1831
if insufficient memory was available
1833
if the logical end of stream was detected.
1838
<para>Possible return values:</para>
1841
number of bytes read
1842
if bzerror is BZ_OK or BZ_STREAM_END
1847
<para>Allowable next actions:</para>
1850
collect data from buf, then BZ2_bzRead or BZ2_bzReadClose
1852
collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused
1853
if bzerror is BZ_SEQUENCE_END
1861
<sect2 id="bzreadgetunused" xreflabel="BZ2_bzReadGetUnused">
1862
<title>BZ2_bzReadGetUnused</title>
1865
void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b,
1866
void** unused, int* nUnused );
1869
<para>Returns data which was read from the compressed file but
1870
was not needed to get to the logical end-of-stream.
1871
<computeroutput>*unused</computeroutput> is set to the address of
1872
the data, and <computeroutput>*nUnused</computeroutput> to the
1873
number of bytes. <computeroutput>*nUnused</computeroutput> will
1874
be set to a value between <computeroutput>0</computeroutput> and
1875
<computeroutput>BZ_MAX_UNUSED</computeroutput> inclusive.</para>
1877
<para>This function may only be called once
1878
<computeroutput>BZ2_bzRead</computeroutput> has signalled
1879
<computeroutput>BZ_STREAM_END</computeroutput> but before
1880
<computeroutput>BZ2_bzReadClose</computeroutput>.</para>
1882
<para>Possible assignments to
1883
<computeroutput>bzerror</computeroutput>:</para>
1888
or unused is NULL or nUnused is NULL
1890
if BZ_STREAM_END has not been signalled
1891
or if b was opened with BZ2_bzWriteOpen
1896
<para>Allowable next actions:</para>
1905
<sect2 id="bzreadclose" xreflabel="BZ2_bzReadClose">
1906
<title>BZ2_bzReadClose</title>
1909
void BZ2_bzReadClose ( int *bzerror, BZFILE *b );
1912
<para>Releases all memory pertaining to the compressed file
1913
<computeroutput>b</computeroutput>.
1914
<computeroutput>BZ2_bzReadClose</computeroutput> does not call
1915
<computeroutput>fclose</computeroutput> on the underlying file
1916
handle, so you should do that yourself if appropriate.
1917
<computeroutput>BZ2_bzReadClose</computeroutput> should be called
1918
to clean up after all error situations.</para>
1920
<para>Possible assignments to
1921
<computeroutput>bzerror</computeroutput>:</para>
1925
if b was opened with BZ2_bzOpenWrite
1930
<para>Allowable next actions:</para>
1939
<sect2 id="bzwriteopen" xreflabel="BZ2_bzWriteOpen">
1940
<title>BZ2_bzWriteOpen</title>
1943
BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f,
1944
int blockSize100k, int verbosity,
1948
<para>Prepare to write compressed data to file handle
1949
<computeroutput>f</computeroutput>.
1950
<computeroutput>f</computeroutput> should refer to a file which
1951
has been opened for writing, and for which the error indicator
1952
(<computeroutput>ferror(f)</computeroutput>)is not set.</para>
1954
<para>For the meaning of parameters
1955
<computeroutput>blockSize100k</computeroutput>,
1956
<computeroutput>verbosity</computeroutput> and
1957
<computeroutput>workFactor</computeroutput>, see
1958
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
1960
<para>All required memory is allocated at this stage, so if the
1961
call completes successfully,
1962
<computeroutput>BZ_MEM_ERROR</computeroutput> cannot be signalled
1963
by a subsequent call to
1964
<computeroutput>BZ2_bzWrite</computeroutput>.</para>
1966
<para>Possible assignments to
1967
<computeroutput>bzerror</computeroutput>:</para>
1971
if the library has been mis-compiled
1974
or blockSize100k < 1 or blockSize100k > 9
1976
if ferror(f) is nonzero
1978
if insufficient memory is available
1983
<para>Possible return values:</para>
1986
Pointer to an abstract BZFILE
1992
<para>Allowable next actions:</para>
1997
(you could go directly to BZ2_bzWriteClose, but this would be pretty pointless)
2005
<sect2 id="bzwrite" xreflabel="BZ2_bzWrite">
2006
<title>BZ2_bzWrite</title>
2009
void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );
2012
<para>Absorbs <computeroutput>len</computeroutput> bytes from the
2013
buffer <computeroutput>buf</computeroutput>, eventually to be
2014
compressed and written to the file.</para>
2016
<para>Possible assignments to
2017
<computeroutput>bzerror</computeroutput>:</para>
2021
if b is NULL or buf is NULL or len < 0
2023
if b was opened with BZ2_bzReadOpen
2025
if there is an error writing the compressed file.
2033
<sect2 id="bzwriteclose" xreflabel="BZ2_bzWriteClose">
2034
<title>BZ2_bzWriteClose</title>
2037
void BZ2_bzWriteClose( int *bzerror, BZFILE* f,
2039
unsigned int* nbytes_in,
2040
unsigned int* nbytes_out );
2042
void BZ2_bzWriteClose64( int *bzerror, BZFILE* f,
2044
unsigned int* nbytes_in_lo32,
2045
unsigned int* nbytes_in_hi32,
2046
unsigned int* nbytes_out_lo32,
2047
unsigned int* nbytes_out_hi32 );
2050
<para>Compresses and flushes to the compressed file all data so
2051
far supplied by <computeroutput>BZ2_bzWrite</computeroutput>.
2052
The logical end-of-stream markers are also written, so subsequent
2053
calls to <computeroutput>BZ2_bzWrite</computeroutput> are
2054
illegal. All memory associated with the compressed file
2055
<computeroutput>b</computeroutput> is released.
2056
<computeroutput>fflush</computeroutput> is called on the
2057
compressed file, but it is not
2058
<computeroutput>fclose</computeroutput>'d.</para>
2060
<para>If <computeroutput>BZ2_bzWriteClose</computeroutput> is
2061
called to clean up after an error, the only action is to release
2062
the memory. The library records the error codes issued by
2063
previous calls, so this situation will be detected automatically.
2064
There is no attempt to complete the compression operation, nor to
2065
<computeroutput>fflush</computeroutput> the compressed file. You
2066
can force this behaviour to happen even in the case of no error,
2067
by passing a nonzero value to
2068
<computeroutput>abandon</computeroutput>.</para>
2070
<para>If <computeroutput>nbytes_in</computeroutput> is non-null,
2071
<computeroutput>*nbytes_in</computeroutput> will be set to be the
2072
total volume of uncompressed data handled. Similarly,
2073
<computeroutput>nbytes_out</computeroutput> will be set to the
2074
total volume of compressed data written. For compatibility with
2075
older versions of the library,
2076
<computeroutput>BZ2_bzWriteClose</computeroutput> only yields the
2077
lower 32 bits of these counts. Use
2078
<computeroutput>BZ2_bzWriteClose64</computeroutput> if you want
2079
the full 64 bit counts. These two functions are otherwise
2080
absolutely identical.</para>
2082
<para>Possible assignments to
2083
<computeroutput>bzerror</computeroutput>:</para>
2087
if b was opened with BZ2_bzReadOpen
2089
if there is an error writing the compressed file
2097
<sect2 id="embed" xreflabel="Handling embedded compressed data streams">
2098
<title>Handling embedded compressed data streams</title>
2100
<para>The high-level library facilitates use of
2101
<computeroutput>bzip2</computeroutput> data streams which form
2102
some part of a surrounding, larger data stream.</para>
2104
<itemizedlist mark='bullet'>
2106
<listitem><para>For writing, the library takes an open file handle,
2107
writes compressed data to it,
2108
<computeroutput>fflush</computeroutput>es it but does not
2109
<computeroutput>fclose</computeroutput> it. The calling
2110
application can write its own data before and after the
2111
compressed data stream, using that same file handle.</para></listitem>
2113
<listitem><para>Reading is more complex, and the facilities are not as
2114
general as they could be since generality is hard to reconcile
2115
with efficiency. <computeroutput>BZ2_bzRead</computeroutput>
2116
reads from the compressed file in blocks of size
2117
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes, and in
2118
doing so probably will overshoot the logical end of compressed
2119
stream. To recover this data once decompression has ended,
2120
call <computeroutput>BZ2_bzReadGetUnused</computeroutput> after
2121
the last call of <computeroutput>BZ2_bzRead</computeroutput>
2123
<computeroutput>BZ_STREAM_END</computeroutput>) but before
2125
<computeroutput>BZ2_bzReadClose</computeroutput>.</para></listitem>
2129
<para>This mechanism makes it easy to decompress multiple
2130
<computeroutput>bzip2</computeroutput> streams placed end-to-end.
2131
As the end of one stream, when
2132
<computeroutput>BZ2_bzRead</computeroutput> returns
2133
<computeroutput>BZ_STREAM_END</computeroutput>, call
2134
<computeroutput>BZ2_bzReadGetUnused</computeroutput> to collect
2135
the unused data (copy it into your own buffer somewhere). That
2136
data forms the start of the next compressed stream. To start
2137
uncompressing that next stream, call
2138
<computeroutput>BZ2_bzReadOpen</computeroutput> again, feeding in
2139
the unused data via the <computeroutput>unused</computeroutput> /
2140
<computeroutput>nUnused</computeroutput> parameters. Keep doing
2141
this until <computeroutput>BZ_STREAM_END</computeroutput> return
2142
coincides with the physical end of file
2143
(<computeroutput>feof(f)</computeroutput>). In this situation
2144
<computeroutput>BZ2_bzReadGetUnused</computeroutput> will of
2145
course return no data.</para>
2147
<para>This should give some feel for how the high-level interface
2148
can be used. If you require extra flexibility, you'll have to
2149
bite the bullet and get to grips with the low-level
2155
<sect2 id="std-rdwr" xreflabel="Standard file-reading/writing code">
2156
<title>Standard file-reading/writing code</title>
2158
<para>Here's how you'd write data to a compressed file:</para>
2164
char buf[ /* whatever size you like */ ];
2168
f = fopen ( "myfile.bz2", "w" );
2172
b = BZ2_bzWriteOpen( &bzerror, f, 9 );
2173
if (bzerror != BZ_OK) {
2174
BZ2_bzWriteClose ( b );
2178
while ( /* condition */ ) {
2179
/* get data to write into buf, and set nBuf appropriately */
2180
nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf );
2181
if (bzerror == BZ_IO_ERROR) {
2182
BZ2_bzWriteClose ( &bzerror, b );
2187
BZ2_bzWriteClose( &bzerror, b );
2188
if (bzerror == BZ_IO_ERROR) {
2193
<para>And to read from a compressed file:</para>
2199
char buf[ /* whatever size you like */ ];
2203
f = fopen ( "myfile.bz2", "r" );
2207
b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 );
2208
if ( bzerror != BZ_OK ) {
2209
BZ2_bzReadClose ( &bzerror, b );
2214
while ( bzerror == BZ_OK && /* arbitrary other conditions */) {
2215
nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ );
2216
if ( bzerror == BZ_OK ) {
2217
/* do something with buf[0 .. nBuf-1] */
2220
if ( bzerror != BZ_STREAM_END ) {
2221
BZ2_bzReadClose ( &bzerror, b );
2224
BZ2_bzReadClose ( &bzerror, b );
2233
<sect1 id="util-fns" xreflabel="Utility functions">
2234
<title>Utility functions</title>
2237
<sect2 id="bzbufftobuffcompress" xreflabel="BZ2_bzBuffToBuffCompress">
2238
<title>BZ2_bzBuffToBuffCompress</title>
2241
int BZ2_bzBuffToBuffCompress( char* dest,
2242
unsigned int* destLen,
2244
unsigned int sourceLen,
2250
<para>Attempts to compress the data in <computeroutput>source[0
2251
.. sourceLen-1]</computeroutput> into the destination buffer,
2252
<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the
2253
destination buffer is big enough,
2254
<computeroutput>*destLen</computeroutput> is set to the size of
2255
the compressed data, and <computeroutput>BZ_OK</computeroutput>
2256
is returned. If the compressed data won't fit,
2257
<computeroutput>*destLen</computeroutput> is unchanged, and
2258
<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
2261
<para>Compression in this manner is a one-shot event, done with a
2262
single call to this function. The resulting compressed data is a
2263
complete <computeroutput>bzip2</computeroutput> format data
2264
stream. There is no mechanism for making additional calls to
2265
provide extra input data. If you want that kind of mechanism,
2266
use the low-level interface.</para>
2268
<para>For the meaning of parameters
2269
<computeroutput>blockSize100k</computeroutput>,
2270
<computeroutput>verbosity</computeroutput> and
2271
<computeroutput>workFactor</computeroutput>, see
2272
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
2274
<para>To guarantee that the compressed data will fit in its
2275
buffer, allocate an output buffer of size 1% larger than the
2276
uncompressed data, plus six hundred extra bytes.</para>
2278
<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
2279
will not write data at or beyond
2280
<computeroutput>dest[*destLen]</computeroutput>, even in case of
2281
buffer overflow.</para>
2283
<para>Possible return values:</para>
2287
if the library has been mis-compiled
2289
if dest is NULL or destLen is NULL
2290
or blockSize100k < 1 or blockSize100k > 9
2291
or verbosity < 0 or verbosity > 4
2292
or workFactor < 0 or workFactor > 250
2294
if insufficient memory is available
2296
if the size of the compressed data exceeds *destLen
2304
<sect2 id="bzbufftobuffdecompress" xreflabel="BZ2_bzBuffToBuffDecompress">
2305
<title>BZ2_bzBuffToBuffDecompress</title>
2308
int BZ2_bzBuffToBuffDecompress( char* dest,
2309
unsigned int* destLen,
2311
unsigned int sourceLen,
2316
<para>Attempts to decompress the data in <computeroutput>source[0
2317
.. sourceLen-1]</computeroutput> into the destination buffer,
2318
<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the
2319
destination buffer is big enough,
2320
<computeroutput>*destLen</computeroutput> is set to the size of
2321
the uncompressed data, and <computeroutput>BZ_OK</computeroutput>
2322
is returned. If the compressed data won't fit,
2323
<computeroutput>*destLen</computeroutput> is unchanged, and
2324
<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
2327
<para><computeroutput>source</computeroutput> is assumed to hold
2328
a complete <computeroutput>bzip2</computeroutput> format data
2330
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> tries
2331
to decompress the entirety of the stream into the output
2334
<para>For the meaning of parameters
2335
<computeroutput>small</computeroutput> and
2336
<computeroutput>verbosity</computeroutput>, see
2337
<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
2339
<para>Because the compression ratio of the compressed data cannot
2340
be known in advance, there is no easy way to guarantee that the
2341
output buffer will be big enough. You may of course make
2342
arrangements in your code to record the size of the uncompressed
2343
data, but such a mechanism is beyond the scope of this
2346
<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
2347
will not write data at or beyond
2348
<computeroutput>dest[*destLen]</computeroutput>, even in case of
2349
buffer overflow.</para>
2351
<para>Possible return values:</para>
2355
if the library has been mis-compiled
2357
if dest is NULL or destLen is NULL
2358
or small != 0 && small != 1
2359
or verbosity < 0 or verbosity > 4
2361
if insufficient memory is available
2363
if the size of the compressed data exceeds *destLen
2365
if a data integrity error was detected in the compressed data
2367
if the compressed data doesn't begin with the right magic bytes
2369
if the compressed data ends unexpectedly
2379
<sect1 id="zlib-compat" xreflabel="zlib compatibility functions">
2380
<title>zlib compatibility functions</title>
2382
<para>Yoshioka Tsuneo has contributed some functions to give
2383
better <computeroutput>zlib</computeroutput> compatibility.
2384
These functions are <computeroutput>BZ2_bzopen</computeroutput>,
2385
<computeroutput>BZ2_bzread</computeroutput>,
2386
<computeroutput>BZ2_bzwrite</computeroutput>,
2387
<computeroutput>BZ2_bzflush</computeroutput>,
2388
<computeroutput>BZ2_bzclose</computeroutput>,
2389
<computeroutput>BZ2_bzerror</computeroutput> and
2390
<computeroutput>BZ2_bzlibVersion</computeroutput>. These
2391
functions are not (yet) officially part of the library. If they
2392
break, you get to keep all the pieces. Nevertheless, I think
2393
they work ok.</para>
2396
typedef void BZFILE;
2398
const char * BZ2_bzlibVersion ( void );
2401
<para>Returns a string indicating the library version.</para>
2404
BZFILE * BZ2_bzopen ( const char *path, const char *mode );
2405
BZFILE * BZ2_bzdopen ( int fd, const char *mode );
2408
<para>Opens a <computeroutput>.bz2</computeroutput> file for
2409
reading or writing, using either its name or a pre-existing file
2410
descriptor. Analogous to <computeroutput>fopen</computeroutput>
2411
and <computeroutput>fdopen</computeroutput>.</para>
2414
int BZ2_bzread ( BZFILE* b, void* buf, int len );
2415
int BZ2_bzwrite ( BZFILE* b, void* buf, int len );
2418
<para>Reads/writes data from/to a previously opened
2419
<computeroutput>BZFILE</computeroutput>. Analogous to
2420
<computeroutput>fread</computeroutput> and
2421
<computeroutput>fwrite</computeroutput>.</para>
2424
int BZ2_bzflush ( BZFILE* b );
2425
void BZ2_bzclose ( BZFILE* b );
2428
<para>Flushes/closes a <computeroutput>BZFILE</computeroutput>.
2429
<computeroutput>BZ2_bzflush</computeroutput> doesn't actually do
2430
anything. Analogous to <computeroutput>fflush</computeroutput>
2431
and <computeroutput>fclose</computeroutput>.</para>
2434
const char * BZ2_bzerror ( BZFILE *b, int *errnum )
2437
<para>Returns a string describing the more recent error status of
2438
<computeroutput>b</computeroutput>, and also sets
2439
<computeroutput>*errnum</computeroutput> to its numerical
2445
<sect1 id="stdio-free"
2446
xreflabel="Using the library in a stdio-free environment">
2447
<title>Using the library in a stdio-free environment</title>
2450
<sect2 id="stdio-bye" xreflabel="Getting rid of stdio">
2451
<title>Getting rid of stdio</title>
2453
<para>In a deeply embedded application, you might want to use
2454
just the memory-to-memory functions. You can do this
2455
conveniently by compiling the library with preprocessor symbol
2456
<computeroutput>BZ_NO_STDIO</computeroutput> defined. Doing this
2457
gives you a library containing only the following eight
2460
<para><computeroutput>BZ2_bzCompressInit</computeroutput>,
2461
<computeroutput>BZ2_bzCompress</computeroutput>,
2462
<computeroutput>BZ2_bzCompressEnd</computeroutput>
2463
<computeroutput>BZ2_bzDecompressInit</computeroutput>,
2464
<computeroutput>BZ2_bzDecompress</computeroutput>,
2465
<computeroutput>BZ2_bzDecompressEnd</computeroutput>
2466
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput>,
2467
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></para>
2469
<para>When compiled like this, all functions will ignore
2470
<computeroutput>verbosity</computeroutput> settings.</para>
2475
<sect2 id="critical-error" xreflabel="Critical error handling">
2476
<title>Critical error handling</title>
2478
<para><computeroutput>libbzip2</computeroutput> contains a number
2479
of internal assertion checks which should, needless to say, never
2480
be activated. Nevertheless, if an assertion should fail,
2481
behaviour depends on whether or not the library was compiled with
2482
<computeroutput>BZ_NO_STDIO</computeroutput> set.</para>
2484
<para>For a normal compile, an assertion failure yields the
2488
<para>bzip2/libbzip2: internal error number N.</para>
2489
<para>This is a bug in bzip2/libbzip2, &bz-version; of &bz-date;.
2490
Please report it to me at: &bz-email;. If this happened
2491
when you were using some program which uses libbzip2 as a
2492
component, you should also report this bug to the author(s)
2493
of that program. Please make an effort to report this bug;
2494
timely and accurate bug reports eventually lead to higher
2495
quality software. Thanks. Julian Seward, &bz-date;.
2496
</para></blockquote>
2498
<para>where <computeroutput>N</computeroutput> is some error code
2499
number. If <computeroutput>N == 1007</computeroutput>, it also
2500
prints some extra text advising the reader that unreliable memory
2501
is often associated with internal error 1007. (This is a
2502
frequently-observed-phenomenon with versions 1.0.0/1.0.1).</para>
2504
<para><computeroutput>exit(3)</computeroutput> is then
2507
<para>For a <computeroutput>stdio</computeroutput>-free library,
2508
assertion failures result in a call to a function declared
2512
extern void bz_internal_error ( int errcode );
2515
<para>The relevant code is passed as a parameter. You should
2516
supply such a function.</para>
2518
<para>In either case, once an assertion failure has occurred, any
2519
<computeroutput>bz_stream</computeroutput> records involved can
2520
be regarded as invalid. You should not attempt to resume normal
2521
operation with them.</para>
2523
<para>You may, of course, change critical error handling to suit
2524
your needs. As I said above, critical errors indicate bugs in
2525
the library and should not occur. All "normal" error situations
2526
are indicated via error return codes from functions, and can be
2527
recovered from.</para>
2534
<sect1 id="win-dll" xreflabel="Making a Windows DLL">
2535
<title>Making a Windows DLL</title>
2537
<para>Everything related to Windows has been contributed by
2539
(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>), so
2540
you should send your queries to him (but perhaps Cc: me,
2541
<computeroutput>&bz-email;</computeroutput>).</para>
2543
<para>My vague understanding of what to do is: using Visual C++
2544
5.0, open the project file
2545
<computeroutput>libbz2.dsp</computeroutput>, and build. That's
2548
<para>If you can't open the project file for some reason, make a
2549
new one, naming these files:
2550
<computeroutput>blocksort.c</computeroutput>,
2551
<computeroutput>bzlib.c</computeroutput>,
2552
<computeroutput>compress.c</computeroutput>,
2553
<computeroutput>crctable.c</computeroutput>,
2554
<computeroutput>decompress.c</computeroutput>,
2555
<computeroutput>huffman.c</computeroutput>,
2556
<computeroutput>randtable.c</computeroutput> and
2557
<computeroutput>libbz2.def</computeroutput>. You will also need
2558
to name the header files <computeroutput>bzlib.h</computeroutput>
2559
and <computeroutput>bzlib_private.h</computeroutput>.</para>
2561
<para>If you don't use VC++, you may need to define the
2563
<computeroutput>_WIN32</computeroutput>.</para>
2565
<para>Finally, <computeroutput>dlltest.c</computeroutput> is a
2566
sample program using the DLL. It has a project file,
2567
<computeroutput>dlltest.dsp</computeroutput>.</para>
2569
<para>If you just want a makefile for Visual C, have a look at
2570
<computeroutput>makefile.msc</computeroutput>.</para>
2572
<para>Be aware that if you compile
2573
<computeroutput>bzip2</computeroutput> itself on Win32, you must
2574
set <computeroutput>BZ_UNIX</computeroutput> to 0 and
2575
<computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the file
2576
<computeroutput>bzip2.c</computeroutput>, before compiling.
2577
Otherwise the resulting binary won't work correctly.</para>
2579
<para>I haven't tried any of this stuff myself, but it all looks
2588
<chapter id="misc" xreflabel="Miscellanea">
2589
<title>Miscellanea</title>
2591
<para>These are just some random thoughts of mine. Your mileage
2595
<sect1 id="limits" xreflabel="Limitations of the compressed file format">
2596
<title>Limitations of the compressed file format</title>
2598
<para><computeroutput>bzip2-1.0.X</computeroutput>,
2599
<computeroutput>0.9.5</computeroutput> and
2600
<computeroutput>0.9.0</computeroutput> use exactly the same file
2601
format as the original version,
2602
<computeroutput>bzip2-0.1</computeroutput>. This decision was
2603
made in the interests of stability. Creating yet another
2604
incompatible compressed file format would create further
2605
confusion and disruption for users.</para>
2607
<para>Nevertheless, this is not a painless decision. Development
2608
work since the release of
2609
<computeroutput>bzip2-0.1</computeroutput> in August 1997 has
2610
shown complexities in the file format which slow down
2611
decompression and, in retrospect, are unnecessary. These
2614
<itemizedlist mark='bullet'>
2616
<listitem><para>The run-length encoder, which is the first of the
2617
compression transformations, is entirely irrelevant. The
2618
original purpose was to protect the sorting algorithm from the
2619
very worst case input: a string of repeated symbols. But
2620
algorithm steps Q6a and Q6b in the original Burrows-Wheeler
2621
technical report (SRC-124) show how repeats can be handled
2622
without difficulty in block sorting.</para></listitem>
2624
<listitem><para>The randomisation mechanism doesn't really need to be
2625
there. Udi Manber and Gene Myers published a suffix array
2626
construction algorithm a few years back, which can be employed
2627
to sort any block, no matter how repetitive, in O(N log N)
2628
time. Subsequent work by Kunihiko Sadakane has produced a
2629
derivative O(N (log N)^2) algorithm which usually outperforms
2630
the Manber-Myers algorithm.</para>
2632
<para>I could have changed to Sadakane's algorithm, but I find
2633
it to be slower than <computeroutput>bzip2</computeroutput>'s
2634
existing algorithm for most inputs, and the randomisation
2635
mechanism protects adequately against bad cases. I didn't
2636
think it was a good tradeoff to make. Partly this is due to
2637
the fact that I was not flooded with email complaints about
2638
<computeroutput>bzip2-0.1</computeroutput>'s performance on
2639
repetitive data, so perhaps it isn't a problem for real
2642
<para>Probably the best long-term solution, and the one I have
2643
incorporated into 0.9.5 and above, is to use the existing
2644
sorting algorithm initially, and fall back to a O(N (log N)^2)
2645
algorithm if the standard algorithm gets into
2646
difficulties.</para></listitem>
2648
<listitem><para>The compressed file format was never designed to be
2649
handled by a library, and I have had to jump though some hoops
2650
to produce an efficient implementation of decompression. It's
2651
a bit hairy. Try passing
2652
<computeroutput>decompress.c</computeroutput> through the C
2653
preprocessor and you'll see what I mean. Much of this
2654
complexity could have been avoided if the compressed size of
2655
each block of data was recorded in the data stream.</para></listitem>
2657
<listitem><para>An Adler-32 checksum, rather than a CRC32 checksum,
2658
would be faster to compute.</para></listitem>
2662
<para>It would be fair to say that the
2663
<computeroutput>bzip2</computeroutput> format was frozen before I
2664
properly and fully understood the performance consequences of
2667
<para>Improvements which I was able to incorporate into 0.9.0,
2668
despite using the same file format, are:</para>
2670
<itemizedlist mark='bullet'>
2672
<listitem><para>Single array implementation of the inverse BWT. This
2673
significantly speeds up decompression, presumably because it
2674
reduces the number of cache misses.</para></listitem>
2676
<listitem><para>Faster inverse MTF transform for large MTF values.
2677
The new implementation is based on the notion of sliding blocks
2678
of values.</para></listitem>
2680
<listitem><para><computeroutput>bzip2-0.9.0</computeroutput> now reads
2681
and writes files with <computeroutput>fread</computeroutput>
2682
and <computeroutput>fwrite</computeroutput>; version 0.1 used
2683
<computeroutput>putc</computeroutput> and
2684
<computeroutput>getc</computeroutput>. Duh! Well, you live
2685
and learn.</para></listitem>
2689
<para>Further ahead, it would be nice to be able to do random
2690
access into files. This will require some careful design of
2691
compressed file formats.</para>
2696
<sect1 id="port-issues" xreflabel="Portability issues">
2697
<title>Portability issues</title>
2699
<para>After some consideration, I have decided not to use GNU
2700
<computeroutput>autoconf</computeroutput> to configure 0.9.5 or
2703
<para><computeroutput>autoconf</computeroutput>, admirable and
2704
wonderful though it is, mainly assists with portability problems
2705
between Unix-like platforms. But
2706
<computeroutput>bzip2</computeroutput> doesn't have much in the
2707
way of portability problems on Unix; most of the difficulties
2708
appear when porting to the Mac, or to Microsoft's operating
2709
systems. <computeroutput>autoconf</computeroutput> doesn't help
2710
in those cases, and brings in a whole load of new
2713
<para>Most people should be able to compile the library and
2714
program under Unix straight out-of-the-box, so to speak,
2715
especially if you have a version of GNU C available.</para>
2717
<para>There are a couple of
2718
<computeroutput>__inline__</computeroutput> directives in the
2719
code. GNU C (<computeroutput>gcc</computeroutput>) should be
2720
able to handle them. If you're not using GNU C, your C compiler
2721
shouldn't see them at all. If your compiler does, for some
2722
reason, see them and doesn't like them, just
2723
<computeroutput>#define</computeroutput>
2724
<computeroutput>__inline__</computeroutput> to be
2725
<computeroutput>/* */</computeroutput>. One easy way to do this
2726
is to compile with the flag
2727
<computeroutput>-D__inline__=</computeroutput>, which should be
2728
understood by most Unix compilers.</para>
2730
<para>If you still have difficulties, try compiling with the
2731
macro <computeroutput>BZ_STRICT_ANSI</computeroutput> defined.
2732
This should enable you to build the library in a strictly ANSI
2733
compliant environment. Building the program itself like this is
2734
dangerous and not supported, since you remove
2735
<computeroutput>bzip2</computeroutput>'s checks against
2736
compressing directories, symbolic links, devices, and other
2737
not-really-a-file entities. This could cause filesystem
2740
<para>One other thing: if you create a
2741
<computeroutput>bzip2</computeroutput> binary for public distribution,
2742
please consider linking it statically (<computeroutput>gcc
2743
-static</computeroutput>). This avoids all sorts of library-version
2744
issues that others may encounter later on.</para>
2746
<para>If you build <computeroutput>bzip2</computeroutput> on
2747
Win32, you must set <computeroutput>BZ_UNIX</computeroutput> to 0
2748
and <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the
2749
file <computeroutput>bzip2.c</computeroutput>, before compiling.
2750
Otherwise the resulting binary won't work correctly.</para>
2755
<sect1 id="bugs" xreflabel="Reporting bugs">
2756
<title>Reporting bugs</title>
2758
<para>I tried pretty hard to make sure
2759
<computeroutput>bzip2</computeroutput> is bug free, both by
2760
design and by testing. Hopefully you'll never need to read this
2761
section for real.</para>
2763
<para>Nevertheless, if <computeroutput>bzip2</computeroutput> dies
2764
with a segmentation fault, a bus error or an internal assertion
2765
failure, it will ask you to email me a bug report. Experience from
2766
years of feedback of bzip2 users indicates that almost all these
2767
problems can be traced to either compiler bugs or hardware
2770
<itemizedlist mark='bullet'>
2772
<listitem><para>Recompile the program with no optimisation, and
2773
see if it works. And/or try a different compiler. I heard all
2774
sorts of stories about various flavours of GNU C (and other
2775
compilers) generating bad code for
2776
<computeroutput>bzip2</computeroutput>, and I've run across two
2777
such examples myself.</para>
2779
<para>2.7.X versions of GNU C are known to generate bad code
2780
from time to time, at high optimisation levels. If you get
2781
problems, try using the flags
2782
<computeroutput>-O2</computeroutput>
2783
<computeroutput>-fomit-frame-pointer</computeroutput>
2784
<computeroutput>-fno-strength-reduce</computeroutput>. You
2785
should specifically <emphasis>not</emphasis> use
2786
<computeroutput>-funroll-loops</computeroutput>.</para>
2788
<para>You may notice that the Makefile runs six tests as part
2789
of the build process. If the program passes all of these, it's
2790
a pretty good (but not 100%) indication that the compiler has
2791
done its job correctly.</para></listitem>
2793
<listitem><para>If <computeroutput>bzip2</computeroutput>
2794
crashes randomly, and the crashes are not repeatable, you may
2795
have a flaky memory subsystem.
2796
<computeroutput>bzip2</computeroutput> really hammers your
2797
memory hierarchy, and if it's a bit marginal, you may get these
2798
problems. Ditto if your disk or I/O subsystem is slowly
2799
failing. Yup, this really does happen.</para>
2801
<para>Try using a different machine of the same type, and see
2802
if you can repeat the problem.</para></listitem>
2804
<listitem><para>This isn't really a bug, but ... If
2805
<computeroutput>bzip2</computeroutput> tells you your file is
2806
corrupted on decompression, and you obtained the file via FTP,
2807
there is a possibility that you forgot to tell FTP to do a
2808
binary mode transfer. That absolutely will cause the file to
2809
be non-decompressible. You'll have to transfer it
2810
again.</para></listitem>
2814
<para>If you've incorporated
2815
<computeroutput>libbzip2</computeroutput> into your own program
2816
and are getting problems, please, please, please, check that the
2817
parameters you are passing in calls to the library, are correct,
2818
and in accordance with what the documentation says is allowable.
2819
I have tried to make the library robust against such problems,
2820
but I'm sure I haven't succeeded.</para>
2822
<para>Finally, if the above comments don't help, you'll have to
2823
send me a bug report. Now, it's just amazing how many people
2824
will send me a bug report saying something like:</para>
2827
bzip2 crashed with segmentation fault on my machine
2830
<para>and absolutely nothing else. Needless to say, a such a
2831
report is <emphasis>totally, utterly, completely and
2832
comprehensively 100% useless; a waste of your time, my time, and
2833
net bandwidth</emphasis>. With no details at all, there's no way
2834
I can possibly begin to figure out what the problem is.</para>
2836
<para>The rules of the game are: facts, facts, facts. Don't omit
2837
them because "oh, they won't be relevant". At the bare
2841
Machine type. Operating system version.
2842
Exact version of bzip2 (do bzip2 -V).
2843
Exact version of the compiler used.
2844
Flags passed to the compiler.
2847
<para>However, the most important single thing that will help me
2848
is the file that you were trying to compress or decompress at the
2849
time the problem happened. Without that, my ability to do
2850
anything more than speculate about the cause, is limited.</para>
2855
<sect1 id="package" xreflabel="Did you get the right package?">
2856
<title>Did you get the right package?</title>
2858
<para><computeroutput>bzip2</computeroutput> is a resource hog.
2859
It soaks up large amounts of CPU cycles and memory. Also, it
2860
gives very large latencies. In the worst case, you can feed many
2861
megabytes of uncompressed data into the library before getting
2862
any compressed output, so this probably rules out applications
2863
requiring interactive behaviour.</para>
2865
<para>These aren't faults of my implementation, I hope, but more
2866
an intrinsic property of the Burrows-Wheeler transform
2867
(unfortunately). Maybe this isn't what you want.</para>
2869
<para>If you want a compressor and/or library which is faster,
2870
uses less memory but gets pretty good compression, and has
2871
minimal latency, consider Jean-loup Gailly's and Mark Adler's
2872
work, <computeroutput>zlib-1.2.1</computeroutput> and
2873
<computeroutput>gzip-1.2.4</computeroutput>. Look for them at
2874
<ulink url="http://www.zlib.org">http://www.zlib.org</ulink> and
2875
<ulink url="http://www.gzip.org">http://www.gzip.org</ulink>
2876
respectively.</para>
2878
<para>For something faster and lighter still, you might try Markus F
2879
X J Oberhumer's <computeroutput>LZO</computeroutput> real-time
2880
compression/decompression library, at
2881
<ulink url="http://www.oberhumer.com/opensource">http://www.oberhumer.com/opensource</ulink>.</para>
2887
<sect1 id="reading" xreflabel="Further Reading">
2888
<title>Further Reading</title>
2890
<para><computeroutput>bzip2</computeroutput> is not research
2891
work, in the sense that it doesn't present any new ideas.
2892
Rather, it's an engineering exercise based on existing
2895
<para>Four documents describe essentially all the ideas behind
2896
<computeroutput>bzip2</computeroutput>:</para>
2898
<literallayout>Michael Burrows and D. J. Wheeler:
2899
"A block-sorting lossless data compression algorithm"
2901
Digital SRC Research Report 124.
2902
ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz
2903
If you have trouble finding it, try searching at the
2904
New Zealand Digital Library, http://www.nzdl.org.
2906
Daniel S. Hirschberg and Debra A. LeLewer
2907
"Efficient Decoding of Prefix Codes"
2908
Communications of the ACM, April 1990, Vol 33, Number 4.
2909
You might be able to get an electronic copy of this
2910
from the ACM Digital Library.
2913
Program bred3.c and accompanying document bred3.ps.
2914
This contains the idea behind the multi-table Huffman coding scheme.
2915
ftp://ftp.cl.cam.ac.uk/users/djw3/
2917
Jon L. Bentley and Robert Sedgewick
2918
"Fast Algorithms for Sorting and Searching Strings"
2919
Available from Sedgewick's web page,
2920
www.cs.princeton.edu/~rs
2923
<para>The following paper gives valuable additional insights into
2924
the algorithm, but is not immediately the basis of any code used
2927
<literallayout>Peter Fenwick:
2928
Block Sorting Text Compression
2929
Proceedings of the 19th Australasian Computer Science Conference,
2930
Melbourne, Australia. Jan 31 - Feb 2, 1996.
2931
ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</literallayout>
2933
<para>Kunihiko Sadakane's sorting algorithm, mentioned above, is
2934
available from:</para>
2936
<literallayout>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
2939
<para>The Manber-Myers suffix array construction algorithm is
2940
described in a paper available from:</para>
2942
<literallayout>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
2945
<para>Finally, the following papers document some
2946
investigations I made into the performance of sorting
2947
and decompression algorithms:</para>
2949
<literallayout>Julian Seward
2950
On the Performance of BWT Sorting Algorithms
2951
Proceedings of the IEEE Data Compression Conference 2000
2952
Snowbird, Utah. 28-30 March 2000.
2955
Space-time Tradeoffs in the Inverse B-W Transform
2956
Proceedings of the IEEE Data Compression Conference 2001
2957
Snowbird, Utah. 27-29 March 2001.
added
removed
win32/3rdparty/bzip2/manual.xml
