~akopytov/percona-xtrabackup/parallel-compression

  • Committer: Alexey Kopytov
  • Date: 2012-02-10 20:05:56 UTC
  • Revision ID: akopytov@gmail.com-20120210200556-6kx41z8wwrqfucro
Rebase of the parallel compression patch on new trunk + post-review
fixes.

Implementation of parallel compression and streaming for XtraBackup.

This revision implements the following changes:

* InnoDB files are now streamed by the xtrabackup binary rather than
innobackupex. As a result, integrity is now verified by xtrabackup and
thus tar4ibd is no longer needed, so it was removed.

* xtrabackup binary now accepts the new '--stream' option which has
exactly the same semantics as the '--stream' option in
innobackupex: it tells xtrabackup to stream all files to the standard
output in the specified format rather than storing them locally.

* The xtrabackup binary can now do parallel compression using the
quicklz library. Two new options were added to xtrabackup to support
this feature:

- '--compress' tells xtrabackup to compress all output data, including
the transaction log file and meta data files, using the specified
compression algorithm. The only currently supported algorithm is
'quicklz'. The resulting files have the qpress archive format,
i.e. every *.qp file produced by xtrabackup is essentially a one-file
qpress archive and can be extracted and uncompressed by the qpress
file archiver (http://www.quicklz.com/).

- '--compress-threads' specifies the number of worker threads used by
xtrabackup for parallel data compression. This option defaults to 1.

Parallel compression ('--compress-threads') can be used together with
parallel file copying ('--parallel'). For example, '--parallel=4
--compress --compress-threads=2' will create 4 IO threads that will
read the data and pipe it to 2 compression threads.

* To support simultaneous compression and streaming, a new custom
streaming format called 'xbstream' was introduced to XtraBackup in
addition to the 'tar' format. That was required to overcome some
limitations of traditional archive formats such as 'tar', 'cpio' and
others that do not allow streaming dynamically generated files, for
example dynamically compressed files.  Other advantages of xbstream over
traditional streaming/archive formats include ability to stream multiple
files concurrently (so it is possible to use streaming in the xbstream
format together with the --parallel option) and more compact data
storage.

* To allow streaming and extracting files to/from the xbstream format
produced by xtrabackup, a new utility aptly called 'xbstream' was
added to the XtraBackup distribution. This utility has a tar-like
interface:

- with the '-x' option it extracts files from the stream read from its
standard input to the current directory unless specified otherwise
with the '-C' option.

- with the '-c' option it streams files specified on the command line
to its standard output.

The utility also tries to minimize its impact on the OS page cache by
using the appropriate posix_fadvise() calls when available.
Filename Latest Rev Last Changed Committer Comment Size
..
contrib 241.3.1 13 years ago Daniël van Eeden Add contrib directory with an backup script exampl Diff
doc 213.3.1 13 years ago Aleksandr Kuzminsky Added XtraBackup manual page Diff
patches 228.1.2 13 years ago Alexey Kopytov Port XtraBackup to Percona Server 5.5.9-beta20.1 c Diff
src 386 12 years ago Alexey Kopytov Rebase of the parallel compression patch on new tr Diff
test 134.1.1 13 years ago Aleksandr Kuzminsky Added the XtraBackup test framework Diff
utils 30.1.2 15 years ago Vadim Tkachenko directory restructure Diff
.bzrignore 290.1.2 12 years ago Rodrigo Gadea Added replication and small fixes 6 bytes Diff Download File
BUILD-WIN.txt 244.4.1 12 years ago Alexey Kopytov Added initial support for Windows builds: - remov 2.9 KB Diff Download File
BUILD.txt 300 12 years ago Stewart Smith merge backport of BUILd.txt fixes 2.7 KB Diff Download File
CMakeLists.txt 244.4.1 12 years ago Alexey Kopytov Added initial support for Windows builds: - remov 1.6 KB Diff Download File
COPYING 174.1.1 13 years ago Stewart Smith add COPYING file with GPLv2 license. xtrabackup is 18.6 KB Diff Download File
File innobackupex 386 12 years ago Alexey Kopytov Rebase of the parallel compression patch on new tr 95.1 KB Diff Download File
VERSION 379 12 years ago Alexey Kopytov Null-merged all missing 1.6 revisions after cherry 25 bytes Diff Download File