Rebase of the parallel compression patch on new trunk + post-review fixes.
Implementation of parallel compression and streaming for XtraBackup.
This revision implements the following changes:
* InnoDB files are now streamed by the xtrabackup binary rather than innobackupex. As a result, integrity is now verified by xtrabackup and thus tar4ibd is no longer needed, so it was removed.
* xtrabackup binary now accepts the new '--stream' option which has exactly the same semantics as the '--stream' option in innobackupex: it tells xtrabackup to stream all files to the standard output in the specified format rather than storing them locally.
* The xtrabackup binary can now do parallel compression using the quicklz library. Two new options were added to xtrabackup to support this feature:
- '--compress' tells xtrabackup to compress all output data, including the transaction log file and meta data files, using the specified compression algorithm. The only currently supported algorithm is 'quicklz'. The resulting files have the qpress archive format, i.e. every *.qp file produced by xtrabackup is essentially a one-file qpress archive and can be extracted and uncompressed by the qpress file archiver (http://www.quicklz.com/).
- '--compress-threads' specifies the number of worker threads used by xtrabackup for parallel data compression. This option defaults to 1.
Parallel compression ('--compress-threads') can be used together with parallel file copying ('--parallel'). For example, '--parallel=4 --compress --compress-threads=2' will create 4 IO threads that will read the data and pipe it to 2 compression threads.
* To support simultaneous compression and streaming, a new custom streaming format called 'xbstream' was introduced to XtraBackup in addition to the 'tar' format. That was required to overcome some limitations of traditional archive formats such as 'tar', 'cpio' and others that do not allow streaming dynamically generated files, for example dynamically compressed files. Other advantages of xbstream over traditional streaming/archive formats include ability to stream multiple files concurrently (so it is possible to use streaming in the xbstream format together with the --parallel option) and more compact data storage.
* To allow streaming and extracting files to/from the xbstream format produced by xtrabackup, a new utility aptly called 'xbstream' was added to the XtraBackup distribution. This utility has a tar-like interface:
- with the '-x' option it extracts files from the stream read from its standard input to the current directory unless specified otherwise with the '-C' option.
- with the '-c' option it streams files specified on the command line to its standard output.
The utility also tries to minimize its impact on the OS page cache by using the appropriate posix_fadvise() calls when available.