1
**********************************
3
**********************************
7
Database Quality Platform. Versatile, pluggable test runner for MySQL-based databases
9
**./kewpie** [ *OPTIONS* ] [ TESTCASE ]
14
:program:`kewpie.py` is intended to provide a standardized
15
platform to facilitate testing MySQL databases. Currently a number of MySQL variants are supported and include:
19
* MySQL using Galera replication
21
Designed as a response to the sometimes painfully diverse ecosystem of testing tools, the intent is to provide a system that standardizes common testing tasks while providing a variety of ways to analyze the behavior of database systems.
23
Long experience with MySQL testing has shown that a number of tasks are common to all tests:
24
* Allocation and management of test servers
25
* Test collection and management
26
* Reporting of test results
28
Kewpie combines this with the flexibity of Python's unittest framework. By allowing users to write tests as Python modules, we have great flexibility in how we start and set up servers, what we do to them, and how we assess their performance of our test tasks. Some of the tasks kewpie test cases cover:
30
* running sql-bench comprehensive tests
31
* running sql-bench crashme
33
* running drizzle-test-run test cases
34
* more direct tests using python code and helper libraries
37
=======================
39
* MySQL / Percona Server / Galera tests:
41
* randgen_basic - basic randgen tests of the server (optimizer stress, etc)
42
* randgen_bugs - holder suite for failing tests
43
* crashme - sql-bench's crashme suite (may take some time to run)
44
* sqlbench - sql-bench comprehensive suite. (may take ~45 min. to execute)
45
* cluster_basic - small, atomic tests of replication functionality. Tests are written in such a way as to be portable across server types (!)
46
* cluster_bugs - holder suite for failing tests
47
* cluster_randgen - tests of replication functionality using the random query generator and relevant test loads
48
* xtrabackup_basic - tests of the Percona Xtrabackup tool
49
* xtrabackup_bugs - hoder suite for failing tests
53
* randgen_basic - basic tests of the server (optimizer stress, etc)
54
* The following all use the same transaction tests, but validate different functionality:
56
* randgen_trxLog - tests the file-based replication log
57
* randgen_innoTrxLog - tests the innodb-table-based replication log
58
* randgen_slavePlugin - tests the functionality of the replication / slave plugin
60
* crashme - sql-bench's crashme suite (may take some time to run)
61
* sqlbench - sql-bench comprehensive suite. (may take ~45 min. to execute)
65
=========================
67
There are several different ways to run tests using :program:`kewpie.py`.
69
It should be noted that unless :option:`kewpie.py --force` is used, the program
70
will stop execution upon encountering the first failing test.
71
:option:`kewpie.py --force` is recommended if you are running several tests
72
- it will allow you to view all successes and failures in one run.
74
Running individual tests
75
------------------------
76
If one only wants to run a few, specific tests, they may do so this way::
78
./kewpie.py [OPTIONS] test1 [test2 ... testN]
80
Running all tests within a suite
81
--------------------------------
82
Many of the tests supplied with Drizzle are organized into suites.
84
The tests within drizzle/tests/t are considered the 'main' suite.
85
Other suites are located in either drizzle/tests/suite or within the various
86
directories in drizzle/plugin. Tests for a specific plugin should live in
87
the plugin's directory - drizzle/plugin/example_plugin/tests
89
To run the tests in a specific suite::
91
./kewpie.py [OPTIONS] --suite=SUITENAME
93
Running specific tests within a suite
94
--------------------------------------
95
To run a specific set of tests within a suite::
97
./kewpie.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
99
Calling tests using <suitename>.<testname> currently does not work. One must
100
specify the test suite via the :option:`kewpie.py --suite` option.
103
Running all available tests
104
---------------------------
105
Currently, the quickest way to execute all tests in all suites is
106
to use 'make test-kewpie' from the drizzle root.
108
Otherwise, one should simply name all suites::
110
./kewpie.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
112
Interpreting test results
113
=========================
114
The output of the test runner is quite simple. Every test should pass.
115
In the event of a test failure, please take the time to file a bug here:
116
*https://bugs.launchpad.net/drizzle*
118
During a run, the program will provide the user with:
119
* test name (suite + name)
120
* test status (pass/fail/skipped)
121
* time spent executing each test
123
At the end of a run, the program will provide the user with a listing of:
124
* how many tests were run
125
* counts and percentages of total exectuted for all test statuses
126
* a listing of failing, skipped, or disabled tests
127
* total time spent executing the tests
132
30 Jan 2011 16:26:31 : main.small_tmp_table [ pass ] 38
133
30 Jan 2011 16:26:31 : main.snowman [ pass ] 42
134
30 Jan 2011 16:26:31 : main.statement_boundaries [ pass ] 47
135
30 Jan 2011 16:26:31 : main.status [ pass ] 51
136
30 Jan 2011 16:26:31 : main.strict [ pass ] 138
137
30 Jan 2011 16:26:43 : main.subselect [ fail ] 12361
138
30 Jan 2011 16:26:43 : --- drizzle/tests/r/subselect.result 2011-01-30 16:23:54.975776148 -0500
139
30 Jan 2011 16:26:43 : +++ drizzle/tests/r/subselect.reject 2011-01-30 16:26:43.835519303 -0500
140
30 Jan 2011 16:26:43 : @@ -5,7 +5,7 @@
141
30 Jan 2011 16:26:43 : 2
142
30 Jan 2011 16:26:43 : explain extended select (select 2);
143
30 Jan 2011 16:26:43 : id select_type table type possible_keys key key_len ref rows filtered Extra
144
30 Jan 2011 16:26:43 : -9 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
145
30 Jan 2011 16:26:43 : +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
147
30 Jan 2011 16:30:20 : ================================================================================
148
30 Jan 2011 16:30:20 INFO: Test execution complete in 314 seconds
149
30 Jan 2011 16:30:20 INFO: Summary report:
150
30 Jan 2011 16:30:20 INFO: Executed 552/552 test cases, 100.00 percent
151
30 Jan 2011 16:30:20 INFO: STATUS: FAIL, 1/552 test cases, 0.18 percent executed
152
30 Jan 2011 16:30:20 INFO: STATUS: PASS, 551/552 test cases, 99.82 percent executed
153
30 Jan 2011 16:30:20 INFO: FAIL tests: main.subselect
154
30 Jan 2011 16:30:20 INFO: Spent 308 / 314 seconds on: TEST(s)
155
30 Jan 2011 16:30:20 INFO: Test execution complete
156
30 Jan 2011 16:30:20 INFO: Stopping all running servers...
161
Starting a server for manual testing
162
------------------------------------
164
:program:`kewpie.py` allows a user to get a Drizzle server up and running
165
quickly. This can be useful for fast ad-hoc testing.
169
./kewpie.py --start-and-exit [*OPTIONS*]
171
This will start a Drizzle server that you can connect to and query
173
Starting a server against a pre-populated DATADIR
174
--------------------------------------------------
176
Using :option:`kewpie.py --start-dirty` prevents :program:`kewpie.py` from
177
attempting to initialize (clean) the datadir. This can be useful if you want
178
to use an already-populated datadir for testing.
180
NOTE: This feature is still being tested, use caution with your data!!!
184
A cleanup mode is provided for user convenience. This simply shuts down
185
any servers whose pid files are detected in the kewpie workdir. It is mainly
186
intended as a quick cleanup for post-testing with :option:`kewpie.py --start-and-exit`::
188
./kewpie.py --mode=cleanup
190
Setting --start-dirty=True for cleanup mode...
191
23 Feb 2011 11:35:59 INFO: Using Drizzle source tree:
192
23 Feb 2011 11:35:59 INFO: basedir: drizzle
193
23 Feb 2011 11:35:59 INFO: clientbindir: drizzle/client
194
23 Feb 2011 11:35:59 INFO: testdir: drizzle/tests
195
23 Feb 2011 11:35:59 INFO: server_version: 2011.02.2188
196
23 Feb 2011 11:35:59 INFO: server_compile_os: unknown-linux-gnu
197
23 Feb 2011 11:35:59 INFO: server_platform: x86_64
198
23 Feb 2011 11:35:59 INFO: server_comment: (Source distribution (kewpie_randgen))
199
23 Feb 2011 11:35:59 INFO: Using --start-dirty, not attempting to touch directories
200
23 Feb 2011 11:35:59 INFO: Using default-storage-engine: innodb
201
23 Feb 2011 11:35:59 INFO: Using testing mode: cleanup
202
23 Feb 2011 11:35:59 INFO: Killing pid 10484 from drizzle/tests/workdir/testbot0/server0/var/run/server0.pid
203
23 Feb 2011 11:35:59 INFO: Stopping all running servers...
208
:program:`kewpie.py`'s 'dtr' mode uses a simple diff-based mechanism for testing.
209
This is the default mode and where the majority of Drizzle testing occurs.
210
It will execute the statements contained in a test and compare the results
211
to pre-recorded expected results. In the event of a test failure, you
212
will be presented with a diff::
215
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
216
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
224
A test case consists of a .test and a .result file. The .test file includes
225
the various statements to be executed for a test. The .result file lists
226
the expected results for a given test file. These files live in tests/t
227
and tests/r, respectively. This structure is the same for all test suites.
232
The :program:`kewpie.py` tool has several available options:
234
./kewpie.py [ OPTIONS ] [ TESTCASE ]
240
.. program:: kewpie.py
242
.. option:: -h, --help
244
show this help message and exit
246
Configuration controls - kewpie can read config files with certain options pre-set:
247
---------------------------------------------------------------------------------------------------
249
.. option:: --sys_config_file=SYSCONFIGFILEPATH
251
The file that specifies system configuration specs for
252
kewpie to execute tests (not yet implemented)
254
Options for the test-runner itself
255
----------------------------------
257
.. program:: kewpie.py
261
Set this to continue test execution beyond the first failed test
263
.. option:: --start-and-exit
265
Spin up the server(s) for the first specified test then exit
266
(will leave servers running)
268
.. option:: --verbose
270
Produces extensive output about test-runner state.
271
Distinct from --debug
275
Provide internal-level debugging output.
276
Distinct from --verbose
278
.. option:: --mode=MODE
281
Currently supporting dtr, sysbench, and native (unittest) modes. The goal is to remove this and have all tests operate via unittest"
286
Record a testcase result
287
(if the testing mode supports it - MTR / DTR specific)
292
Don't try to cleanup from earlier runs
293
(currently just a placeholder) [False]
296
Options for controlling which tests are executed
297
------------------------------------------------
299
.. program:: kewpie.py
301
.. option:: --suite=SUITELIST
303
The name of the suite containing tests we want.
304
Can accept comma-separated list (with no spaces).
305
Additional --suite args are appended to existing list
308
.. option:: --suitepath=SUITEPATHS
310
The path containing the suite(s) you wish to execute.
311
Use on --suitepath for each suite you want to use.
313
.. option:: --do-test=DOTEST
315
input can either be a prefix or a regex.
316
Will only execute tests that match the provided pattern
318
.. option:: --skip-test=SKIPTEST
320
input can either be a prefix or a regex.
321
Will exclude tests that match the provided pattern
323
.. option:: --reorder
325
sort the testcases so that they are executed optimally
326
for the given mode [False]
328
.. option:: --repeat=REPEAT
330
Run each test case the specified number of times. For
331
a given sequence, the first test will be run n times,
332
then the second, etc [1]
334
Options for defining the code that will be under test
335
-----------------------------------------------------
337
.. program:: kewpie.py
339
.. option:: --basedir=BASEDIR
341
Pass this argument to signal to the test-runner
342
that this is an in-tree test (not required).
343
We automatically set a number of variables
344
relative to the argument (client-bindir,
345
serverdir, testdir) [../]
347
.. option:: --serverdir=SERVERPATH
349
Path to the server executable. [auto-search]
351
.. option:: --client-bindir=CLIENTBINDIR
353
Path to the directory containing client program
354
binaries for use in testing [auto-search]
356
.. option:: --default-storage-engine=DEFAULTENGINE
358
Start drizzled using the specified engine [innodb]
360
Options for defining the testing environment
361
--------------------------------------------
363
.. program:: kewpie.py
365
.. option:: --testdir=TESTDIR
367
Path to the test dir, containing additional files for
368
test execution. [pwd]
370
.. option:: --workdir=WORKDIR
372
Path to the directory test-run will use to store
373
generated files and directories.
374
[basedir/tests/kewpie_work]
376
.. option:: --top-srcdir=TOPSRCDIR
378
build option [basedir_default]
380
.. option:: --top-builddir=TOPBUILDDIR
382
build option [basedir_default]
386
By default, we symlink workdir to a location in shm.
387
Use this flag to not symlink [False]
389
.. option:: --libeatmydata
391
We use libeatmydata (if available) to disable fsyncs
392
and speed up test execution. Implies --no-shm
394
.. option:: --libeatmydata-path=LIBEATMYDATAPATH
396
Path to the libeatmydata install you want to use
397
[/usr/local/lib/libeatmydata.so]
399
.. option:: --start-dirty
401
Don't try to clean up working directories before test
404
.. option:: --no-secure-file-priv
406
Turn off the use of --secure-file-priv=vardir for
409
.. option:: --randgen-path=RANDGENPATH
411
The path to a randgen installation that can be used to
412
execute randgen-based tests [kewpie/randgen]
414
.. option:: --innobackupex-path=INNOBACKUPEXPATH
416
The path to the innobackupex script that facilitates
417
the use of Xtrabackup
419
.. option:: --xtrabackup-path=XTRABACKUPPATH
421
The path the xtrabackup binary to be tested
423
.. option:: --wsrep-provider-path=WSREPPROVIDER
425
The path to a wsrep provider library for use with
428
.. option:: --subunit-outfile=SUBUNITOUTFILE
430
File path where subunit output will be logged
431
[/kewpie/workdir/test_results.subunit]
433
Options to pass options on to the server
434
-----------------------------------------
436
.. program:: kewpie.py
438
.. option:: --drizzled=DRIZZLEDOPTIONS
440
Pass additional options to the server. Will be passed
441
to all servers for all tests (mostly for --start-and-
445
Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)
446
------------------------------------------------------------------------------------
448
.. program:: kewpie.py
450
.. option:: --valgrind
452
Run drizzletest and drizzled executables using
453
valgrind with default options [False]
455
.. option:: --valgrind-option=VALGRINDARGLIST
457
Pass an option to valgrind (overrides/removes default
460
.. option:: --valgrind-suppressions=VALGRINDSUPPRESSIONS
462
Point at a valgrind suppression file
463
[kewpie/valgrind.supp]
465
.. option:: --helgrind
467
Use the helgrind tool for valgrind. Implies / will
470
Options for controlling the use of debuggers with test execution
471
----------------------------------------------------------------
473
.. program:: kewpie.py
477
Start the drizzled server(s) in gdb
479
.. option:: --manual-gdb
481
Allows you to start the drizzled server(s) in gdb
482
manually (in another window, etc
484
Options to call additional utilities such as datagen
485
------------------------------------------------------
487
.. program:: kewpie.py
489
.. option:: --gendata=GENDATAFILE
491
Call the randgen's gendata utility to use the
492
specified configuration file. This will populate the
493
server prior to any test execution