~ubuntu-branches/ubuntu/saucy/drizzle/saucy-proposed

Designed as a response to the sometimes painfully diverse ecosystem of testing tools, the intent is to provide a system that standardizes common testing tasks while providing a variety of ways to analyze the behavior of database systems.

Long experience with MySQL testing has shown that a number of tasks are common to all tests:

* Allocation and management of test servers

* Test collection and management

* Reporting of test results

Kewpie combines this with the flexibity of Python's unittest framework. By allowing users to write tests as Python modules, we have great flexibility in how we start and set up servers, what we do to them, and how we assess their performance of our test tasks. Some of the tasks kewpie test cases cover:

* randgen tests

* running sql-bench comprehensive tests

* running sql-bench crashme

* running sysbench

* running drizzle-test-run test cases

* more direct tests using python code and helper libraries

Provided test suites

=======================

* MySQL / Percona Server / Galera tests:

* randgen_basic - basic randgen tests of the server (optimizer stress, etc)

* randgen_bugs - holder suite for failing tests

* crashme - sql-bench's crashme suite (may take some time to run)

* sqlbench - sql-bench comprehensive suite. (may take ~45 min. to execute)

* cluster_basic - small, atomic tests of replication functionality. Tests are written in such a way as to be portable across server types (!)

* cluster_bugs - holder suite for failing tests

* cluster_randgen - tests of replication functionality using the random query generator and relevant test loads

* xtrabackup_basic - tests of the Percona Xtrabackup tool

* xtrabackup_bugs - hoder suite for failing tests

* Drizzle tests:

* randgen_basic - basic tests of the server (optimizer stress, etc)

* The following all use the same transaction tests, but validate different functionality:

* randgen_trxLog - tests the file-based replication log

* randgen_innoTrxLog - tests the innodb-table-based replication log

* randgen_slavePlugin - tests the functionality of the replication / slave plugin

* crashme - sql-bench's crashme suite (may take some time to run)

* sqlbench - sql-bench comprehensive suite. (may take ~45 min. to execute)

Running tests

=========================

There are several different ways to run tests using :program:`kewpie.py`.

It should be noted that unless :option:`kewpie.py --force` is used, the program

will stop execution upon encountering the first failing test.

:option:`kewpie.py --force` is recommended if you are running several tests

- it will allow you to view all successes and failures in one run.

Running individual tests

------------------------

If one only wants to run a few, specific tests, they may do so this way::

./kewpie.py [OPTIONS] test1 [test2 ... testN]

Running all tests within a suite

--------------------------------

Many of the tests supplied with Drizzle are organized into suites.

The tests within drizzle/tests/t are considered the 'main' suite.

Other suites are located in either drizzle/tests/suite or within the various

directories in drizzle/plugin. Tests for a specific plugin should live in

the plugin's directory - drizzle/plugin/example_plugin/tests

To run the tests in a specific suite::

./kewpie.py [OPTIONS] --suite=SUITENAME

Running specific tests within a suite

--------------------------------------

To run a specific set of tests within a suite::

./kewpie.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]

Calling tests using <suitename>.<testname> currently does not work. One must

100

specify the test suite via the :option:`kewpie.py --suite` option.

101

102

103

Running all available tests

104

---------------------------

105

Currently, the quickest way to execute all tests in all suites is

106

to use 'make test-kewpie' from the drizzle root.

107

108

Otherwise, one should simply name all suites::

109

110

./kewpie.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN

111

112

Interpreting test results

113

=========================

114

The output of the test runner is quite simple. Every test should pass.

115

In the event of a test failure, please take the time to file a bug here:

116

*https://bugs.launchpad.net/drizzle*

117

118

During a run, the program will provide the user with:

119

* test name (suite + name)

120

* test status (pass/fail/skipped)

121

* time spent executing each test

122

123

At the end of a run, the program will provide the user with a listing of:

124

* how many tests were run

125

* counts and percentages of total exectuted for all test statuses

126

* a listing of failing, skipped, or disabled tests

127

* total time spent executing the tests

128

129

Example output::

130

131

<snip>

132

30 Jan 2011 16:26:31 : main.small_tmp_table [ pass ] 38

133

30 Jan 2011 16:26:31 : main.snowman [ pass ] 42

134

30 Jan 2011 16:26:31 : main.statement_boundaries [ pass ] 47

135

30 Jan 2011 16:26:31 : main.status [ pass ] 51

136

30 Jan 2011 16:26:31 : main.strict [ pass ] 138

137

30 Jan 2011 16:26:43 : main.subselect [ fail ] 12361

138

30 Jan 2011 16:26:43 : --- drizzle/tests/r/subselect.result 2011-01-30 16:23:54.975776148 -0500

139

30 Jan 2011 16:26:43 : +++ drizzle/tests/r/subselect.reject 2011-01-30 16:26:43.835519303 -0500

140

30 Jan 2011 16:26:43 : @@ -5,7 +5,7 @@

141

30 Jan 2011 16:26:43 : 2

142

30 Jan 2011 16:26:43 : explain extended select (select 2);

143

30 Jan 2011 16:26:43 : id select_type table type possible_keys key key_len ref rows filtered Extra

144

30 Jan 2011 16:26:43 : -9 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used

145

30 Jan 2011 16:26:43 : +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used

146

<snip>

147

30 Jan 2011 16:30:20 : ================================================================================

148

30 Jan 2011 16:30:20 INFO: Test execution complete in 314 seconds

149

30 Jan 2011 16:30:20 INFO: Summary report:

150

30 Jan 2011 16:30:20 INFO: Executed 552/552 test cases, 100.00 percent

151

30 Jan 2011 16:30:20 INFO: STATUS: FAIL, 1/552 test cases, 0.18 percent executed

152

30 Jan 2011 16:30:20 INFO: STATUS: PASS, 551/552 test cases, 99.82 percent executed

153

30 Jan 2011 16:30:20 INFO: FAIL tests: main.subselect

154

30 Jan 2011 16:30:20 INFO: Spent 308 / 314 seconds on: TEST(s)

155

30 Jan 2011 16:30:20 INFO: Test execution complete

156

30 Jan 2011 16:30:20 INFO: Stopping all running servers...

157

158

159

Additional uses

160

===============

161

Starting a server for manual testing

162

------------------------------------

163

164

:program:`kewpie.py` allows a user to get a Drizzle server up and running

165

quickly. This can be useful for fast ad-hoc testing.

166

167

To do so call::

168

169

./kewpie.py --start-and-exit [*OPTIONS*]

170

171

This will start a Drizzle server that you can connect to and query

172

173

Starting a server against a pre-populated DATADIR

174

--------------------------------------------------

175

176

Using :option:`kewpie.py --start-dirty` prevents :program:`kewpie.py` from

177

attempting to initialize (clean) the datadir. This can be useful if you want

178

to use an already-populated datadir for testing.

179

180

NOTE: This feature is still being tested, use caution with your data!!!

181

182

Cleanup mode

183

-------------

184

A cleanup mode is provided for user convenience. This simply shuts down

185

any servers whose pid files are detected in the kewpie workdir. It is mainly

186

intended as a quick cleanup for post-testing with :option:`kewpie.py --start-and-exit`::

187

188

./kewpie.py --mode=cleanup

189

190

Setting --start-dirty=True for cleanup mode...

191

23 Feb 2011 11:35:59 INFO: Using Drizzle source tree:

192

23 Feb 2011 11:35:59 INFO: basedir: drizzle

193

23 Feb 2011 11:35:59 INFO: clientbindir: drizzle/client

194

23 Feb 2011 11:35:59 INFO: testdir: drizzle/tests

195

23 Feb 2011 11:35:59 INFO: server_version: 2011.02.2188

196

23 Feb 2011 11:35:59 INFO: server_compile_os: unknown-linux-gnu

197

23 Feb 2011 11:35:59 INFO: server_platform: x86_64

198

23 Feb 2011 11:35:59 INFO: server_comment: (Source distribution (kewpie_randgen))

199

23 Feb 2011 11:35:59 INFO: Using --start-dirty, not attempting to touch directories

200

23 Feb 2011 11:35:59 INFO: Using default-storage-engine: innodb

201

23 Feb 2011 11:35:59 INFO: Using testing mode: cleanup

202

23 Feb 2011 11:35:59 INFO: Killing pid 10484 from drizzle/tests/workdir/testbot0/server0/var/run/server0.pid

203

23 Feb 2011 11:35:59 INFO: Stopping all running servers...

204

205

Program architecture

206

====================

207

208

:program:`kewpie.py`'s 'dtr' mode uses a simple diff-based mechanism for testing.

209

This is the default mode and where the majority of Drizzle testing occurs.

210

It will execute the statements contained in a test and compare the results

211

to pre-recorded expected results. In the event of a test failure, you

212

will be presented with a diff::

213

214

main.exp1 [ fail ]

215

--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300

216

+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300

217

@@ -5,4 +5,5 @@

218

219

220

221

222

DROP TABLE t1;

223

224

A test case consists of a .test and a .result file. The .test file includes

225

the various statements to be executed for a test. The .result file lists

226

the expected results for a given test file. These files live in tests/t

227

and tests/r, respectively. This structure is the same for all test suites.

228

229

kewpie.py options

230

===================

231

232

The :program:`kewpie.py` tool has several available options:

233

234

./kewpie.py [ OPTIONS ] [ TESTCASE ]

235

236

237

Options

238

-------

239

240

.. program:: kewpie.py

241

242

.. option:: -h, --help

243

244

show this help message and exit

245

246

Configuration controls - kewpie can read config files with certain options pre-set:

247

---------------------------------------------------------------------------------------------------

248

249

.. option:: --sys_config_file=SYSCONFIGFILEPATH

250

251

The file that specifies system configuration specs for

252

kewpie to execute tests (not yet implemented)

253

254

Options for the test-runner itself

255

----------------------------------

256

257

.. program:: kewpie.py

258

259

.. option:: --force

260

261

Set this to continue test execution beyond the first failed test

262

263

.. option:: --start-and-exit

264

265

Spin up the server(s) for the first specified test then exit

266

(will leave servers running)

267

268

.. option:: --verbose

269

270

Produces extensive output about test-runner state.

271

Distinct from --debug

272

273

.. option:: --debug

274

275

Provide internal-level debugging output.

276

Distinct from --verbose

277

278

.. option:: --mode=MODE

279

280

Testing mode.

281

Currently supporting dtr, sysbench, and native (unittest) modes. The goal is to remove this and have all tests operate via unittest"

282

[native]

283

284

.. option:: --record

285

286

Record a testcase result

287

(if the testing mode supports it - MTR / DTR specific)

288

[False]

289

290

.. option:: --fast

291

292

Don't try to cleanup from earlier runs

293

(currently just a placeholder) [False]

294

295

296

Options for controlling which tests are executed

297

------------------------------------------------

298

299

.. program:: kewpie.py

300

301

.. option:: --suite=SUITELIST

302

303

The name of the suite containing tests we want.

304

Can accept comma-separated list (with no spaces).

305

Additional --suite args are appended to existing list

306

[autosearch]

307

308

.. option:: --suitepath=SUITEPATHS

309

310

The path containing the suite(s) you wish to execute.

311

Use on --suitepath for each suite you want to use.

312

313

.. option:: --do-test=DOTEST

314

315

input can either be a prefix or a regex.

316

Will only execute tests that match the provided pattern

317

318

.. option:: --skip-test=SKIPTEST

319

320

input can either be a prefix or a regex.

321

Will exclude tests that match the provided pattern

322

323

.. option:: --reorder

324

325

sort the testcases so that they are executed optimally

326

for the given mode [False]

327

328

.. option:: --repeat=REPEAT

329

330

Run each test case the specified number of times. For

331

a given sequence, the first test will be run n times,

332

then the second, etc [1]

333

334

Options for defining the code that will be under test

335

-----------------------------------------------------

336

337

.. program:: kewpie.py

338

339

.. option:: --basedir=BASEDIR

340

341

Pass this argument to signal to the test-runner

342

that this is an in-tree test (not required).

343

We automatically set a number of variables

344

relative to the argument (client-bindir,

345

serverdir, testdir) [../]

346

347

.. option:: --serverdir=SERVERPATH

348

349

Path to the server executable. [auto-search]

350

351

.. option:: --client-bindir=CLIENTBINDIR

352

353

Path to the directory containing client program

354

binaries for use in testing [auto-search]

355

356

.. option:: --default-storage-engine=DEFAULTENGINE

357

358

Start drizzled using the specified engine [innodb]

359

360

Options for defining the testing environment

361

--------------------------------------------

362

363

.. program:: kewpie.py

364

365

.. option:: --testdir=TESTDIR

366

367

Path to the test dir, containing additional files for

368

test execution. [pwd]

369

370

.. option:: --workdir=WORKDIR

371

372

Path to the directory test-run will use to store

373

generated files and directories.

374

[basedir/tests/kewpie_work]

375

376

.. option:: --top-srcdir=TOPSRCDIR

377

378

build option [basedir_default]

379

380

.. option:: --top-builddir=TOPBUILDDIR

381

382

build option [basedir_default]

383

384

.. option:: --no-shm

385

386

By default, we symlink workdir to a location in shm.

387

Use this flag to not symlink [False]

388

389

.. option:: --libeatmydata

390

391

We use libeatmydata (if available) to disable fsyncs

392

and speed up test execution. Implies --no-shm

393

394

.. option:: --libeatmydata-path=LIBEATMYDATAPATH

395

396

Path to the libeatmydata install you want to use

397

[/usr/local/lib/libeatmydata.so]

398

399

.. option:: --start-dirty

400

401

Don't try to clean up working directories before test

402

execution [False]

403

404

.. option:: --no-secure-file-priv

405

406

Turn off the use of --secure-file-priv=vardir for

407

started servers

408

409

.. option:: --randgen-path=RANDGENPATH

410

411

The path to a randgen installation that can be used to

412

execute randgen-based tests [kewpie/randgen]

413

414

.. option:: --innobackupex-path=INNOBACKUPEXPATH

415

416

The path to the innobackupex script that facilitates

417

the use of Xtrabackup

418

419

.. option:: --xtrabackup-path=XTRABACKUPPATH

420

421

The path the xtrabackup binary to be tested

422

423

.. option:: --wsrep-provider-path=WSREPPROVIDER

424

425

The path to a wsrep provider library for use with

426

mysql

427

428

.. option:: --subunit-outfile=SUBUNITOUTFILE

429

430

File path where subunit output will be logged

431

[/kewpie/workdir/test_results.subunit]

432

433

Options to pass options on to the server

434

-----------------------------------------

435

436

.. program:: kewpie.py

437

438

.. option:: --drizzled=DRIZZLEDOPTIONS

439

440

Pass additional options to the server. Will be passed

441

to all servers for all tests (mostly for --start-and-

442

exit)

443

444

445

Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)

446

------------------------------------------------------------------------------------

447

448

.. program:: kewpie.py

449

450

.. option:: --valgrind

451

452

Run drizzletest and drizzled executables using

453

valgrind with default options [False]

454

455

.. option:: --valgrind-option=VALGRINDARGLIST

456

457

Pass an option to valgrind (overrides/removes default

458

valgrind options)

459

460

.. option:: --valgrind-suppressions=VALGRINDSUPPRESSIONS

461

462

Point at a valgrind suppression file

463

[kewpie/valgrind.supp]

464

465

.. option:: --helgrind

466

467

Use the helgrind tool for valgrind. Implies / will

468

auto-use --valgrind

469

470

Options for controlling the use of debuggers with test execution

471

----------------------------------------------------------------

472

473

.. program:: kewpie.py

474

475

.. option:: --gdb

476

477

Start the drizzled server(s) in gdb

478

479

.. option:: --manual-gdb

480

481

Allows you to start the drizzled server(s) in gdb

482

manually (in another window, etc

483

484

Options to call additional utilities such as datagen

485

------------------------------------------------------

486

487

.. program:: kewpie.py

488

489

.. option:: --gendata=GENDATAFILE

490

491

Call the randgen's gendata utility to use the

492

specified configuration file. This will populate the

493

server prior to any test execution

494

Older »