1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
3
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
4
<title>How SQLite Is Tested</title>
5
<style type="text/css">
8
font-family: Verdana, sans-serif;
13
a:visited { color: #734559 }
15
.logo { position:absolute; margin:3px; }
31
.toolbar a { color: white; text-decoration: none; padding: 6px 12px; }
32
.toolbar a:visited { color: white; }
33
.toolbar a:hover { color: #044a64; background: white; }
35
.content { margin: 5%; }
36
.content dt { font-weight:bold; }
37
.content dd { margin-bottom: 25px; margin-left:20%; }
38
.content ul { padding:0px; padding-left: 15px; margin:0px; }
41
.se { background: url(images/se.gif) 100% 100% no-repeat #044a64}
42
.sw { background: url(images/sw.gif) 0% 100% no-repeat }
43
.ne { background: url(images/ne.gif) 100% 0% no-repeat }
44
.nw { background: url(images/nw.gif) 0% 0% no-repeat }
46
/* Things for "fancyformat" documents start here. */
47
.fancy img+p {font-style:italic}
48
.fancy .codeblock i { color: darkblue; }
49
.fancy h1,.fancy h2,.fancy h3,.fancy h4 {font-weight:normal;color:#044a64}
50
.fancy h2 { margin-left: 10px }
51
.fancy h3 { margin-left: 20px }
52
.fancy h4 { margin-left: 30px }
53
.fancy th {white-space:nowrap;text-align:left;border-bottom:solid 1px #444}
54
.fancy th, .fancy td {padding: 0.2em 1ex; vertical-align:top}
55
.fancy #toc a { color: darkblue ; text-decoration: none }
56
.fancy .todo { color: #AA3333 ; font-style : italic }
57
.fancy .todo:before { content: 'TODO:' }
58
.fancy p.todo { border: solid #AA3333 1px; padding: 1ex }
59
.fancy img { display:block; }
60
.fancy :link:hover, .fancy :visited:hover { background: wheat }
61
.fancy p,.fancy ul,.fancy ol { margin: 1em 5ex }
62
.fancy li p { margin: 1em 0 }
63
/* End of "fancyformat" specific rules. */
69
<div><!-- container div to satisfy validator -->
72
<img class="logo" src="images/sqlite370_banner.gif" alt="SQLite Logo"
74
<div><!-- IE hack to prevent disappearing logo--></div>
75
<div class="tagline">Small. Fast. Reliable.<br>Choose any three.</div>
77
<table width=100% style="clear:both"><tr><td>
78
<div class="se"><div class="sw"><div class="ne"><div class="nw">
79
<table width=100% style="padding:0;margin:0;cell-spacing:0"><tr>
82
<a href="about.html">About</a>
83
<a href="sitemap.html">Sitemap</a>
84
<a href="docs.html">Documentation</a>
85
<a href="download.html">Download</a>
86
<a href="copyright.html">License</a>
87
<a href="news.html">News</a>
88
<a href="support.html">Support</a>
91
gMsg = "Search SQLite Docs..."
92
function entersearch() {
93
var q = document.getElementById("q");
94
if( q.value == gMsg ) { q.value = "" }
95
q.style.color = "black"
96
q.style.fontStyle = "normal"
98
function leavesearch() {
99
var q = document.getElementById("q");
100
if( q.value == "" ) {
102
q.style.color = "#044a64"
103
q.style.fontStyle = "italic"
108
<div style="padding:0 1em 0px 0;white-space:nowrap">
109
<form name=f method="GET" action="http://www.sqlite.org/search">
110
<input id=q name=q type=text
111
onfocus="entersearch()" onblur="leavesearch()" style="width:24ex;padding:1px 1ex; border:solid white 1px; font-size:0.9em ; font-style:italic;color:#044a64;" value="Search SQLite Docs...">
112
<input type=submit value="Go" style="border:solid white 1px;background-color:#044a64;color:white;font-size:0.9em;padding:0 1ex">
116
</div></div></div></div>
118
<div class=startsearch></div>
125
<h1 align="center">How SQLite Is Tested</h1>
127
<h2>1.0 Introduction</h2>
129
<p>The reliability and robustness of SQLite is achieved in part
130
by thorough and careful testing.</p>
132
<p>As of <a href="releaselog/3_7_8.html">version 3.7.8</a>,
133
the SQLite library consists of approximately
134
77.6 KSLOC of C code.
135
(KSLOC means thousands of "Source Lines Of Code" or, in other words,
136
lines of code excluding blank lines and comments.)
137
By comparison, the project has
139
test code and test scripts -
142
<h3>1.1 Executive Summary</h3>
145
<li> Three independently developed test harnesses
146
<li> 100% branch test coverage in an as-deployed configuration
147
<li> Millions and millions of test cases
148
<li> Out-of-memory tests
150
<li> Crash and power loss tests
152
<li> Boundary value tests
153
<li> Disabled optimization tests
154
<li> Regression tests
155
<li> Malformed database tests
156
<li> Extensive use of assert() and run-time checks
157
<li> Valgrind analysis
158
<li> Signed-integer overflow checks
161
<a name="harnesses"></a>
163
<h2>2.0 Test Harnesses</h2>
165
<p>There are three independent test harnesses used for testing the
167
Each test harness is designed, maintained, and managed separately
175
The <b>TCL Tests</b> are the oldest set of tests for SQLite.
176
They are contained in the same source tree as the
177
SQLite core and like the SQLite core are in the public domain. The
178
TCL tests are the primary tests used during development.
179
The TCL tests are written using the
180
<a href="http://www.tcl.tk/">TCL scripting language</a>.
181
The TCL test harness itself consists of 21.6 KSLOC
182
of C code used to create the TCL interface. The test scripts are contained
183
in 661 files totaling
184
9.6MB in size. There are
185
28278 distinct test cases, but many of the test
186
cases are parameterized and run multiple times (with different parameters)
187
so that on a full test run, about 1.7 million
188
separate tests are performed.
193
The <b><a href="th3.html">TH3</a></b> test harness is a set of proprietary tests, written in
194
C that provide 100% branch test coverage (and 100% MC/DC test coverage) to
195
the core SQLite library. The TH3 tests are designed to run
196
on embedded and specialized platforms that would not easily support
197
TCL or other workstation services. TH3 tests use only the published
198
SQLite interfaces. TH3 is free to <a href="consortium.html">SQLite Consortium</a> members
199
and is available by license to others. TH3 consists of about
200
48.9 MB or 650.5 KSLOC
201
of C code implementing 33595 distinct test cases.
202
TH3 tests are heavily parameterized, though, so a full-coverage test runs
203
about 0.8 million different test
204
instances. The cases that provide 100% branch test coverage constitute
205
a subset of the total TH3 test suite. A soak test
206
prior to release does hundreds of millions of tests.
207
Additional information on TH3 is <a href="th3.html">available separately</a>.</p></li>
212
The <a href="http://www.sqlite.org/sqllogictest"><b>SQL Logic Test</b></a>
213
or SLT test harness is used to run huge numbers
214
of SQL statements against both SQLite and several other SQL database engines
215
and verify that they all get the same answers. SLT currently compares
216
SQLite against PostgreSQL, MySQL, Microsoft SQL Server, and Oracle 10g.
217
SLT runs 7.2 million queries comprising
222
<p>All of the tests above must run successfully, on multiple platforms
223
and under multiple compile-time configurations,
224
before each release of SQLite.</p>
226
<p>Prior to each check-in to the SQLite source tree, developers
227
typically run a subset (called "veryquick") of the Tcl tests
229
132.2 thousand test cases.
230
The veryquick tests include everything except the anomaly, fuzz, and
231
soak tests. The idea behind the veryquick tests are that they are
232
sufficient to catch most errors, but also run in only a few minutes
233
instead of a few hours.</p>
235
<a name="anomoly"></a>
237
<h2>3.0 Anomaly Testing</h2>
239
<p>Anomaly tests are tests designed to verify the correct behavior
240
of SQLite when something goes wrong. It is (relatively) easy to build
241
an SQL database engine that behaves correctly on well-formed inputs
242
on a fully functional computer. It is more difficult to build a system
243
that responds sanely to invalid inputs and continues to function following
244
system malfunctions. The anomaly tests are designed to verify the latter
247
<a name="oomtesting"></a>
249
<h3>3.1 Out-Of-Memory Testing</h3>
251
<p>SQLite, like all SQL database engines, makes extensive use of
252
malloc() (See the separate report on
253
<a href="malloc.html">dynamic memory allocation in SQLite</a> for
255
On servers and workstations, malloc() never fails in practice and so correct
256
handling of out-of-memory (OOM) errors is not particularly important.
257
But on embedded devices, OOM errors are frighteningly common and since
258
SQLite is frequently used on embedded devices, it is important that
259
SQLite be able to gracefully handle OOM errors.</p>
261
<p>OOM testing is accomplished by simulating OOM errors.
262
SQLite allows an application to substitute an alternative malloc()
263
implementation using the <a href="c3ref/config.html">sqlite3_config</a>(<a href="c3ref/c_config_getmalloc.html#sqliteconfigmalloc">SQLITE_CONFIG_MALLOC</a>,...)
264
interface. The TCL and TH3 test harnesses are both capable of
265
inserting a modified version of malloc() that can be rigged to fail
266
after a certain number of allocations. These instrumented mallocs
267
can be set to fail only once and then start working again, or to
268
continue failing after the first failure. OOM tests are done in a
269
loop. On the first iteration of the loop, the instrumented malloc
270
is rigged to fail on the first allocation. Then some SQLite operation
271
is carried out and checks are done to make sure SQLite handled the
272
OOM error correctly. Then the time-to-failure counter
273
on the instrumented malloc is increased by one and the test is
274
repeated. The loop continues until the entire operation runs to
275
completion without ever encountering a simulated OOM failure.
276
Tests like this are run twice, once with the instrumented malloc
277
set to fail only once, and again with the instrumented malloc set
278
to fail continuously after the first failure.</p>
280
<a name="ioerrtesting"></a>
282
<h3>3.2 I/O Error Testing</h3>
284
<p>I/O error testing seeks to verify that SQLite responds sanely
285
to failed I/O operations. I/O errors might result from a full disk drive,
286
malfunctioning disk hardware, network outages when using a network
287
file system, system configuration or permission changes that occur in the
288
middle of an SQL operation, or other hardware or operating system
289
malfunctions. Whatever the cause, it is important that SQLite be able
290
to respond correctly to these errors and I/O error testing seeks to
291
verify that it does.</p>
293
<p>I/O error testing is similar in concept to OOM testing; I/O errors
294
are simulated and checks are made to verify that SQLite responds
295
correctly to the simulated errors. I/O errors are simulated in both
296
the TCL and TH3 test harnesses by inserting a new
297
<a href="c3ref/vfs.html">Virtual File System object</a> that is specially rigged
298
to simulate an I/O error after a set number of I/O operations.
299
As with OOM error testing, the I/O error simulators can be set to
300
fail just once, or to fail continuously after the first failure.
301
Tests are run in a loop, slowly increasing the point of failure until
302
the test case runs to completion without error. The loop is run twice,
303
once with the I/O error simulator set to simulate only a single failure
304
and a second time with it set to fail all I/O operations after the first
307
<p>In I/O error tests, after the I/O error simulation failure mechanism
308
is disabled, the database is examined using
309
<a href="pragma.html#pragma_integrity_check">PRAGMA integrity_check</a> to make sure that the I/O error has not
310
introduced database corruption.</p>
312
<a name="crashtesting"></a>
314
<h3>3.3 Crash Testing</h3>
316
<p>Crash testing seeks to demonstrate that an SQLite database will not
317
go corrupt if the application or operating system crashes or if there
318
is a power failure in the middle of a database update. A separate
320
<a href="atomiccommit.html">Atomic Commit in SQLite</a> describes the
321
defensive measure SQLite takes to prevent database corruption following
322
a crash. Crash tests strive to verify that those defensive measures
323
are working correctly.</p>
325
<p>It is impractical to do crash testing using real power failures, of
326
course, and so crash testing is done in simulation. An alternative
327
<a href="c3ref/vfs.html">Virtual File System</a> is inserted that allows the test
328
harness to simulate the state of the database file following a crash.</p>
330
<p>In the TCL test harness, the crash simulation is done in a separate
331
process. The main testing process spawns a child process which runs
332
some SQLite operation and randomly crashes somewhere in the middle of
333
a write operation. A special <a href="vfs.html">VFS</a> randomly reorders and corrupts
335
write operations to simulate the effect of buffered filesystems. After
336
the child dies, the original test process opens and reads the test
337
database and verifies that the changes attempted by the child either
338
completed successfully or else were completely rolled back. The
339
<a href="pragma.html#pragma_integrity_check">integrity_check</a> <a href="pragma.html#syntax">PRAGMA</a> is used to make sure no database corruption
342
<p>The TH3 test harness needs to run on embedded systems that do not
343
necessarily have the ability to spawn child processes, so it uses
344
an in-memory <a href="vfs.html">VFS</a> to simulate crashes. The in-memory <a href="vfs.html">VFS</a> can be rigged
345
to make a snapshot of the entire filesystem after a set number of I/O
346
operations. Crash tests run in a loop. On each iteration of the loop,
347
the point at which a snapshot is made is advanced until the SQLite
348
operations being tested run to completion without ever hitting a
349
snapshot. Within the loop, after the SQLite operation under test has
350
completed, the filesystem is reverted to the snapshot and random file
351
damage is introduced that is characteristic of the kinds of damage
352
one expects to see following a power loss. Then the database is opened
353
and checks are made to ensure that it is well-formed and that the
354
transaction either ran to completion or was completely rolled back.
355
The interior of the loop is repeated multiple times for each
356
snapshot with different random damage each time.</p>
358
<a name="multifail"></a>
360
<h3>3.4 Compound failure tests</h3>
362
<p>The test suites for SQLite also explore the result of stacking
363
multiple failures. For example, tests are run to ensure correct behavior
364
when an I/O error or OOM fault occurs while trying to recover from a
367
<a name="fuzztesting"></a>
369
<h2>4.0 Fuzz Testing</h2>
371
<p><a href="http://en.wikipedia.org/wiki/Fuzz_testing">Fuzz testing</a>
372
seeks to establish that SQLite responds correctly to invalid, out-of-range,
373
or malformed inputs.</p>
375
<h3>4.1 SQL Fuzz</h3>
377
<p>SQL fuzz testing consists of creating syntactically correct yet
378
wildly nonsensical SQL statements and feeding them to SQLite to see
379
what it will do with them. Usually some kind of error is returned
380
(such as "no such table"). Sometimes, purely by chance, the SQL
381
statement also happens to be semantically correct. In that case, the
382
resulting prepared statement is run to make sure it gives a reasonable
385
<p>The SQL fuzz generator tests are part of the TCL test suite.
386
During a full test run, about 102.1
387
thousand fuzz SQL statements are
388
generated and tested.</p>
390
<h3>4.2 Malformed Database Files</h3>
392
<p>There are numerous test cases that verify that SQLite is able to
393
deal with malformed database files.
394
These tests first build a well-formed database file, then add
395
corruption by changing one or more bytes in the file by some means
396
other than SQLite. Then SQLite is used to read the database.
397
In some cases, the bytes changes are in the middle of data.
398
This causes the content of the database to change while keeping the
399
database well-formed.
400
In other cases, unused bytes of the file are modified, which has
401
no effect on the integrity of the database.
402
The interesting cases are when bytes of the file that
403
define database structure get changed. The malformed database tests
404
verify that SQLite finds the file format errors and reports them
405
using the <a href="c3ref/c_abort.html">SQLITE_CORRUPT</a> return code without overflowing
406
buffers, dereferencing NULL pointers, or performing other
407
unwholesome actions.</p>
409
<h3>4.3 Boundary Value Tests</h3>
411
<p>SQLite defines certain <a href="limits.html">limits</a> on its operation, such as the
412
maximum number of columns in a table, the maximum length of an
413
SQL statement, or the maximum value of an integer. The TCL and TH3 test
414
suites both contains numerous tests that push SQLite right to the edge
415
of its defined limits and verify that it performs correctly for
416
all allowed values. Additional tests go beyond the defined limits
417
and verify that SQLite correctly returns errors. The source code
418
contains <a href="testing.html#testcase">testcase macros</a> to verify that both sides of each boundary
419
have been tested.</p>
421
<a name="regressiontesting"></a>
423
<h2>5.0 Regression Testing</h2>
425
<p>Whenever a bug is reported against SQLite, that bug is not considered
426
fixed until new test cases have been added to the TCL test suite which
427
would exhibit the bug in an unpatched version of SQLite. Over the years,
428
this has resulted in thousands and thousands of new tests being added
429
to the TCL test suite. These regression tests ensure that bugs that have
430
been fixed in the past are not reintroduced into future versions of
433
<a name="leakcheck"></a>
435
<h2>6.0 Automatic Resource Leak Detection</h2>
437
<p>Resource leak occurs when system resources
438
are allocated and never freed. The most troublesome resource leaks
439
in many applications are memory leaks - when memory is allocated using
440
malloc() but never released using free(). But other kinds of resources
441
can also be leaked: file descriptors, threads, mutexes, etc.</p>
443
<p>Both the TCL and TH3 test harnesses automatically track system
444
resources and report resource leaks on <u>every</u> test run.
445
No special configuration or setup is required. The test harnesses
446
are especially vigilant with regard to memory leaks. If a change
447
causes a memory leak, the test harnesses will recognize this
448
quickly. SQLite is designed to never leak memory, even after
449
an exception such as an OOM error or disk I/O error. The test
450
harnesses are zealous to enforce this.</p>
452
<a name="coverage"></a>
454
<h2>7.0 Test Coverage</h2>
456
<p>The SQLite core has 100% branch test coverage under <a href="th3.html">TH3</a> as of
457
2009-07-25, in its default configuration as measured by
458
<a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
459
utility on SuSE Linux 10.1 on x86 hardware with the GCC 4.0.1 compiler.</p>
461
<p>The "SQLite core" in the previous paragraph excludes the
462
operating-system dependent <a href="vfs.html">VFS</a> backends, since it is
463
not possible to write cross-platform tests for those modules. Extensions
464
such as FTS3 and RTree are also excluded from the analysis.</p>
466
<a name="stmtvbr"></a>
468
<h3>7.1 Statement versus branch coverage</h3>
470
<p>There are many ways to measure test coverage. The most popular
471
metric is "statement coverage". When you hear someone say that their
472
program as "XX% test coverage" without further explanation, they usually
473
mean statement coverage. Statement coverage measures what percentage
474
of lines of code are executed at least once by the test suite.</p>
476
<p>Branch coverage is more rigorous than statement coverage. Branch
477
coverage measures the number of machine-code branch instructions that
478
are evaluated at least once on both directions.</p>
480
<p>To illustrate the difference between statement coverage and
481
branch coverage, consider the following hypothetical
485
if( a>b && c!=25 ){ d++; }
488
<p>Such a line of C code might generate a dozen separate machine code
489
instructions. If any one of those instructions is ever evaluated, then
490
we say that the statement has been tested. So, for example, it might
491
be the case that the conditional expression is
492
always false and the "d" variable is
493
never incremented. Even so, statement coverage counts this line of
494
code as having been tested.</p>
496
<p>Branch coverage is more strict. With branch coverage, each test and
497
each subblock within the statement is considered separately. In order
498
to achieve 100% branch coverage in the example above, there must be at
499
least three test cases:</p>
507
<p>Any one of the above test cases would provide 100% statement coverage
508
but all three are required for 100% branch coverage. Generally speaking,
509
100% branch coverage implies 100% statement coverage, but the converse is
510
not true. To reemphasize, the
511
<a href="th3.html">TH3</a> test harness for SQLite provides the stronger form of
512
test coverage - 100% branch test coverage.</p>
514
<a name="defensivecode"></a>
516
<h3>7.2 Coverage testing of defensive code</h3>
518
<p>A well-written C program will typically contain some defensive tests
519
which in practice are always true or always false. This leads to a
520
programming dilemma: Does one remove defensive code in order to obtain
521
100% branch coverage?</p>
523
<p>In SQLite, the answer to the previous question is "no".
524
For testing purposes, the SQLite source code defines
525
macros called ALWAYS() and NEVER(). The ALWAYS() macro
527
which are expected to always evaluate as true and NEVER() surrounds
528
conditions that are always evaluated to false. These macros serve as
529
comments to indicate that the conditions are defensive code.
530
For standard builds, these macros are pass-throughs:</p>
533
#define ALWAYS(X) (X)
537
<p>During most testing, however, these macros will throw an assertion
538
fault if their argument does not have the expected truth value. This
539
alerts the developers quickly to incorrect design assumptions.
542
#define ALWAYS(X) ((X)?1:assert(0),0)
543
#define NEVER(X) ((X)?assert(0),1:0)
546
<p>When measuring test coverage, these macros are defined to be constant
547
truth values so that they do not generate assembly language branch
548
instructions, and hence do not come into play when calculating the
552
#define ALWAYS(X) (1)
556
<p>The test suite is designed to be run three times, once for each of
557
the ALWAYS() and NEVER() definitions shown above. All three test runs
558
should yield exactly the same result. There is a run-time test using
559
the <a href="c3ref/test_control.html">sqlite3_test_control</a>(<a href="c3ref/c_testctrl_always.html">SQLITE_TESTCTRL_ALWAYS</a>, ...) interface that
560
can be used to verify that the macros are correctly set to the first
561
form (the pass-through form) for deployment.</p>
563
<a name="testcase"></a>
565
<h3>7.3 Forcing coverage of boundary values and boolean vector tests</h3>
567
<p>Another macro used in conjunction with test coverage measurement is
568
the <tt>testcase()</tt> macro. The argument is a condition for which
569
we want test cases that evaluate to both true and false.
570
In non-coverage builds (that is to say, in release builds) the
571
<tt>testcase()</tt> macro is a no-op:</p>
577
<p>But in a coverage measuring build, the <tt>testcase()</tt> macro
578
generates code that evaluates the conditional expression in its argument.
579
Then during analysis, a check
580
is made to ensure tests exist that evaluate the conditional to both true
581
and false. <tt>Testcase()</tt> macros are used, for example, to help verify
582
that boundary values are tested. For example:</p>
587
if( a>b && c!=25 ){ d++; }
590
<p>Testcase macros are also used when two or more cases of a switch
591
statement go to the same block of code, to make sure that the code was
592
reached for all cases:</p>
598
testcase( op==OP_Add );
599
testcase( op==OP_Subtract );
607
<p>For bitmask tests, <tt>testcase()</tt> macros are used to verify that every
608
bit of the bitmask affects the test. For example, in the following block
609
of code, the condition is true if the mask contains either of two bits
610
indicating either a MAIN_DB or a TEMP_DB is being opened.
611
The <tt>testcase()</tt>
612
macros that precede the if statement verify that both cases are tested:</p>
615
testcase( mask & SQLITE_OPEN_MAIN_DB );
616
testcase( mask & SQLITE_OPEN_TEMP_DB );
617
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
620
<p>The SQLite source code contains 669
621
uses of the <tt>testcase()</tt> macro.</p>
625
<h3>7.4 Branch coverage versus MC/DC</h3>
627
<p>Two methods of measuring test coverage were described above:
628
"statement" and "branch" coverage. There are many other test coverage
629
metrics besides these two. Another popular metric is "Modified
630
Condition/Decision Coverage" or MC/DC.
631
<a href="http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage">Wikipedia</a>
632
defines MC/DC as follows:</p>
635
<li> Each decision tries every possible outcome.
636
<li> Each condition in a decision takes on every possible outcome.
637
<li> Each entry and exit point is invoked.
638
<li> Each condition in a decision is shown to independently
639
affect the outcome of the decision.
642
<p>In the C programming language
643
where <b><tt>&&</tt></b> and <b><tt>||</tt></b>
644
are "short-circuit" operators, MC/DC and branch coverage are very nearly
645
the same thing. The primary difference is in boolean vector tests.
646
One can test for any of several bits in bit-vector and still obtain
647
100% branch test coverage even though the second element of MC/DC - the
648
requirement that each condition in a decision take on every possible outcome -
649
might not be satisfied.</p>
651
<p>SQLite uses <tt>testcase()</tt> macros as described in the previous
652
subsection to make sure that every condition in a bit-vector decision takes
653
on every possible outcome. In this way, SQLite also achieves 100% MC/DC
654
in addition to 100% branch coverage.</p>
656
<a name="thoughts1"></a>
658
<h3>7.5 Experience with full test coverage</h3>
660
<p>The developers of SQLite have found that full coverage testing is an
661
extremely productive method for preventing the introduction of new bugs
662
as the system evolves. Because every single branch
663
instruction in SQLite core code is covered by test cases, the developers
664
can be confident that changes they make in one part of the code
665
do not have unintended consequences in other parts of the code.
666
It would be extremely difficult maintain the quality of SQLite without such
669
<a name="dynamicanalysis"></a>
671
<h2>8.0 Dynamic Analysis</h2>
673
<p>Dynamic analysis refers to internal and external checks on the
674
SQLite code which are performed while the code is live and running.
675
Dynamic analysis has proven to be a great help in maintaining the
676
quality of SQLite.</p>
678
<a name="asserts"></a>
682
<p>The SQLite core contains 3432 <tt>assert()</tt>
683
statements that verify function preconditions and postconditions and
684
loop invariants. Assert() is a macro which is a standard part of
685
ANSI-C. The argument is a boolean value that is assumed to always be
686
true. If the assertion is false, the program prints an error message
689
<p>Assert() macros are disabled by compiling with the NDEBUG macro defined.
690
In most systems, asserts are enabled by default. But in SQLite, the
691
asserts are so numerous and are in such performance critical places, that
692
the database engine runs about three times slower when asserts are enabled.
693
Hence, the default (production) build of SQLite disables asserts.
694
Assert statements are only enabled when SQLite is compiled with the
695
SQLITE_DEBUG preprocessor macro defined.</p>
697
<a name="valgrind"></a>
699
<h3>8.2 Valgrind</h3>
701
<p><a href="http://valgrind.org/">Valgrind</a> is perhaps the most amazing
702
and useful developer tool in the world. Valgrind is a simulator - it simulates
703
an x86 running a Linux binary. (Ports of Valgrind for platforms other
704
than Linux are in development, but as of this writing, Valgrind only
705
works reliably on Linux, which in the opinion of the SQLite developers
706
means that Linux should be the preferred platform for all software development.)
707
As Valgrind runs a Linux binary, it looks for all kinds of interesting
708
errors such as array overruns, reading from uninitialized memory,
709
stack overflows, memory leaks, and so forth. Valgrind finds problems
710
that can easily slip through all of the other tests run against SQLite.
711
And, when Valgrind does find an error, it can dump the developer directly
712
into a symbolic debugger at the exact point where the error occur, to
713
facilitate a quick fix.</p>
715
<p>Because it is a simulator, running a binary in Valgrind is slower than
716
running it on native hardware. (To a first approximation, an application
717
running in Valgrind on a workstation will perform about the same as it
718
would running natively on a smartphone.) So it is impractical to run the full
719
SQLite test suite through Valgrind. However, the veryquick tests and
720
the coverage of the TH3 tests are run through Valgrind prior to every
723
<a name="memtesting"></a>
727
<p>SQLite contains a pluggable
728
<a href="malloc.html">memory allocation subsystem</a>.
729
The default implementation uses system malloc() and free().
730
However, if SQLite is compiled with <a href="compile.html#memdebug">SQLITE_MEMDEBUG</a>, an alternative
731
memory allocation wrapper (<a href="malloc.html#memdebug">memsys2</a>)
732
is inserted that looks for memory allocation
733
errors at run-time. The memsys2 wrapper checks for memory leaks, of
734
course, but also looks for buffer overruns, uses of uninitialized memory,
735
and attempts to use memory after it has been freed. These same checks
736
are also done by valgrind (and, indeed, Valgrind does them better)
737
but memsys2 has the advantage of being much faster than Valgrind, which
738
means the checks can be done more often and for longer tests.</p>
740
<a name="mutextesting"></a>
742
<h3>8.4 Mutex Asserts</h3>
744
<p>SQLite contains a pluggable mutex subsystem. Depending on
745
compile-time options, the default mutex system contains interfaces
746
<a href="c3ref/mutex_held.html">sqlite3_mutex_held()</a> and <a href="c3ref/mutex_held.html">sqlite3_mutex_notheld()</a> that detect
747
whether or not a particular mutex is held by the calling thread.
748
These two interfaces are used extensively within assert() statements
749
in SQLite to verify mutexes are held and released at all the right
750
moments, in order to double-check that SQLite does work correctly
751
in multi-threaded applications.</p>
753
<a name="journaltest"></a>
755
<h3>8.5 Journal Tests</h3>
757
<p>One of the things that SQLite does to ensure that transactions
758
are atomic across system crashes and power failures is to write
759
all changes into the rollback journal file prior to changing the
760
database. The TCL test harness contains an alternative
761
<a href="vfs.html">OS backend</a> implementation that helps to
762
verify this is occurring correctly. The "journal-test VFS" monitors
763
all disk I/O traffic between the database file and rollback journal,
764
checking to make sure that nothing is written into the database
765
file which has not first been written and synced to the rollback journal.
766
If any discrepancies are found, an assertion fault is raised.</p>
768
<p>The journal tests are an additional double-check over and above
769
the crash tests to make sure that SQLite transactions will be atomic
770
across system crashes and power failures.</p>
772
<a name="intoverflow"></a>
774
<h3>8.6 Signed-Integer Overflow Checks</h3>
776
<p>The various C language standards say that the signed-integer overflow
777
behavior is undefined. In other words, when you add a value to a signed
778
integer such that the result is too large to fit in that integer, the
779
value does not necessarily wrap around to a negative number, as most
780
programmers expect. It might do that. But it might do something completely
781
different. See, for example,
782
<a href="http://thiemonagel.de/2010/01/signed-integer-overflow/">here</a>
783
and <a href="http://blog.regehr.org/archives/482">here</a>. Even the
784
same compiler might do something different with signed integer overflow
785
in different places in the code or at different optimizations settings.</p>
787
<p>SQLite never overflows a signed integer.
788
To verify this, the test suites are run at least once when compiled with
789
the -ftrapv option to GCC. The -ftrapv option causes GCC to generate code
790
that will panic() on a signed integer overflow. In addition, there are
791
many test cases the strive to provoke integer overflows
792
using boundary value calculations such as
793
"<b>SELECT -1*(-9223372036854775808);</b>".
796
<a name="disopttest"></a>
798
<h2>9.0 Disabled Optimization Tests</h2>
800
<p>The <a href="c3ref/test_control.html">sqlite3_test_control</a>(<a href="c3ref/c_testctrl_always.html">SQLITE_TESTCTRL_OPTIMIZATIONS</a>, ...) interface
801
allows selected SQL statement optimizations to be disabled at run-time.
802
SQLite should always generate exactly the same answer with optimizations
803
enabled and with optimizations disabled; the answer simply arrives quicker
804
with the optimizations turned on. So in a production environment, one always
805
leaves the optimizations turned on (the default setting).</p>
807
<p>One verification technique used on SQLite is to run an entire test suite
808
twice, once with optimizations left on and a second time with optimizations
809
turned off, and verify that the same output is obtained both times. This
810
shows that the optimizations do not introduce errors.</p>
812
<p>Not all test cases can be handled this way. Some test cases check
813
to verify that the optimizations really are reducing the amount of
814
computation by counting the number of disk accesses, sort operations,
815
full-scan steps, or other processing steps that occur during queries.
816
Those test cases will appear to fail when optimizations are disabled.
817
But the majority of test cases simply check that the correct answer
818
was obtained, and all of those cases can be run successfully with and
819
without the optimizations, in order to show that the optimizations do not
820
cause malfunctions.</p>
823
<a name="staticanalysis"></a>
825
<h2>10.0 Static Analysis</h2>
827
<p>Static analysis means analyzing code at or before compile-time to
828
check for correctness. Static analysis includes looking at compiler
829
warning messages and running the code through more in-depth
830
analysis engines such as the
831
<a href="http://clang-analyzer.llvm.org/">Clang Static Analyzer</a>.
832
SQLite compiles without warnings on GCC using the -Wall and -Wextra flags.
833
And the Clang Static Analyzer finds nothing wrong with SQLite either.
834
VC++ often will generate a number of warnings from SQLite source code,
835
but the experience of SQLite developers is that VC++ warnings are of
836
lower quality and can be safely ignored. Users are encouraged not
837
to stress over VC++ warnings.</p>
839
<p>Static analysis has not proven to be especially helpful in finding
840
bugs in SQLite. Static analysis has found a few bugs in SQLite, but
841
those are the exceptions. More bugs have been
842
introduced into SQLite while trying to get it to compile without
843
warnings than have been found by static analysis.
845
<p>Our experience, then, is that static analysis is not especially helpful
846
to maintaining high code quality and it is therefore deemphasized
847
in SQLite development and maintenance.</p>
849
<a name="summary"></a>
851
<h2>11.0 Summary</h2>
853
<p>SQLite is open source. This gives many people the idea that
854
it is not well tested as commercial software and is perhaps unreliable.
855
But that impression is false.
856
SQLite has exhibited very high reliability in the field and
857
a very low defect rate, especially considering how rapidly it is evolving.
858
The quality of SQLite is achieved in part by careful code design and
859
implementation. But extensive testing also plays a vital role in
860
maintaining and improving the quality of SQLite. This document has
861
summarized the testing procedures that every release of SQLite undergoes
862
with the hopes of inspiring the reader to understand that SQLite is
863
suitable for use in mission-critical applications.</p>
added
removed
testing.html
