1
+++++++++++++++++++++++++++++++
2
Writing Python Regression Tests
3
+++++++++++++++++++++++++++++++
5
:Author: Skip Montanaro
6
:Contact: skip@pobox.com
11
If you add a new module to Python or modify the functionality of an existing
12
module, you should write one or more test cases to exercise that new
13
functionality. There are different ways to do this within the regression
14
testing facility provided with Python; any particular test should use only
15
one of these options. Each option requires writing a test module using the
16
conventions of the selected option:
18
- unittest_ based tests
19
- doctest_ based tests
20
- "traditional" Python test modules
22
Regardless of the mechanics of the testing approach you choose,
23
you will be writing unit tests (isolated tests of functions and objects
24
defined by the module) using white box techniques. Unlike black box
25
testing, where you only have the external interfaces to guide your test case
26
writing, in white box testing you can see the code being tested and tailor
27
your test cases to exercise it more completely. In particular, you will be
28
able to refer to the C and Python code in the CVS repository when writing
29
your regression test cases.
31
.. _unittest: http://www.python.org/doc/current/lib/module-unittest.html
32
.. _doctest: http://www.python.org/doc/current/lib/module-doctest.html
36
The unittest_ framework is based on the ideas of unit testing as espoused
37
by Kent Beck and the `Extreme Programming`_ (XP) movement. The specific
38
interface provided by the framework is tightly based on the JUnit_
39
Java implementation of Beck's original SmallTalk test framework. Please
40
see the documentation of the unittest_ module for detailed information on
41
the interface and general guidelines on writing unittest-based tests.
43
The test_support helper module provides a function for use by
44
unittest-based tests in the Python regression testing framework,
45
``run_unittest()``. This is the primary way of running tests in the
46
standard library. You can pass it any number of the following:
48
- classes derived from or instances of ``unittest.TestCase`` or
49
``unittest.TestSuite``. These will be handed off to unittest for
50
converting into a proper TestSuite instance.
52
- a string; this must be a key in sys.modules. The module associated with
53
that string will be scanned by ``unittest.TestLoader.loadTestsFromModule``.
54
This is usually seen as ``test_support.run_unittest(__name__)`` in a test
55
module's ``test_main()`` function. This has the advantage of picking up
56
new tests automatically, without you having to add each new test case
59
All test methods in the Python regression framework have names that
60
start with "``test_``" and use lower-case names with words separated with
63
Test methods should *not* have docstrings! The unittest module prints
64
the docstring if there is one, but otherwise prints the function name
65
and the full class name. When there's a problem with a test, the
66
latter information makes it easier to find the source for the test
69
All unittest-based tests in the Python test suite use boilerplate that
70
looks like this (with minor variations)::
73
from test import test_support
75
class MyTestCase1(unittest.TestCase):
77
# Define setUp and tearDown only if needed
80
unittest.TestCase.setUp(self)
81
... additional initialization...
84
... additional finalization...
85
unittest.TestCase.tearDown(self)
87
def test_feature_one(self):
89
...unit test for feature one...
91
def test_feature_two(self):
93
...unit test for feature two...
97
class MyTestCase2(unittest.TestCase):
98
...same structure as MyTestCase1...
103
test_support.run_unittest(__name__)
105
if __name__ == "__main__":
108
This has the advantage that it allows the unittest module to be used
109
as a script to run individual tests as well as working well with the
112
.. _Extreme Programming: http://www.extremeprogramming.org/
113
.. _JUnit: http://www.junit.org/
117
Tests written to use doctest_ are actually part of the docstrings for
118
the module being tested. Each test is written as a display of an
119
interactive session, including the Python prompts, statements that would
120
be typed by the user, and the output of those statements (including
121
tracebacks, although only the exception msg needs to be retained then).
122
The module in the test package is simply a wrapper that causes doctest
123
to run over the tests in the module. The test for the difflib module
124
provides a convenient example::
127
from test import test_support
128
test_support.run_doctest(difflib)
130
If the test is successful, nothing is written to stdout (so you should not
131
create a corresponding output/test_difflib file), but running regrtest
132
with -v will give a detailed report, the same as if passing -v to doctest.
134
A second argument can be passed to run_doctest to tell doctest to search
135
``sys.argv`` for -v instead of using test_support's idea of verbosity. This
136
is useful for writing doctest-based tests that aren't simply running a
137
doctest'ed Lib module, but contain the doctests themselves. Then at
138
times you may want to run such a test directly as a doctest, independent
139
of the regrtest framework. The tail end of test_descrtut.py is a good
142
def test_main(verbose=None):
143
from test import test_support, test_descrtut
144
test_support.run_doctest(test_descrtut, verbose)
146
if __name__ == "__main__":
149
If run via regrtest, ``test_main()`` is called (by regrtest) without
150
specifying verbose, and then test_support's idea of verbosity is used. But
151
when run directly, ``test_main(1)`` is called, and then doctest's idea of
154
See the documentation for the doctest module for information on
155
writing tests using the doctest framework.
157
"traditional" Python test modules
158
---------------------------------
159
The mechanics of how the "traditional" test system operates are fairly
160
straightforward. When a test case is run, the output is compared with the
161
expected output that is stored in .../Lib/test/output. If the test runs to
162
completion and the actual and expected outputs match, the test succeeds, if
163
not, it fails. If an ``ImportError`` or ``test_support.TestSkipped`` error
164
is raised, the test is not run.
168
If you are writing test cases for module spam, you need to create a file
169
in .../Lib/test named test_spam.py. In addition, if the tests are expected
170
to write to stdout during a successful run, you also need to create an
171
expected output file in .../Lib/test/output named test_spam ("..."
172
represents the top-level directory in the Python source tree, the directory
173
containing the configure script). If needed, generate the initial version
174
of the test output file by executing::
176
./python Lib/test/regrtest.py -g test_spam.py
178
from the top-level directory.
180
Any time you modify test_spam.py you need to generate a new expected
181
output file. Don't forget to desk check the generated output to make sure
182
it's really what you expected to find! All in all it's usually better
183
not to have an expected-out file (note that doctest- and unittest-based
186
To run a single test after modifying a module, simply run regrtest.py
187
without the -g flag::
189
./python Lib/test/regrtest.py test_spam.py
191
While debugging a regression test, you can of course execute it
192
independently of the regression testing framework and see what it prints::
194
./python Lib/test/test_spam.py
196
To run the entire test suite:
198
- [UNIX, + other platforms where "make" works] Make the "test" target at the
203
- [WINDOWS] Run rt.bat from your PCBuild directory. Read the comments at
204
the top of rt.bat for the use of special -d, -O and -q options processed
207
- [OTHER] You can simply execute the two runs of regrtest (optimized and
208
non-optimized) directly::
210
./python Lib/test/regrtest.py
211
./python -O Lib/test/regrtest.py
213
But note that this way picks up whatever .pyc and .pyo files happen to be
214
around. The makefile and rt.bat ways run the tests twice, the first time
215
removing all .pyc and .pyo files from the subtree rooted at Lib/.
217
Test cases generate output based upon values computed by the test code.
218
When executed, regrtest.py compares the actual output generated by executing
219
the test case with the expected output and reports success or failure. It
220
stands to reason that if the actual and expected outputs are to match, they
221
must not contain any machine dependencies. This means your test cases
222
should not print out absolute machine addresses (e.g. the return value of
223
the id() builtin function) or floating point numbers with large numbers of
224
significant digits (unless you understand what you are doing!).
227
Test Case Writing Tips
228
======================
229
Writing good test cases is a skilled task and is too complex to discuss in
230
detail in this short document. Many books have been written on the subject.
231
I'll show my age by suggesting that Glenford Myers' `"The Art of Software
232
Testing"`_, published in 1979, is still the best introduction to the subject
233
available. It is short (177 pages), easy to read, and discusses the major
234
elements of software testing, though its publication predates the
235
object-oriented software revolution, so doesn't cover that subject at all.
236
Unfortunately, it is very expensive (about $100 new). If you can borrow it
237
or find it used (around $20), I strongly urge you to pick up a copy.
239
The most important goal when writing test cases is to break things. A test
240
case that doesn't uncover a bug is much less valuable than one that does.
241
In designing test cases you should pay attention to the following:
243
* Your test cases should exercise all the functions and objects defined
244
in the module, not just the ones meant to be called by users of your
245
module. This may require you to write test code that uses the module
246
in ways you don't expect (explicitly calling internal functions, for
247
example - see test_atexit.py).
249
* You should consider any boundary values that may tickle exceptional
250
conditions (e.g. if you were writing regression tests for division,
251
you might well want to generate tests with numerators and denominators
252
at the limits of floating point and integer numbers on the machine
253
performing the tests as well as a denominator of zero).
255
* You should exercise as many paths through the code as possible. This
256
may not always be possible, but is a goal to strive for. In
257
particular, when considering if statements (or their equivalent), you
258
want to create test cases that exercise both the true and false
259
branches. For loops, you should create test cases that exercise the
260
loop zero, one and multiple times.
262
* You should test with obviously invalid input. If you know that a
263
function requires an integer input, try calling it with other types of
264
objects to see how it responds.
266
* You should test with obviously out-of-range input. If the domain of a
267
function is only defined for positive integers, try calling it with a
270
* If you are going to fix a bug that wasn't uncovered by an existing
271
test, try to write a test case that exposes the bug (preferably before
274
* If you need to create a temporary file, you can use the filename in
275
``test_support.TESTFN`` to do so. It is important to remove the file
276
when done; other tests should be able to use the name without cleaning
279
.. _"The Art of Software Testing":
280
http://www.amazon.com/exec/obidos/ISBN=0471043281
282
Regression Test Writing Rules
283
=============================
284
Each test case is different. There is no "standard" form for a Python
285
regression test case, though there are some general rules (note that
286
these mostly apply only to the "classic" tests; unittest_- and doctest_-
287
based tests should follow the conventions natural to those frameworks)::
289
* If your test case detects a failure, raise ``TestFailed`` (found in
290
``test.test_support``).
292
* Import everything you'll need as early as possible.
294
* If you'll be importing objects from a module that is at least
295
partially platform-dependent, only import those objects you need for
296
the current test case to avoid spurious ``ImportError`` exceptions
297
that prevent the test from running to completion.
299
* Print all your test case results using the ``print`` statement. For
300
non-fatal errors, print an error message (or omit a successful
301
completion print) to indicate the failure, but proceed instead of
302
raising ``TestFailed``.
304
* Use ``assert`` sparingly, if at all. It's usually better to just print
305
what you got, and rely on regrtest's got-vs-expected comparison to
306
catch deviations from what you expect. ``assert`` statements aren't
307
executed at all when regrtest is run in -O mode; and, because they
308
cause the test to stop immediately, can lead to a long & tedious
309
test-fix, test-fix, test-fix, ... cycle when things are badly broken
310
(and note that "badly broken" often includes running the test suite
311
for the first time on new platforms or under new implementations of
316
There is a test_support module in the test package you can import for
317
your test case. Import this module using either::
319
import test.test_support
323
from test import test_support
325
test_support provides the following useful objects:
327
* ``TestFailed`` - raise this exception when your regression test detects
330
* ``TestSkipped`` - raise this if the test could not be run because the
331
platform doesn't offer all the required facilities (like large
332
file support), even if all the required modules are available.
334
* ``ResourceDenied`` - this is raised when a test requires a resource that
335
is not available. Primarily used by 'requires'.
337
* ``verbose`` - you can use this variable to control print output. Many
338
modules use it. Search for "verbose" in the test_*.py files to see
341
* ``forget(module_name)`` - attempts to cause Python to "forget" that it
342
loaded a module and erase any PYC files.
344
* ``is_resource_enabled(resource)`` - Returns a boolean based on whether
345
the resource is enabled or not.
347
* ``requires(resource [, msg])`` - if the required resource is not
348
available the ResourceDenied exception is raised.
350
* ``verify(condition, reason='test failed')``. Use this instead of::
352
assert condition[, reason]
354
``verify()`` has two advantages over ``assert``: it works even in -O
355
mode, and it raises ``TestFailed`` on failure instead of
358
* ``is_jython`` - true if the interpreter is Jython, false otherwise.
360
* ``TESTFN`` - a string that should always be used as the filename when
361
you need to create a temp file. Also use ``try``/``finally`` to
362
ensure that your temp files are deleted before your test completes.
363
Note that you cannot unlink an open file on all operating systems, so
364
also be sure to close temp files before trying to unlink them.
366
* ``sortdict(dict)`` - acts like ``repr(dict.items())``, but sorts the
367
items first. This is important when printing a dict value, because
368
the order of items produced by ``dict.items()`` is not defined by the
371
* ``findfile(file)`` - you can call this function to locate a file
372
somewhere along sys.path or in the Lib/test tree - see
373
test_ossaudiodev.py for an example of its use.
375
* ``fcmp(x,y)`` - you can call this function to compare two floating
376
point numbers when you expect them to only be approximately equal
377
withing a fuzz factor (``test_support.FUZZ``, which defaults to 1e-6).
379
* ``check_syntax_error(testcase, statement)`` - make sure that the
380
statement is *not* correct Python syntax.
383
Some Non-Obvious regrtest Features
384
==================================
385
* Automagic test detection: When you create a new test file
386
test_spam.py, you do not need to modify regrtest (or anything else)
387
to advertise its existence. regrtest searches for and runs all
388
modules in the test directory with names of the form test_xxx.py.
390
* Miranda output: If, when running test_spam.py, regrtest does not
391
find an expected-output file test/output/test_spam, regrtest
392
pretends that it did find one, containing the single line
396
This allows new tests that don't expect to print anything to stdout
397
to not bother creating expected-output files.
399
* Two-stage testing: To run test_spam.py, regrtest imports test_spam
400
as a module. Most tests run to completion as a side-effect of
401
getting imported. After importing test_spam, regrtest also executes
402
``test_spam.test_main()``, if test_spam has a ``test_main`` attribute.
403
This is rarely required with the "traditional" Python tests, and
404
you shouldn't create a module global with name test_main unless
405
you're specifically exploiting this gimmick. This usage does
406
prove useful with unittest-based tests as well, however; defining
407
a ``test_main()`` which is run by regrtest and a script-stub in the
408
test module ("``if __name__ == '__main__': test_main()``") allows
409
the test to be used like any other Python test and also work
410
with the unittest.py-as-a-script approach, allowing a developer
411
to run specific tests from the command line.
added
removed
Lib/test/README
