4
If you're on a Unix-like system, you can ask for colorized output. The test
5
runner emits terminal control sequences to highlight important pieces of
6
information (such as the names of failing tests) in different colors.
8
>>> import os.path, sys
9
>>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
11
... '--path', directory_with_tests,
12
... '--tests-pattern', '^sampletestsf?$',
15
>>> from zope import testrunner
17
Since it wouldn't be a good idea to have terminal control characters in a
18
test file, let's wrap sys.stdout in a simple terminal interpreter
21
>>> class Terminal(object):
22
... _color_regexp = re.compile('\033[[]([0-9;]*)m')
23
... _colors = {'0': 'normal', '1': 'bold', '30': 'black', '31': 'red',
24
... '32': 'green', '33': 'yellow', '34': 'blue',
25
... '35': 'magenta', '36': 'cyan', '37': 'grey'}
26
... def __init__(self, stream):
27
... self._stream = stream
28
... def __getattr__(self, attr):
29
... return getattr(self._stream, attr)
32
... def write(self, text):
33
... if '\033[' in text:
34
... text = self._color_regexp.sub(self._color, text)
35
... self._stream.write(text)
36
... def writelines(self, lines):
37
... for line in lines:
39
... def _color(self, match):
41
... for number in match.group(1).split(';'):
42
... colorstring += self._colors.get(number, '?')
43
... return colorstring + '}'
45
>>> real_stdout = sys.stdout
46
>>> sys.stdout = Terminal(sys.stdout)
52
A successful test run soothes the developer with warm green colors:
54
>>> sys.argv = 'test --layer 122 -c'.split()
55
>>> testrunner.run_internal(defaults)
56
{normal}Running samplelayers.Layer122 tests:{normal}
57
Set up samplelayers.Layer1 in {green}0.000{normal} seconds.
58
Set up samplelayers.Layer12 in {green}0.000{normal} seconds.
59
Set up samplelayers.Layer122 in {green}0.000{normal} seconds.
60
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.007{normal} seconds.{normal}
61
{normal}Tearing down left over layers:{normal}
62
Tear down samplelayers.Layer122 in {green}0.000{normal} seconds.
63
Tear down samplelayers.Layer12 in {green}0.000{normal} seconds.
64
Tear down samplelayers.Layer1 in {green}0.000{normal} seconds.
71
A failed test run highlights the failures in red:
73
>>> sys.argv = 'test -c --tests-pattern ^sampletests(f|_e|_f)?$ '.split()
74
>>> testrunner.run_internal(defaults)
75
{normal}Running samplelayers.Layer1 tests:{normal}
76
Set up samplelayers.Layer1 in {green}0.000{normal} seconds.
77
{normal} Ran {green}9{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.001{normal} seconds.{normal}
78
{normal}Running samplelayers.Layer11 tests:{normal}
79
Set up samplelayers.Layer11 in {green}0.000{normal} seconds.
80
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.007{normal} seconds.{normal}
81
{normal}Running samplelayers.Layer111 tests:{normal}
82
Set up samplelayers.Layerx in {green}0.000{normal} seconds.
83
Set up samplelayers.Layer111 in {green}0.000{normal} seconds.
84
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.008{normal} seconds.{normal}
85
{normal}Running samplelayers.Layer112 tests:{normal}
86
Tear down samplelayers.Layer111 in {green}0.000{normal} seconds.
87
Set up samplelayers.Layer112 in {green}0.000{normal} seconds.
88
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.008{normal} seconds.{normal}
89
{normal}Running samplelayers.Layer12 tests:{normal}
90
Tear down samplelayers.Layer112 in {green}0.000{normal} seconds.
91
Tear down samplelayers.Layerx in {green}0.000{normal} seconds.
92
Tear down samplelayers.Layer11 in {green}0.000{normal} seconds.
93
Set up samplelayers.Layer12 in {green}0.000{normal} seconds.
94
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.007{normal} seconds.{normal}
95
{normal}Running samplelayers.Layer121 tests:{normal}
96
Set up samplelayers.Layer121 in {green}0.000{normal} seconds.
97
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.007{normal} seconds.{normal}
98
{normal}Running samplelayers.Layer122 tests:{normal}
99
Tear down samplelayers.Layer121 in {green}0.000{normal} seconds.
100
Set up samplelayers.Layer122 in {green}0.000{normal} seconds.
101
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.008{normal} seconds.{normal}
102
{normal}Running zope.testrunner.layer.UnitTests tests:{normal}
103
Tear down samplelayers.Layer122 in {green}0.000{normal} seconds.
104
Tear down samplelayers.Layer12 in {green}0.000{normal} seconds.
105
Tear down samplelayers.Layer1 in {green}0.000{normal} seconds.
106
Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
109
{boldred}Failure in test eek (sample2.sampletests_e){normal}
110
Failed doctest test for sample2.sampletests_e.eek
111
File "testrunner-ex/sample2/sampletests_e.py", line 28, in eek
113
----------------------------------------------------------------------
114
{normal}File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}30{normal}, in {boldcyan}sample2.sampletests_e.eek{normal}
118
{red} Traceback (most recent call last):{normal}
119
{red} File ".../doctest.py", line 1356, in __run{normal}
120
{red} compileflags, 1)...{normal}
121
{red} File "<doctest sample2.sampletests_e.eek[0]>", line 1, in ?{normal}
123
{red} File "testrunner-ex/sample2/sampletests_e.py", line 19, in f{normal}
125
{red} File "testrunner-ex/sample2/sampletests_e.py", line 24, in g{normal}
126
{red} x = y + 1{normal}
127
{red} - __traceback_info__: I don't know what Y should be.{normal}
128
{red} NameError: global name 'y' is not defined{normal}
132
{boldred}Error in test test3 (sample2.sampletests_e.Test){normal}
133
Traceback (most recent call last):
134
{normal} File "{boldblue}unittest.py{normal}", line {boldred}260{normal}, in {boldcyan}run{normal}
135
{cyan} testMethod(){normal}
136
{normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}43{normal}, in {boldcyan}test3{normal}
138
{normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}19{normal}, in {boldcyan}f{normal}
140
{normal} File "{boldblue}testrunner-ex/sample2/sampletests_e.py{normal}", line {boldred}24{normal}, in {boldcyan}g{normal}
141
{cyan} x = y + 1{normal}
142
{red} - __traceback_info__: I don't know what Y should be.{normal}
143
{red}NameError: global name 'y' is not defined{normal}
147
{boldred}Failure in test testrunner-ex/sample2/e.txt{normal}
148
Failed doctest test for e.txt
149
File "testrunner-ex/sample2/e.txt", line 0
151
----------------------------------------------------------------------
152
{normal}File "{boldblue}testrunner-ex/sample2/e.txt{normal}", line {boldred}4{normal}, in {boldcyan}e.txt{normal}
156
{red} Traceback (most recent call last):{normal}
157
{red} File ".../doctest.py", line 1356, in __run{normal}
158
{red} compileflags, 1)...{normal}
159
{red} File "<doctest e.txt[1]>", line 1, in ?{normal}
161
{red} File "<doctest e.txt[0]>", line 2, in f{normal}
162
{red} return x{normal}
163
{red} NameError: global name 'x' is not defined{normal}
167
{boldred}Failure in test test (sample2.sampletests_f.Test){normal}
168
Traceback (most recent call last):
169
{normal} File "{boldblue}unittest.py{normal}", line {boldred}260{normal}, in {boldcyan}run{normal}
170
{cyan} testMethod(){normal}
171
{normal} File "{boldblue}testrunner-ex/sample2/sampletests_f.py{normal}", line {boldred}21{normal}, in {boldcyan}test{normal}
172
{cyan} self.assertEqual(1,0){normal}
173
{normal} File "{boldblue}unittest.py{normal}", line {boldred}333{normal}, in {boldcyan}failUnlessEqual{normal}
174
{cyan} raise self.failureException, \{normal}
175
{red}AssertionError: 1 != 0{normal}
177
{normal} Ran {green}164{normal} tests with {boldred}3{normal} failures and {boldred}1{normal} errors in {green}0.045{normal} seconds.{normal}
178
{normal}Tearing down left over layers:{normal}
179
Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
180
{normal}Total: {green}329{normal} tests, {boldred}3{normal} failures, {boldred}1{normal} errors in {green}0.023{normal} seconds.{normal}
187
The expected and actual outputs of failed doctests are shown in different
190
>>> sys.argv = 'test --tests-pattern ^pledge$ -c'.split()
191
>>> _ = testrunner.run_internal(defaults)
192
{normal}Running zope.testrunner.layer.UnitTests tests:{normal}
193
Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
196
{boldred}Failure in test pledge (pledge){normal}
197
Failed doctest test for pledge.pledge
198
File "testrunner-ex/pledge.py", line 24, in pledge
200
----------------------------------------------------------------------
201
{normal}File testrunner-ex/pledge.py{normal}", line {boldred}26{normal}, in {boldcyan}pledge.pledge{normal}
203
{cyan} print_pledge(){normal}
205
{green} I give my pledge, as an earthling,{normal}
206
{green} to save, and faithfully, to defend from waste,{normal}
207
{green} the natural resources of my planet.{normal}
208
{green} It's soils, minerals, forests, waters, and wildlife.{normal}
209
{green} <BLANKLINE>{normal}
211
{red} I give my pledge, as and earthling,{normal}
212
{red} to save, and faithfully, to defend from waste,{normal}
213
{red} the natural resources of my planet.{normal}
214
{red} It's soils, minerals, forests, waters, and wildlife.{normal}
215
{red} <BLANKLINE>{normal}
217
{normal} Ran {green}1{normal} tests with {boldred}1{normal} failures and {green}0{normal} errors in {green}0.002{normal} seconds.{normal}
218
{normal}Tearing down left over layers:{normal}
219
Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
221
Diffs are highlighted so you can easily tell the context and the mismatches
224
>>> sys.argv = 'test --tests-pattern ^pledge$ --ndiff -c'.split()
225
>>> _ = testrunner.run_internal(defaults)
226
{normal}Running zope.testrunner.layer.UnitTests tests:{normal}
227
Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
230
{boldred}Failure in test pledge (pledge){normal}
231
Failed doctest test for pledge.pledge
232
File "testrunner-ex/pledge.py", line 24, in pledge
234
----------------------------------------------------------------------
235
{normal}File testrunner-ex/pledge.py{normal}", line {boldred}26{normal}, in {boldcyan}pledge.pledge{normal}
237
{cyan} print_pledge(){normal}
238
Differences (ndiff with -expected +actual):
239
{green} - I give my pledge, as an earthling,{normal}
240
{red} + I give my pledge, as and earthling,{normal}
241
{magenta} ? +{normal}
242
{normal} to save, and faithfully, to defend from waste,{normal}
243
{normal} the natural resources of my planet.{normal}
244
{normal} It's soils, minerals, forests, waters, and wildlife.{normal}
245
{normal} <BLANKLINE>{normal}
247
{normal} Ran {green}1{normal} tests with {boldred}1{normal} failures and {green}0{normal} errors in {green}0.003{normal} seconds.{normal}
248
{normal}Tearing down left over layers:{normal}
249
Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
251
Even test failures that have actual blank lines (as opposed to <BLANKLINE>) in
252
them are highlighted correctly.
254
>>> import zope.testrunner.formatter
255
>>> formatter = zope.testrunner.formatter.ColorfulOutputFormatter(None)
256
>>> formatter.print_doctest_failure("""\
257
... File "sometest.txt", line 221, in sometest.txt
261
... Output that contains
265
... Output that still contains
268
{normal} File "sometest.txt", line 221, in sometest.txt{normal}
272
{green} Output that contains{normal}
274
{green} blank lines.{normal}
276
{red} Output that still contains{normal}
278
{red} blank lines.{normal}
282
Timing individual tests
283
-----------------------
285
At very high verbosity levels you can see the time taken by each test
287
>>> sys.argv = 'test -u -t test_one.TestNotMuch -c -vvv'.split()
288
>>> testrunner.run_internal(defaults)
289
{normal}Running tests at level 1{normal}
290
{normal}Running zope.testrunner.layer.UnitTests tests:{normal}
291
Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
292
{normal} Running:{normal}
293
test_1 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
294
test_2 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
295
test_3 (sample1.sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
296
test_1 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
297
test_2 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
298
test_3 (sampletests.test_one.TestNotMuch) ({green}N.NNN s{normal})
299
{normal} Ran {green}6{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}N.NNN{normal} seconds.{normal}
300
{normal}Tearing down left over layers:{normal}
301
Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
305
If we had very slow tests we would see their times highlighted in a different color.
306
Instead of creating a test that waits 10 seconds, let's lower the slow test threshold
307
in the test runner to 0 seconds to make all of the tests seem slow.
309
>>> sys.argv = 'test -u -t test_one.TestNotMuch -c -vvv --slow-test 0'.split()
310
>>> testrunner.run_internal(defaults)
311
{normal}Running tests at level 1{normal}
312
{normal}Running zope.testrunner.layer.UnitTests tests:{normal}
313
Set up zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
314
{normal} Running:{normal}
315
test_1 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
316
test_2 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
317
test_3 (sample1.sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
318
test_1 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
319
test_2 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
320
test_3 (sampletests.test_one.TestNotMuch) ({boldmagenta}N.NNN s{normal})
321
{normal} Ran {green}6{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}N.NNN{normal} seconds.{normal}
322
{normal}Tearing down left over layers:{normal}
323
Tear down zope.testrunner.layer.UnitTests in {green}N.NNN{normal} seconds.
331
If -c or --color have been previously provided on the command line (perhaps by
332
a test runner wrapper script), but no colorized output is desired, the -C or
333
--no-color options will disable colorized output:
335
>>> sys.argv = 'test --layer 122 -c -C'.split()
336
>>> testrunner.run_internal(defaults)
337
Running samplelayers.Layer122 tests:
338
Set up samplelayers.Layer1 in 0.000 seconds.
339
Set up samplelayers.Layer12 in 0.000 seconds.
340
Set up samplelayers.Layer122 in 0.000 seconds.
341
Ran 26 tests with 0 failures and 0 errors in 0.007 seconds.
342
Tearing down left over layers:
343
Tear down samplelayers.Layer122 in 0.000 seconds.
344
Tear down samplelayers.Layer12 in 0.000 seconds.
345
Tear down samplelayers.Layer1 in 0.000 seconds.
348
>>> sys.argv = 'test --layer 122 -c --no-color'.split()
349
>>> testrunner.run_internal(defaults)
350
Running samplelayers.Layer122 tests:
351
Set up samplelayers.Layer1 in 0.000 seconds.
352
Set up samplelayers.Layer12 in 0.000 seconds.
353
Set up samplelayers.Layer122 in 0.000 seconds.
354
Ran 26 tests with 0 failures and 0 errors in 0.007 seconds.
355
Tearing down left over layers:
356
Tear down samplelayers.Layer122 in 0.000 seconds.
357
Tear down samplelayers.Layer12 in 0.000 seconds.
358
Tear down samplelayers.Layer1 in 0.000 seconds.
365
The --auto-color option will determine if stdout is a terminal that supports
366
colors, and only enable colorized output if so. Our ``Terminal`` wrapper
367
pretends it is a terminal, but the curses module will realize it isn't:
369
>>> sys.argv = 'test --layer 122 --auto-color'.split()
370
>>> testrunner.run_internal(defaults)
371
Running samplelayers.Layer122 tests:
372
Set up samplelayers.Layer1 in 0.000 seconds.
373
Set up samplelayers.Layer12 in 0.000 seconds.
374
Set up samplelayers.Layer122 in 0.000 seconds.
375
Ran 26 tests with 0 failures and 0 errors in 0.007 seconds.
376
Tearing down left over layers:
377
Tear down samplelayers.Layer122 in 0.000 seconds.
378
Tear down samplelayers.Layer12 in 0.000 seconds.
379
Tear down samplelayers.Layer1 in 0.000 seconds.
384
>>> class FakeCurses(object):
385
... class error(Exception):
387
... def setupterm(self):
389
... def tigetnum(self, attr):
390
... return dict(colors=8).get(attr, -2)
391
>>> sys.modules['curses'] = FakeCurses()
393
>>> sys.argv = 'test --layer 122 --auto-color'.split()
394
>>> testrunner.run_internal(defaults)
395
{normal}Running samplelayers.Layer122 tests:{normal}
396
Set up samplelayers.Layer1 in {green}0.000{normal} seconds.
397
Set up samplelayers.Layer12 in {green}0.000{normal} seconds.
398
Set up samplelayers.Layer122 in {green}0.000{normal} seconds.
399
{normal} Ran {green}26{normal} tests with {green}0{normal} failures and {green}0{normal} errors in {green}0.007{normal} seconds.{normal}
400
{normal}Tearing down left over layers:{normal}
401
Tear down samplelayers.Layer122 in {green}0.000{normal} seconds.
402
Tear down samplelayers.Layer12 in {green}0.000{normal} seconds.
403
Tear down samplelayers.Layer1 in {green}0.000{normal} seconds.
406
>>> del sys.modules['curses']
408
The real stdout is not a terminal in a doctest:
410
>>> sys.stdout = real_stdout
412
>>> sys.argv = 'test --layer 122 --auto-color'.split()
413
>>> testrunner.run_internal(defaults)
414
Running samplelayers.Layer122 tests:
415
Set up samplelayers.Layer1 in 0.000 seconds.
416
Set up samplelayers.Layer12 in 0.000 seconds.
417
Set up samplelayers.Layer122 in 0.000 seconds.
418
Ran 26 tests with 0 failures and 0 errors in 0.007 seconds.
419
Tearing down left over layers:
420
Tear down samplelayers.Layer122 in 0.000 seconds.
421
Tear down samplelayers.Layer12 in 0.000 seconds.
422
Tear down samplelayers.Layer1 in 0.000 seconds.
added
removed
src/zope/testrunner/testrunner-colors.txt
