~lifeless/subunit/xfail

  subunit: A streaming protocol for test results
  Copyright (C) 2005  Robert Collins <robertc@robertcollins.net>

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; either version 2 of the License, or
  (at your option) any later version.

  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


Subunit
-------

Subunit is a streaming protocol for test results. The protocol is human
readable and easily generated and parsed. By design all the components of 
the protocol conceptually fit into the xUnit TestCase->TestResult interaction.

Subunit comes with command line filters to process a subunit stream and
language bindings for python, C, C++ and shell. Bindings are easy to write
for other languages.

A number of useful things can be done easily with subunit:
 * Test aggregation: Tests run separately can be combined and then
   reported/displayed together. For instance, tests from different languages
   can be shown as a seamless whole.
 * Test archiving: A test run may be recorded and replayed later.
 * Test isolation: Tests that may crash or otherwise interact badly with each
   other can be run seperately and then aggregated, rather than interfering
   with each other.
 * Grid testing: subunit can act as the necessary serialisation and
   deserialiation to get test runs on distributed machines to be reported in
   real time.

Subunit supplies the following filters:
 * tap2subunit - convert perl's TestAnythingProtocol to subunit.
 * subunit2pyunit - convert a subunit stream to pyunit test results.
 * subunit-filter - filter out tests from a subunit stream.
 * subunit-ls - list the tests present in a subunit stream.
 * subunit-stats - generate a summary of a subunit stream.
 * subunit-tags - add or remove tags from a stream.

Integration with other tools
----------------------------

Subunit's language bindings act as integration with various test runners like
'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue
(typically a few lines) will allow Subunit to be used in more sophisticated
ways.

Python
======

As a TestResult, Subunit can translate method calls from a test run into a
Subunit stream::

 # Get a TestSuite or TestCase to run
 suite = make_suite()
 # Create a stream (any object with a 'write' method)
 stream = file('tests.log', 'wb')
 # Create a subunit result object which will output to the stream
 result = subunit.TestProtocolClient(stream)
 # Run the test suite reporting to the subunit result object
 suite.run(result)
 # Close the stream.
 stream.close()

As a TestCase, subunit can read from a stream and inform a TestResult
of the activity from the stream::

 # Get a stream (any object with a readline() method), in this case example the
 # stream output by the example before.
 stream = file('tests.log', 'rb')
 # Create a subunit ProtocolTestCase which will read from the stream and emit 
 # activity to a result when run() is called.
 suite = subunit.ProtocolTestCase(stream)
 # Create a result object to show the contents of the stream.
 result = unittest._TextTestResult(sys.stdout)
 # 'run' the tests - process the stream and feed its contents to result.
 suite.run(result)
 stream.close()

Subunit has support for non-blocking usage too, for use with asyncore or
Twisted. See the TestProtocolServer class for more details.

Building on these foundations, Subunit also offers some convenience tools.

The ``IsolatedTestSuite`` class is a decorator that will fork() before running
the decorated item, and gather the results from the child process via subunit.
This is useful for handlings tests that mutate global state, or are testing C
extensions that could crash the VM.

Similarly, ``IsolatedTestCase`` is a base class which can be subclassed to get
tests that will fork() before the test is run.

`ExecTestCase`` is a convenience wrapper for running an external 
program to get a subunit stream and then report that back to an arbitrary
result object::

 class AggregateTests(subunit.ExecTestCase):

     def test_script_one(self):
         """./bin/script_one"""

     def test_script_two(self):
         """./bin/script_two"""
 
 # Normally your normal test loading would take of this automatically,
 # It is only spelt out in detail here for clarity.
 suite = unittest.TestSuite([AggregateTests("test_script_one"),
     AggregateTests("test_script_two")])
 # Create any TestResult class you like.
 result = unittest._TextTestResult(sys.stdout)
 # And run your suite as normal, subunit will exec each external script as
 # needed and report to your result object.
 suite.run(result)

Finally, subunit.run is a convenience wrapper to run a python test suite via
the command line, reporting via subunit::

  $ python -m subunit.run mylib.tests.test_suite
 

C
=

Subunit has C bindings to emit the protocol, and comes with a patch for 'check'
which has been nominally accepted by the 'check' developers. See 'c/README' for
more details.

C++
===

C++ uses the C bindings and includes a patch for cppunit. See 'c++/README' for
details.

shell
=====

Similar to C, the shell bindings consist of simple functions to output protocol
elements, and a patch for adding subunit output to the 'ShUnit' shell test
runner. See 'shell/README' for details.


The protocol
------------

Sample subunit wire contents
----------------------------

The following::
  test: test foo works
  success: test foo works.
  test: tar a file.
  failure: tar a file. [
  ..
   ]..  space is eaten.
  foo.c:34 WARNING foo is not defined.
  ]
  a writeln to stdout

When run through subunit2pyunit::
  .F
  a writeln to stdout

  ========================
  FAILURE: tar a file.
  -------------------
  ..
  ]..  space is eaten.
  foo.c:34 WARNING foo is not defined.


Subunit protocol description
============================
test|testing|test:|testing: test label
success|success:|successful|successful: test label
success|success:|successful|successful: test label [
...
]
failure test label
failure: test label
failure test label [
...
]
failure: test label [
...
]
error: test label
error: test label [
]
skip[:] test label
skip[:] test label [
]
xfail[:] test label
xfail[:] test label [
]
tags: [-]TAG ...
time: YYYY-MM-DD HH:MM:SSZ
unexpected output on stdout -> stdout.
exit w/0 or last test -> error
Tags given outside a test are applied to all following tests
Tags given after a test: line and before the result line for the same test
apply only to that test, and inheric the current global tags.
A '-' before a tag is used to remove tags - e.g. to prevent a global tag
applying to a single test, or to cancel a global tag.
In Python, tags are assigned to the .tags attribute on the RemoteTest objects
created by the TestProtocolServer.

The time element acts as a clock event - it sets the time for all future events.
Currently this is not exposed at the python API layer.

The skip result is used to indicate a test that was found by the runner but not
fully executed due to some policy or dependency issue. This is represented in
python using the addSkip interface that testtools
(https://edge.launchpad.net/testtools) defines. When communicating with a non
skip aware test result, the test is reported as an error.
The xfail result is used to indicate a test that was expected to fail failing
in the expected manner. As this is a normal condition for such tests it is
represented as a successful test in Python.
In future, skip and xfail results will be represented semantically in Python,
but some discussion is underway on the right way to do this.