4
Autopilot is a tool for *functional* testing an application. The basic idea is
5
that we simulate user input at a very low level, and verify that the application
6
under test responds the way we expect it to. Autopilot provides two main
7
facilities to make this possible:
10
1. **Input Device Emulation**. Autopilot provides several classes that are able
11
to generate input device events. Keyboard and Mouse events are trivial to
12
generate, and other devices may be added in the future. See for example the
13
:class:`Keyboard <autopilot.emulators.input.Keyboard>` and :class:`Mouse
14
<autopilot.emulators.input.Mouse>` classes.
16
2. **State Introspection**. Autopilot provides several methods of inspecting
17
the applications state, and using the results of that introspection in a test.
19
Autopilot builds on top of several python unity testing tools. In particular,
20
autopilot tests are written in much the same manner as a python unit test would
23
Autopilot tests are based on classic python unit tests. Specifically, autopilot
24
is built on top of the `python-testtools` module. Autopilot tests also
25
frequently make use of the `python-testscenarios` package, so familiarity with
26
this will help you understand existing test suites.
31
Autopilot allows you to write tests for several different targets, including:
33
* The Unity desktop shell. This was the original target for the autopilot
34
tool, and as such contains the most comprehensive test suite.
36
* Qt 4.x applications. Autopilot can introspect Qt 4.x applications with the
37
help of the libautopilot-qt package.
39
* Qt 5.x applications. Autopilot can introspect these applications with the
40
help of the libautopilot-qt package.
42
* Gtk 3.x applications. Autopilot can introspect Gtk applications with the
43
help of the libautopilot-gtk package.
45
The details of how to write an autopilot test are remarkably similar across
46
these different targets. The only thing that changes is the way in which the
47
application under test is started.
49
When testing the unity desktop shell, Unity must be started before the
50
autopilot test is run. There are no special steps that must be taken in order
51
to enable autopilot introspection within Unity - it is enabled by default.
53
For applications, however, the application under test must be started from
54
within the autopilot test.
56
Starting a Qt application
57
-------------------------
59
Test suites for Qt applications must derive from both the
60
:class:`~autopilot.testcase.AutopilotTestCase` class and
61
the :class:`~autopilot.introspection.qt.QtIntrospectionTestMixin` class. The application under test can then be started by calling the
62
:meth:`launch_test_application(application)` method, like so::
64
class MyFirstQtTests(AutopilotTestCase, QtIntrospectionTestMixin):
67
super(MyFirstQtTests, self).setUp()
68
self.application = self.launch_test_application('myappname')
70
Note that the :meth:`launch_test_application(application)` accepts several different options, including:
72
* The name of the executable file, without a path component. In this case, the executable will be searched for in $PATH.
73
* The name of the executable file, with a path component.
74
* A .desktop file, either with, or without a path component.
76
Starting a Qt/Qml application
77
-----------------------------
79
There are two different approaches to running a Qml application under autopilot - either compile the Qml application into a binary that can be run as described above, or run the Qml file within qmlview, like so::
81
class MyFirstQtTests(AutopilotTestCase, QtIntrospectionTestMixin):
84
super(MyFirstQtTests, self).setUp()
85
self.application = self.launch_test_application('qmlviewer', 'myfule.qml')
87
Starting a Gtk application
88
--------------------------
90
Gtk applications are started in a similar manner to Qt applications. The test case class must derive from :class:`~autopilot.introspection.gtk.GtkIntrospectionTestMixin`. Simply call :meth:`launch_test_application(application)` with the application path::
92
class MyFirstGtkTests(AutopilotTestCase, GtkIntrospectionTestMixin):
95
super(MyFirstQtTests, self).setUp()
96
self.application = self.launch_test_application('myappname')
101
Autopilot tests typically have three distinct stages:
103
1. **Test Setup.** Do whatever it takes to get to the point where the thing you're trying to test is ready. This typically involves launching the application under test (not applicable to the Unity shell, as discussed above) and navigating to the component that you want to test.
105
2. **Test Actions.** Send input events to the application under test to mimic a user interaction. This typically involves using the :class:`~autopilot.emulators.input.Keyboard` or :class:`~autopilot.emulators.input.Mouse` classes.
107
3. **Test Assertions.** Do one or more test assertions to verify that the application under test performed as expected.
109
We will examine these three stages in detail.
114
Setup actions generally fall into one of two categories:
116
If the setup action needs to be performed in exactly the same way for every test in the test case, the setup action can be placed inside the setUp method of the test case class. On the other hand, if the setup action is specific to one test, it should be placed at the beginning of the test in question.
121
Make sure that where applicable, any action performed during a test that affects the system is undone at the end of the test. The recommended way of doing this is to call :meth:`~autopilot.testcase.AutopilotTestCase.addCleanup`, passing in a callable and (optionally) arguments that undo the specific action. For example, a test may need to write files to disk during the test setup phase, and clear them up again afterwards. This might be written like so::
123
from os import remove
124
from tempfile import mktemp
127
class CleanupExamplTests(AutopilotTestCase):
129
def test_something(self):
131
open(file_path, 'w').write("Hello World")
132
self.addCleanup(remove, file_path)
134
# test code goes here - 'file_path' will be removed at test end.
136
The addCleanup method can be used anywhere in the test code, including the setUp method. Using addCleanup is recommended over using the tearDown method.
138
You may use addCleanup as many times as you want - they will be run in the reverse order in which they were added. If a cleanup action raises an exception, the exception will be caught, and the test will error, but all remaining cleanup actions will still be run.
143
Test actions will almost always involve simulating user interaction with the application under test. The two principal means of achieving this are generating Keyboard and Mouse events.
148
All classes that derive from :class:`~autopilot.testcase.AutopilotTestCase` have a 'keyboard' attribute that is an instance of :class:`~autopilot.emulators.input.Keyboard`. We recommend that test authors use this instance of the Keyboard class instead of creating new instances. The :class:`~autopilot.emulators.input.Keyboard` class has several capabilities:
150
* **Typing Text**. The most common operation is typing text. This can be achieved by calling the 'type' method, like so::
152
self.keyboard.type("Hello World")
154
Here, each character in the string passed in is pressed and released in sequence. If all goes well, the application under test will recieve the characters 'H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd' - in that order.
156
* **Key Combinations**. Often a test needs to simulate a user pressing a key combination, like 'Ctrl+a'. This is achieved like this::
158
self.keyboard.press_and_release('Ctrl+a')
160
Here, each key is represented with a code separated by a '+' character. Key names follow the standard set in the X11 headers. All the keys mentioned in the string are pressed, and then all the keys are released. Key release events are generated in the reverse order than they are pressed, so the example above generated the following events:
167
* The keyboard class also contains 'press' and 'release' methods. These take the same parameters as the press_and_release method.
169
The Keybindings System
170
~~~~~~~~~~~~~~~~~~~~~~
172
Autopilot includes the :mod:`autopilot.keybindings` module, which includes code to make it easier to send pre-configured keybindings to the application under test. The difficulty with keybindings is that most applications allow the user to configure the keybindings at will. If a user has changed the default keybindings, your autopilot tests will break if you have the default keys hard-coded in your tests. To overcome this, the keybindings system allows you to name a keybinding, and autopilot will read the actual keys to press and release from the application under test.
174
.. note:: At the time of writing, the keybindings system only works when testing Unity. Work is in progress to make this feature work with Qt and Gtk targets.
176
To use the keybindings system, you need to derive from the :class:`~autopilot.keybindings.KeybindingsHelper` class. This class adds the :meth:`~autopilot.keybindings.KeybindingsHelper.keybinding(binding_name, delay)` method, which allows you to send keybindings, like so::
178
from autopilot.testcase import AutopilotTestCase
179
from autopilot.keybindings import KeybindingsHelper
182
class DashRevealTest(AutopilotTestCase, KeybindingsHelper):
184
def test_dash_reveals_with_keybindings(self):
185
self.keybinding("dash/reveal")
186
self.addCleanup(self.dash.ensure_hidden)
188
self.assertThat(dash.visible, Eventually(Equals(True)))
193
All classes that derive from :class:`~autopilot.testcase.AutopilotTestCase` have a 'mouse' attribute that is an instance of :class:`~autopilot.emulators.input.Mouse`. We recommend that test authors use this instance of the Mouse class instead of creating new instances. The :class:`~autopilot.emulators.input.Mouse` class has several capabilities:
195
* **Querying mouse pointer location**. The Mouse class contains two attributes that give the x and y position of the mouse. These can be used to work out where the mouse is::
197
class MouseQueryTests(AutopilotTestCase):
199
def test_mouse_position(self):
200
print "Mouse is at %d, %d." % (self.mouse.x, self.mouse.y)
202
* **Moving the Mouse**. There are two ways to move the mouse pointer, either by calling the 'move' method::
204
self.mouse.move(123, 767)
206
This will move the mouse to position (123, 767) on the screen. Often within a test you need to move the mouse to the position of an object that you already have (a button, perhaps). One (boring) way to achieve this is::
208
self.mouyse.move( btn.x + btn.width / 2, btn.y + btn.height / 2)
210
However, that's a lot of typing. There's a convenience method that works for most objects called 'move_to_object'. Use it like so::
212
self.mouse.move_to_object(btn)
214
This method does exactly what you'd expect it to do.
216
* **Clicking mouse buttons**. The most common action is to click the left mouse button once. This can be achieved simply::
220
Clicking a button other than the left mouse button is easy too::
222
self.mouse.click(button=2)
224
* **Mouse Drag & Drop**. Like the Keyboard class, the Mouse class has methods for pressing and releasing buttons, so a mouse drag might look like this::
227
self.mouse.move(100,100)
233
Autopilot is built on top of the standard python unit test tools - all the test assertion methods that are provided by the :mod:`unittest` and :mod:`testtools` modules are available to an autopilot test. However, autopilot adds a few additional test assertions that may be useful to test authors.
235
The authors of autopilot recommend that test authors make use of the :meth:`~testtools.TestCase.assertThat` method in their tests. The :mod:`autopilot.matchers` module provides the :class:`~autopilot.matchers.Eventually` matcher, which introduces a timeout to the thing being tested. This keeps autopilot tests accurate, since the application under test is in a separate process, and event handling usually happens in an asynchronous fashion. As an example, here's a simple test that ensures that the unity dash is revealed when the 'Super' key is pressed::
237
test_dash_is_revealed(self):
238
dash = ... # Get the dash object from somewhere
239
self.keyboard.press_and_release('Super')
241
self.assertThat(dash.visible, Eventually(Equals(True)))
243
If we didn't use the Eventually matcher, this test might fail if the assertion method was executed before Unity had a chance to reveal the dash. The Eventually matcher is usable on any property that has been transmitted over DBus. The Eventually matcher will not work on calculated values, or values that have been obtained from some other source.
245
To do assertions with a similar timeout in places where Eventually does not work, the :class:`~autopilot.testcase.AutopilotTestCase` class includes the :meth:`~autopilot.testcase.AutopilotTestCase.assertProperty` method. This method takes an object, and a number of keyword arguments. These arguments will be applied to the object and tested, using a similar timeout mechanism to the Eventually matcher. For example, the above example could be re-written to use the assertProperty method::
247
test_dash_is_revealed(self):
248
dash = ... # Get the dash object from somewhere
249
self.keyboard.press_and_release('Super')
251
self.assertProperty(dash, visible=True)
253
One large drawback of the assertProperty method is that it can only test for equality, while other methods of assertion can test anything there is a testtools matcher class for.
1
Writing Your First Test
2
#######################
4
This document contains everything you need to know to write your first autopilot test. It covers writing several simple tests for a sample Qt5/Qml application. However, it's important to note that nothing in this tutorial is specific to Qt5/Qml, and will work equally well with any other kind of application.
9
Your autopilot test suite will grow to several files, possibly spread across several directories. We recommend that you follow this simple directory layout::
12
autopilot/<projectname>/
13
autopilot/<projectname>/emulators/
14
autopilot/<projectname>/tests/
16
The ``autopilot`` folder can be anywhere within your project's source tree. It will likely contain a `setup.py <http://docs.python.org/2/distutils/setupscript.html>`_ file.
18
The ``autopilot/<projectname>/`` folder is the base package for your autopilot tests. This folder, and all child folders, are python packages, and so must contain an `__init__.py file <http://docs.python.org/2/tutorial/modules.html#packages>`_.
20
The ``autopilot/<projectname>/emulators/`` directory is optional, and will only be used if you write custom emulators. This is an advanced topic, and is covered in a later section.
22
.. TODO: Link to the later section once we've written it.
24
Each test file should be named ``test_<component>.py``, where *<component>* is the logical component you are testing in that file. Test files must be written in the ``autopilot/<projectname>/tests/`` folder.
29
Autopilot tests follow a similar pattern to other python test libraries: you must declare a class that derives from :class:`~autopilot.testcase.AutopilotTestCase`. A minimal test case looks like this::
31
from autopilot.testcase import AutopilotTestCase
34
class MyTests(AutopilotTestCase):
36
def test_something(self):
37
"""An example test case that will always pass."""
40
.. otto:: **Make your tests expressive!**
42
It's important to make sure that your tests express your *intent* as clearly as possible. We recommend choosing long, descriptive names for test functions and classes (even breaking :pep:`8`, if you need to), and give your tests a detailed docstring explaining exactly what you are trying to test. For more detailed advice on this point, see :ref:`write-expressive-tests`
47
Before each test is run, the ``setUp`` method is called. Test authors may override this method to run any setup that needs to happen before the test is run. However, care must be taken when using the ``setUp`` method: it tends to hide code from the test case, which can make your tests less readable. It is our recommendation, therefore, that you use this feature sparingly. A more suitable alternative is often to put the setup code in a separate function or method and call it from the test function.
49
Should you wish to put code in a setup method, it looks like this:
51
.. code-block:: python
53
from autopilot.testcase import AutopilotTestCase
56
class MyTests(AutopilotTestCase):
59
super(MyTests, self).setUp()
60
# This code gets run before every test!
62
def test_something(self):
63
"""An example test case that will always pass."""
67
Any action you take in the setup phase must be undone if it alters the system state. See :ref:`cleaning-up` for more details.
69
Starting the Application
70
++++++++++++++++++++++++
72
At the start of your test, you need to tell autopilot to launch your application. To do this, call :meth:`~autopilot.testcase.AutopilotTestCase.launch_test_application`. The minimum required argument to this method is the application name or path. If you pass in the application name, autopilot will look in the current working directory, and then will search the :envvar:`PATH` environment variable. Otherwise, autopilot looks for the executable at the path specified. Positional arguments to this method are passed to the executable being launched.
74
Autopilot will try and guess what type of application you are launching, and therefore what kind of introspection libraries it should load. Sometimes autopilot will need some assistance however. For example, at the time of writing, autopilot cannot automatically detect the introspection type for python / Qt4 applications. In that case, a :class:`RuntimeError` will be raised. To provide autopilot with a hint as to which introspection type to load, you can provide the ``app_type`` keyword argument. For example::
76
class MyTests(AutopilotTestCase):
78
def test_python_qt4_application(self):
79
self.app = self.launch_test_application(
84
See the documentation for :meth:`~autopilot.testcase.AutopilotTestCase.launch_test_application` for more details.
86
The return value from :meth:`~autopilot.testcase.AutopilotTestCase.launch_test_application` is a proxy object representing the root of the introspection tree of the application you just launched.
88
.. otto:: **What is a Proxy Object?**
90
Whenever you launch an application, autopilot gives you a "proxy object". These are instances of the :class:`~autopilot.introspection.DBusIntrospectionObject` class, with all the data from your application mirrored in the proxy object instances. For example, if you have a proxy object for a push button class (say, ``QPushButton``, for example), the proxy object will have attribute to match every attribute in the class within your application. Autopilot automatically keeps the data in these instances up to date, so you can use them in your test assertions.
92
User interfaces are made up of a tree of widgets, and autopilot represents these widgets as a tree of proxy objects. Proxy objects have a number of methods on them for selecting child objects in the introspection tree, so test authors can easily inspect the parts of the UI tree they care about.
97
To demonstrate the material covered so far, this selection will outline a simple application, and a single test for it. Instead of testing a third-party application, we will write the simplest possible application in Python and Qt4. The application, named 'testapp.py', is listed below::
101
from PyQt4 import QtGui
105
app = QtGui.QApplication(argv)
106
win = QtGui.QMainWindow()
108
win.setWindowTitle("Hello World")
111
if __name__ == '__main__':
114
As you can see, this is a trivial application, but it serves our purpose. We will write a single autopilot test that asserts that the title of the main window is equal to the string "Hello World". Our test file is named "test_window.py", and contains the following code::
116
from autopilot.testcase import AutopilotTestCase
117
from os.path import abspath, dirname, join
118
from testtools.matchers import Equals
120
class MainWindowTitleTests(AutopilotTestCase):
122
def launch_application(self):
123
"""Work out the full path to the application and launch it.
125
This is necessary since our test application will not be in $PATH.
127
:returns: The application proxy object.
130
full_path = abspath(join(dirname(__file__), '..', '..', 'testapp.py'))
131
return self.launch_test_application(full_path, app_type='qt')
133
def test_main_window_title_string(self):
134
"""The main window title must be 'Hello World'."""
135
app_root = self.launch_application()
136
main_window = app_root.select_single('QMainWindow')
138
self.assertThat(main_window.windowTitle, Equals("Hello World"))
141
Note that we have made the test method as readable as possible by hiding the complexities of finding the full path to the application we want to test. Of course, if you can guarantee that the application is in :envvar:`PATH`, then this step becomes a lot simpler.
143
The entire directory structure looks like this::
145
./example/__init__.py
146
./example/tests/__init__.py
147
./example/tests/test_window.py
150
The ``__init__.py`` files are empty, and are needed to make these directories importable by python.
155
From the root of this directory structure, we can ask autopilot to list all the tests it can find::
157
$ autopilot list example
158
Loading tests from: /home/thomi/code/canonical/autopilot/example_test
160
example.tests.test_window.MainWindowTitleTests.test_main_window_title_string
165
Note that on the first line, autopilot will tell you where it has loaded the test definitions from. Autopilot will look in the current directory for a python package that matches the package name specified on the command line. If it does not find nay suitable packages, it will look in the standard python module search path instead.
167
To run our test, we use the autopilot 'run' command::
169
$ autopilot run example
170
Loading tests from: /home/thomi/code/canonical/autopilot/example_test
177
You will notice that the test application launches, and then dissapears shortly afterwards. Since this test doesn't manipulate the application in any way, this is a rather boring test to look at. If you ever want more output from the run command, you may specify the '-v' flag::
179
$ autopilot run -v example
180
Loading tests from: /home/thomi/code/canonical/autopilot/example_test
183
13:41:11.614 INFO globals:49 - ************************************************************
184
13:41:11.614 INFO globals:50 - Starting test example.tests.test_window.MainWindowTitleTests.test_main_window_title_string
185
13:41:11.693 INFO __init__:136 - Launching process: ['/home/thomi/code/canonical/autopilot/example_test/testapp.py', '-testability']
186
13:41:11.699 INFO __init__:169 - Looking for autopilot interface for PID 12013 (and children)
187
13:41:11.727 WARNING __init__:185 - Caught exception while searching for autopilot interface: 'DBusException("Could not get PID of name 'org.freedesktop.DBus': no such name",)'
188
13:41:12.773 WARNING __init__:185 - Caught exception while searching for autopilot interface: 'DBusException("Could not get PID of name 'org.freedesktop.DBus': no such name",)'
189
13:41:12.848 WARNING __init__:185 - Caught exception while searching for autopilot interface: 'RuntimeError("Could not find Autopilot interface on DBus backend '<session bus :1.5967 /com/canonical/Autopilot/Introspection>'",)'
190
13:41:12.852 WARNING __init__:185 - Caught exception while searching for autopilot interface: 'RuntimeError("Could not find Autopilot interface on DBus backend '<session bus :1.5968 /com/canonical/Autopilot/Introspection>'",)'
191
13:41:12.863 WARNING dbus:464 - Generating introspection instance for type 'Root' based on generic class.
192
13:41:12.864 DEBUG dbus:338 - Selecting objects of type QMainWindow with attributes: {}
193
13:41:12.871 WARNING dbus:464 - Generating introspection instance for type 'QMainWindow' based on generic class.
194
13:41:12.886 INFO testcase:380 - waiting for process to exit.
195
13:41:13.983 INFO testresult:35 - OK: example.tests.test_window.MainWindowTitleTests.test_main_window_title_string
200
You may also specify '-v' twice for even more output (this is rarely useful for test authors however).
202
Both the 'list' and 'run' commands take a test id as an argument. You may be as generic, or as specific as you like. In the examples above, we will list and run all tests in the 'example' package (i.e.- all tests), but we could specify a more specific run criteria if we only wanted to run some of the tests. For example, to only run the single test we've written, we can execute::
204
$ autopilot run example.tests.test_window.MainWindowTitleTests.test_main_window_title_string
206
A Test with Interaction
207
=======================
209
.. TODO: Add a second test, one that adds some keyboard / mouse interaction.
211
The Eventually Matcher
212
======================
214
.. TODO: Discuss the issues with running tests & application in separate processes, and how the Eventually matcher helps us overcome these problems. Cover the various ways the matcher can be used.
added
removed
docs/tutorial/getting_started.rst
