~canonical-platform-qa/ubuntu-system-tests/trunk

Ubuntu System Tests
###################

Automated Autopilot system test suite for Ubuntu Desktop

This README explains how to install Ubuntu System Tests on the host, how to
prepare a device for testing and execute system tests on it.


Project structure
=================

The ubuntu-system-tests project is split into 4 main packages:

::

ubuntu-system-tests-host
ubuntu-system-tests-helpers
ubuntu-system-tests-suite
ubuntu-system-tests-common

ubuntu-system-tests-host package contains all the scripts necessary to run the
tests from a host machine on a connected device. This package should be
installed on the host.

ubuntu-system-tests-helpers package contains all the autopilot custom proxy
object helpers used by the tests. This package is installed automatically
on the device being tested.

ubuntu-system-tests-suite package contains all the test cases and test data
required for running the tests. This package is installed automatically on
the device being tested.

ubuntu-system-tests-common package contains common scripts shared by the other
packages and is installed automatically.

The ubuntu-system-tests package includes all of the 4 above packages.

Test can be run using installed packages or from local source repository.


Installing ubuntu-system-tests on host
======================================

To install ubuntu-system-tests-host scripts on the host machine:

::

  $ sudo add-apt-repository ppa:canonical-platform-qa/ubuntu-system-tests
  $ sudo apt-get update
  $ sudo apt-get install ubuntu-system-tests-host

This will install the test runner scripts only.

If you wish to list all available tests on the host machine install the
ubuntu-system-tests package:

::

  $ sudo apt-get install ubuntu-system-tests

This will install all test cases onto the host machine so they can be listed.

To see command options run:

::

  $ ubuntu-system-tests --help


Branching the project
=====================

In order to debug and develop new tests for ubuntu-system-tests the code must
be branched locally. Tests will then be run against the branched version
of the code rather than the installed version. To branch the code run:

::

  $ bzr branch lp:ubuntu-system-tests


Command line interface
======================

The command line interface provided by the installed ubuntu-system-tests
package is */usr/bin/ubuntu-system-tests*:

::

  $ ubuntu-system-tests

However if running the tests from a branched version of the project you will
want to make sure you use the local version rather than the installed version.
For this purpose there is a helper script *system-tests* located at the root
of the project:

  $ ./system-tests

This uses a virtual environment to install the local packages and use that
version rather than the native version installed on the host. The
*system-tests* helper script passes all arguments onto ubuntu-system-tests
running in the virtual environment so the interface remains the same.

You can also directly call the command interface in a branched project using:

::

  $ python3 -m ubuntu_system_tests.run


.. _list-tests-label:

Listing available tests
=======================

To list the available tests in ubuntu_system_tests use:

::

  $ autopilot3 list ubuntu_system_tests.tests


This should be run from the host and will require that
ubuntu-system-tests-suite package is installed.


Ubuntu System Tests also provides the ability to lookup tests directly from
Practitest, see :ref:`_practitest-list-tests-label` for more details.


Configuration data
==================

Ubuntu System Tests requires configuration data about the local environment
that will be used for testing. This includes items such as WiFi SSID and
password, device phone number(s), SIM PIN values, result output directory etc.
Not all items are mandatory and if you attempt to run a test without having
the required configuration data specified it will automatically report
result as skipped.

The first time you run ubuntu-system-tests it will prompt you to enter all
configuration items. Configuration data is saved by default at
~/.config/ubuntu-system-tests.conf on the host machine. If required a different
configuration file can be specified using command line options.


Preparing a device for testing
==============================

Before attempting to run tests on a device, ensure it has been flashed
correctly using the following instructions:

::

  https://wiki.ubuntu.com/QATeam/ChannelsToFlash#Flashing_the_Current_Release

 
Once the device is flashed it needs to be prepared for running tests. This
is done using the setup command:

::

  $ ubuntu-system-tests setup

By default this will apt-get update, dist-upgrade and enable dependency
caching by installing apt-cacher-ng. Some testability libraries will also
be installed on the device.

For all setup command options run:

::
 
  $ ubuntu-system-tests setup --help


Running Ubuntu System Tests from host
=====================================

Once the device has been setup, tests can be executed from host using the run
command:

  $ ubuntu-system-tests run


For all run command options run:

::

  $ ubuntu-system-tests run --help


Running Ubuntu System Tests on device
=====================================

It is possible to run tests from ubuntu-system-tests directly on the device
using autopilot, rather than running from the host using the
ubuntu-system-tests run command. In order to enable this, the device will need
to have additional autopilot test dependencies installed. See
:ref:`_install-tests-label` for details on how to do this.

Once the dependencies are installed on the device you will be able to list
the availble tests on the device by running:

::

  $ autopilot3 list ubuntu_system_tests.tests

To run tests directly on the device, run following command where you can also
include a list of specific test ids to run:

::

  $ autopilot3 run ubuntu_system_tests.tests


Running specific test cases
===========================

The run command takes an optional list of tests to execute as parameter. By
default all tests in the suite will be run, but this can be restricted to
specific tests using the provided list. For a full list of available tests,
use the autopilot3 list command as describled in :ref:`_list-tests-label`.

::

  $ ubuntu-system-tests run package1.module1.test1 package2.module2.test2

You can also pass a package or module id to run all the tests defined in it,
like:

::

  $ ubuntu-system-tests run package1 package2.module2


.. _practitest-list-tests-label:

Using Practitest to generate test lists
=======================================

It is possible to query Practitest for a list of tests to use in the
setup and run commands. This is done by using the *--filters* option and
specifying a comma seperated list of Practitest IDs for either the required
filter or test cases.

::

  $ ubuntu-system-tests run --filters filter:1234,test:5678

  $ ubuntu-system-tests setup --filters filter:1234,test:5678

A filter ID must use the "filter:" prefix and a test ID must use the "test:"
prefix. If no prefix is supplied then the ID is assumed to be a filter ID.

Any tests returned by Practitest for the specified filter will be added to the
list of tests specified in the run command.

Filter alias names of "sanity" and "regression" are already defined so that it
is easy to select these required filters.

::

  $ ubuntu-system-tests run --filters sanity

In order to use this feature you must have the required credentials to access
the Practitest service. Please email qa-team@lists.canonical.com to request
these credentials.


Using the list command
======================

Ubuntu System Tests also provides a command to print a list of tests ids using
a specified filter or suite ID. This can be used to show which test cases will
be executed for a particular test suite or filter without having to run any
tests.

::

  $ ubuntu-system-tests list --filters filter:1234,test:5678 suite1.module1

In order to use this feature you must have the required credentials to access
the Practitest service. Please email qa-team@lists.canonical.com to request
these credentials.


Running the project self-tests
==============================

The self tests for this project check that the helper code to run the Ubuntu
System Tests is correct. It does not run the Ubuntu System Tests in the
testbed.

::

  $ python3 setup.py test


Running external Autopilot test suites
======================================

It is possible to run tests from external Autopilot project test suites using
ubuntu-system-tests. In order to enable this the test case IDs must be added
to the list of tests specified in the run command.

It may also be required to install additional dependencies on the device
to allow the external test suite to run. See
:ref:`_defining-dependencies-label` for more information on how to add
dependencies to the test suite.


Command line arguments
======================

ubuntu-system-tests supports the *setup* and *run* commands. Both of these
commands support a range of options that can be configured to suit the
device and environment being used for testing.

In the following sections you will find some information about how
to use the arguments for some common scenarios.

For more detailed command specific help:

::

  $ ubuntu-system-tests [setup | run] --help


Device network setup
--------------------

By default the test suite will configure the device to use the
Wi-Fi network specified by the device configuration data.

Optionally the user can specify the *--network-ready* or *-n* argument which
will assume that a working network connection is already setup on the
device. With this option specified no action will be taken to configure
a new Wi-Fi connection to use for testing. However the existing connection will
be tested as a pre-requisite to make sure it is working successfully.


Run with a different configuration file
---------------------------------------

You can specify the configuration file to use by using the *--config* or *-c*
argument.

::

  $ ubuntu-system-tests [setup | run] --config myconfig.conf

By default, the config file should be in the *~/.config* directory, but you can
use a config file from a different directory by setting the *XDG_CONFIG_HOME*
environment variable. For example, to use /tmp/myconfig.conf:

::

  $ XDG_CONFIG_HOME=/tmp ubuntu-system-tests run --config myconfig.conf

You can use the different configuration files to store values for different
test profiles that you commonly use.


Run with a different section of configuration file
--------------------------------------------------

Within a configuration file you can specify different section headers. This
allows you to have a set of default configuration data which applies to all
devices under test, and a section specific to each device being tested.

When the test suite runs with a section header specified, any data from this
section will take precedence over the default section. If an item is not
defined in the named section, then the default value will be used.

If no section name is specified then only the default section will be used.

The default configuration section is named *[default]* and any other section
names can be defined by the user:

::

    [default]
    wifi_ssid = my-wifi-network-ssid
    wifi_password = my-wifi-password
    output_dir = /home/username/results

    [krillin]
    device_serial = 1234567890
    output_dir = /home/username/results-krillin

    [arale]
    device_serial = 9876543210
    output_dir = /home/username/results-arale

When running tests you can specify the configuration section to use by
adding *--config-section* or *-e* argument.

::

  $ ubuntu-system-tests [setup | run] --config-section arale

Using this option also allows concurrent execution of tests on multiple devices
from the same host machine. In order to do this *device_serial* and
*output_dir* should be specified for each device.


Silent mode operation
---------------------

When an item of configuration data is missing from config file, silent mode
will allow the default value to be selected without prompting the user to
enter a configuration value. This is useful for automated execution where
a user prompt could block test execution.

To enable silent mode operation use *--silent* or *-s* option:

::

$ ubuntu-system-tests [setup | run] --silent


Building a qemu image file
==========================

It is possible to build a qemu image file to be used as a target for test
execution. In order to do so a suitable Ubuntu iso file should be available to
use for installation. This can either be downloaded automatically or a local
iso file can be specified to use instead. If automatic download is enabled
then a new iso file will only be downloaded when a newer file becomes
available.

All auto downloaded iso files and auto generated img files will be stored under
the cache_data_path value specified in config file.

To build a qemu image file use build-qemu command.

Specify an iso file to use with *--iso* option. If this is not specified then
the qemu_iso_url value from config will be used.

The required img output file can also be specified. If this is omitted then
a unique name will be generated for the file under the cache_data_path
specified in config file.

Example to use all default config values:

::

$ ubuntu-system-tests build-qemu

Example to use a local iso and specify the output img file name:

::

$ ubuntu-system-tests build-qemu --iso /home/user/Downloads/zesty-desktop-amd64.iso --img /home/user/dev/disk.img

Example to download specific iso from url and use local img folder path to store resulting output image file:

::

$ ubuntu-system-tests build-qemu --iso http://cdimage.ubuntu.com/xenial/daily-live/current/xenial-desktop-amd64.iso --img /home/user/dev

Example to download latest iso from url and use default img folder path:

::

$ ubuntu-system-tests build-qemu --iso http://cdimage.ubuntu.com/xenial/daily-live/current


Specifying target type
======================

It is possible to run tests using either an abd or qemu target. To specify
a particular target type use target_type config value. Supported values
include *adb*, *qemu* or *ssh*. The default target type is qemu if no other target is
specified.

Example to config for targets:

::

    [krillin]
    target_type = adb

    [qemu]
    target_type = qemu

    [ssh]
    target_type = ssh

Specifying target username and password
=======================================

A username and password is required to connect to the target device over ssh.

For adb devices the default username and password should be phablet and 0000.
For qemu devices the username and password can be any value.

The username and password values are specified in config file. It is useful
to have these values under different sections within the config file to make
using different targets easy. When running tests it is possible to use
*--config-section* or *-e* options to specify the section name to use.

Example config file:

::

    [default]
    device_ip = localhost

    [krillin]
    device_username = phablet
    device_password = 0000

    [qemu]
    device_username = ubuntu
    device_password = ubuntu

Example to run a test using qemu and krillin config:

::

$ ubuntu-system-tests run --config-section qemu ubuntu_system_tests.tests

$ ubuntu-system-tests run --config-section krillin ubuntu_system_tests.tests


Using launch command with qemu target
=====================================

If a test is run with target type qemu, then the qemu will be booted before
running the test and shutdown after the test has completed. This is not helpful
when you need to debug the qemu or copy artifacts to and from the qemu.

For this reason, there is a launch command that will launch a qemu instance
and leave it running until it is shutdown by the user. Once the qemu is running
it is possible to create an ssh connection to debug or copy files to and from
using scp.

::

$ ubuntu-system-tests launch --config-section qemu

The qemu ip and port will be printed along with some other useful commands to
ssh and scp to the qemu.

It is also possible to run tests against a running qemu instance by using the
ssh target type in config and specifying the required address and port using the
--ip and --port options.

$ ubuntu-system-tests run --config-section qemu-ssh --port 2222 --ip localhost ubuntu_system_tests.tests

The config section for qemu-ssh would be defined as:

::

    [qemu-ssh]
    target_type = ssh
    device_username = ubuntu
    device_password = ubuntu


Setup command options
=====================

By default the setup command will apt-get update, dist-upgrade and enable
dependency caching. It is possible to restrict this set of actions by using any
of the following options. When a single option is specified only this action
will be taken.


.. _install-tests-label:

Install tests dependencies in the device
----------------------------------------

It is possible to install the test dependencies in the root dir instead of
install those in /tmp. This is a useful aid for debugging and test development.
Not all the dependencies will be installed permanently, the ones with
"tmpinstall" set "true" in the debian/tests/dependencies.json file will be
skipped due to size limitations. See :ref:`_defining-dependencies-label` for
futher information on defining test dependencies.

It can be enabled by using *--install-tests* or *-i* from command line:

::

  $ ubuntu-system-tests setup --install-tests

This will install all dependencies defined for ubuntu_system_tests. It is also
possible to specify a list of tests to install dependencies for:

::

  $ ubuntu-system-tests setup --install-tests package1 package2.module2


Upgrade device using dist-upgrade
---------------------------------

It is possible to update the device using dist-upgrade before running tests.
This is useful when there have been updates made to the archive that are not
yet in the image. This can be enabled by using *--dist-upgrade* or *-p* from
command line:

::

  $ ubuntu-system-tests setup --dist-upgrade


Add additional PPAs to the device
---------------------------------

PPAs can be added and installed onto the device using the *--ppa* or *-a*
option from command line.

::

  $ ubuntu-system-tests setup --ppa ppa:owner/name

It is possible to specify multiple PPAs using a comma separated list.

::

  $ ubuntu-system-tests setup --ppa ppa:owner/name_1,ppa:owner/name_2


It is also possible use "silo" alias to lookup a specific ppa used by the
CI train service.

::

  $ ubuntu-system-tests setup --ppa silo:1

After a PPA is added to the device it will perform a dist-upgrade and reboot
operation to install any updated components.


Cache downloaded dependencies on the device
-------------------------------------------

When dependencies are copied to the /tmp directory on the device, they must
be re-downloaded for every test run. This is not useful when developing tests
as the tests may need to be executed many times, leading to large amounts of
time spent waiting for dependencies to download. It may also not be an option
to install all dependencies on the device due to limited space on the system
partition.

In order to over-come this it is possible to use a dependency cacher to store
all the downloaded dependencies on the /userdata partition, which is typically
much larger than the system partition. This means that the dependencies are
only downloaded once and can be quickly installed to /tmp without taking up
space on system partition.

In order to enable dependency caching, use *--cache-deps* or *-d* option from
command line:

::

  $ ubuntu-system-tests setup --cache-deps

This will install the dependency cacher and download the required dependencies
onto the device. Subsequent test executions will not require download of
cached dependencies so will be much faster to complete.

This will download all dependencies defined for ubuntu_system_tests. It is also
possible to specify a list of tests to download dependencies for:

::

  $ ubuntu-system-tests setup --cache-deps package1 package2.module2

It is only necessary to specify the *--cache-deps* option once on a device.
Dependency caching will remain enabled on the device after installation.

Note that it may also be required to use *--update-apt* option if it has not
been done so to allow the dependency cacher to be installed.


.. _defining-dependencies-label:

Defining test dependencies
--------------------------

Any test dependencies to be installed on the device must be defined in
*debian/tests/dependencies.json*.

This file lists all test cases and provides a mapping to the required
dependencies that should be used for each defined test. This includes tests
from ubuntu_system_tests as well as tests in external projects.

A list of test ids is defined that maps to a list of required dependency ids.
Each dependency id is then defined in a separate list of dependencies,
which includes other parameters such as minimum version, whether the dependency
should only be installed in /tmp location or any special conditions that
are required.

When a test is run using ubuntu-system-tests, the required dependencies will
be calculated based on the list of tests provided. This ensures that only the
required dependencies are used.

When adding new tests that require additional dependencies, or trying to run
tests from an external project, the dependencies.json file must be updated
to include all required dependencies.


Run command options
===================

The following options apply to the run command.

Enable touch point visualization on screen
------------------------------------------

It is possible to display the touch points on the screen during test execution.
This is a useful aid for debugging. It can be enabled by using
*--touch-visualization* or *-t* options from command line:

::

 $ ubuntu-system-tests run --touch-visualization