~fginther/ubuntu-test-cases/click-install-workarounds

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
Touch Testing From the CLI
==========================

The CI touch testing execution framework was written so that it's very easy to
run tests from home in the exact same way tests are run for the CI dashboard
and CI MP testing. The only things you need are:

 * This bzr branch: `lp:ubuntu-test-cases/touch <https://code.launchpad.net/~ubuntu-test-case-dev/ubuntu-test-cases/touch>`_
 * The phablet-tools_ and ubuntu-device-flash_ packages
 * An Ubuntu Touch supported_ device

.. _phablet-tools: http://launchpad.net/phablet-tools
.. _ubuntu-device-flash: http://launchpad.net/goget-ubuntu-touch
.. _supported: http://wiki.ubuntu.com/Touch/Devices

There are two pieces to touch testing, provisioning and test execution.
The provisioning step is required, but once provisioned, you can proceed with
any of the test methods described in this document.

.. note::
  Re-creating test results to match those on http://ci.ubuntu.com/ is not
  always possible. The Ubuntu touch images are created using the Ubuntu
  archive which changes over time as packages are constantly being updated.
  Even when using identical images, test dependencies may change making it
  impossible to recreate an identical test environment.

  For best results, it's recommended to always use the most recent image.

Prerequisites
-------------

Before proceeding, a network configuration file is required. This is a one time
setup and can be used for all future provisioning and testing as long as your
wifi network does not change. This file is the same network configuration file
used by *phablet-network* and can be found by running *phablet-network* with no
options. For example::

  $ phablet-network
  Network file is /etc/NetworkManager/system-connections/my-network-ssid
  Provisioning network on device
  
  Network setup complete
  PING launchpad.net (91.189.89.222) 56(84) bytes of data.

Once the network configuration file is identified, it can be copied into the
default location to be used by the provisioning scripts::

  mkdir ~/.ubuntu-ci
  sudo cp /etc/NetworkManager/system-connections/my-network-ssid ~/.ubuntu-ci/wifi.conf
  sudo chown $USER:$USER ~/.ubuntu-ci/wifi.conf

Provisioning
------------

Provisioning is a required step before performing CI dashboard or MP testing.
It is necessary to configure the device into a known state so that the tests
can run in a predictible manner.

.. warning::
  This provisioning step will completely erase your device. Be sure to
  backup any data before proceeding.

The provisioning script is a simple wrapper to commands from phablet-tools
and some additional device setup to get it ready for testing. Provisioning
is performed with the scripts/provision.sh command. Running::

  ./scripts/provision.sh -w

will install the latest ubuntu-touch/devel-proposed image.  If you
wish to install the latest ubuntu-rtm image instead, use::

  export IMAGE_OPT="--bootstrap --developer-mode --channel=ubuntu-touch/ubuntu-rtm/14.09-proposed"
  ./scripts/provision.sh -w

Provisioning using this script requires that you start off with the
device booted and accessible via ADB. The device will be rebooted
automatically and completely reinstalled - **ALL DATA WILL BE LOST**.

.. note::
  provision.sh requires a path to a network-manager wifi configuration file
  that can be copied to the target device as described above in the
  Prerequisites section. By default this is set to
  ${HOME}/.ubuntu-ci/wifi.conf. This can be overridden with the -n parameter.

Other customizations can be performed during provisioning, for a full list
of options, use::

  ./scripts/provision.sh -h

Executing Tests
---------------

The touch testing tools are intended to provide CI Dashboard and CI MP testing
for specific applications. The supported applications and test suites are
visible in the `CI Dashboard <http://ci.ubuntu.com/>`_.  These tests include
both autopilot and UTAH test definitions. The following sections describe how
to use these tools for testing on a local device.

.. note::
  These tools will *only* work on a device that has been provisioned using the
  methods described above.

.. note::

  Some of the tests generate subunit result files. These subunit result files
  provide richer content and potentially more test artifacts over the xml
  result files. To view the contents of these files, use the trv_ viewer
  application or your favorite subunit viewer.

.. _trv: https://launchpad.net/trv

Executing Autopilot Tests
~~~~~~~~~~~~~~~~~~~~~~~~~

One or more autopilot tests can be run on the target device using the command::

  ./scripts/run-autopilot-tests.sh

This is a small wrapper that uses phablet-tools to drive the tests. The
script can run one or more autopilot tests. By default it will reboot the
device between each test and ensure the device is settled using the
*system-settle* script. Both of those options can be disabled via command
line options. By default the script will create a directory named
*clientlogs* and then a subdirectory for each testsuite with result files.
These sub-directories include a xUnit XML formatted file, *test_results.xml*,
a subunit result stream, *test_results.subunit*, as well as several log files
from the device to help with debugging failures.

.. note::
  run-autopilot-tests.sh will call a script that installs 
  unity8-autopilot if it is not already installed, to allow the device to
  be unlocked automatically.

Examples
^^^^^^^^

An example testing two applications::

  ./scripts/provision.sh -w
  ./scripts/run-autopilot-tests.sh -a dropping_letters_app -a music_app

And an example for running all dashboard autopilot tests::

  ./scripts/provision.sh -w
  ./scripts/run-autopilot-tests.sh

Executing UTAH Tests
~~~~~~~~~~~~~~~~~~~~

Executing UTAH tests locally will require you to install the UTAH client
package from a PPA::

  sudo add-apt-repository ppa:utah/stable
  sudo apt-get update
  sudo apt-get install utah-client

With that package installed UTAH tests can be run with::

  ./scripts/jenkins.sh

This script runs one test at a time and will put its test artifacts under the
*clientlogs* directory similar to the autopilot runner. The UTAH result file
will be named clientlogs/utah.yaml.

An example of running the sdk test suite::

  ./scripts/jenkins.sh -a sdk

Provisioning and Executing Autopilot Tests for an MP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These scripts are used by jenkins for the testing of MPs that generate Debian
packages. The *run-mp.sh* script used below includes provisioning, so for this
one case, it is not necessary to call provision.sh separately.

To re-create the testing performed by jenkins, set the following
environment variables based on the jenkins build parameters::

  export package_archive=<from jenkins build parameter>
  export test_packages=<from jenkins build parameter>
  export test_suite=<from jenkins build parameter>

and set the variable::

  export ANDROID_SERIAL=<adb id from your test device>

Then execute the following script::

  ./scripts/run-mp.sh

Running Autopilot Tests for a Modified Click Application
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First provision the device with the desired image using the instructions
in the "Provisioning" section of this README.

Once the image has been provisioned, install the click app to test.
The dropping-letters application is used in this example::

  adb push com.ubuntu.dropping-letters_0.1.2.2.67_all.click /tmp
  adb shell pkcon --allow-untrusted install-local \
      /tmp/com.ubuntu.dropping-letters_0.1.2.2.67_all.click

Now install the test sources ('--wipe' will remove any previously installed
test sources)::

  phablet-click-test-setup --wipe --click com.ubuntu.dropping-letters

The above phablet-click-test-setup command will install the standard test
dependencies and the click application's test sources as specified in the
manifest. This is usually the application's trunk branch. To override the test
sources with local changes, replace the test sources that were copied to the
device. This example assumes the application code is checked out under the
'dropping-letters' directory with the test sources under 'tests/autopilot'::

  adb shell rm -rf /home/phablet/autopilot/dropping_letters_app
  adb push dropping-letters/tests/autopilot \
      /home/phablet/autopilot

Finally, run the application tests::

  ./scripts/run-autopilot-tests.sh -a dropping_letters_app

The test results are available under::

  clientlogs/dropping_letters_app/test_results.subunit
  clientlogs/dropping_letters_app/test_results.xml

Running Autopilot Tests for a Modified Debian Package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First provision the device with the desired image using the instructions
in the "Provisioning" section of this README.

If the device is provisioned, and you have built the debian package
you wish to test with locally, install it on the device. For instance,
if you are building and installing dialer-app::

  phablet-config writable-image -r 0000 --package-dir /path/to/packages -p dialer-app

Alternatively, if you have built the packages in a ppa, you could use::

  phablet-config writable-image -r 0000 --ppa ppa:ci-train-ppa-service/landing-004 -p dialer-app

.. note::
  If you have updates to the dependencies or tests in debian
  packages, make sure to also install packages for those if required for
  the change you are making. Some tests need a few extra dependencies 
  installed for the tests to function correctly.  To see a list of them, 
  look at jenkins/testconfig.py.

Finally, run the application tests::

  ./scripts/run-autopilot-tests.sh -a dialer_app

The test results are available under::

  clientlogs/dialer_app/test_results.subunit
  clientlogs/dialer_app/test_results.xml