1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
Introduction
============
This document describes how to setup and run automated autopilot tests of
Ubiquity. These testcases work for Ubuntu, Xubuntu and Lubuntu. and should work
for all other flavours of Ubuntu.
Source code: `lp:ubiquity`
Tests use python3 version of autopilot.
When this code is modified, check that it is compliant with PEP8 and there is
no pyflakes error by executing the following command in the top directory of
the checkout:
$ fakeroot debian/rules check
Project Structure
=================
The project is structured as follow:
* *autopilot/* Contains the tests, the runner and a wrapper for autopilot and
ubiquity
* *ubiquity-autopilot-runner/* Runner to setup a VM from an ISO
and execute autopilot tests
* *config/* Contains configuration examples to override default values of
the runner
* *custom-installation/iso-override/* Content of this directory will
override the content on the ISO
* *jenkins/* Scripts, templates and configuration files to deply jenkins
jobs
* *ubiquity_autopilot_tests/* Autopilot tests
Running the tests
=================
Directly from a Live Session
----------------------------
Install the following dependencies :
$ sudo apt-get install python-autopilot libautopilot-gtk python-xlib
To run the tests, open 2 terminals.
Run in Terminal 1 :
$ cd ubiquity/autopilot
$ ./run_ubiquity
To execute the test *test_english_default* Run in Terminal 2:
$ cd ubiquity/autopilot
$ ./autopilot run ubiquity_autopilot_tests.tests.test_english_default
Other tests are available from *ubiquity_autopilot_tests/tests/*. To get a list
of available tests run:
$ autopilot-py3 list ubiquity_autopilot_tests.tests
On a local machine with the runner
----------------------------------
* Install the following dependencies:
$ sudo apt-get install bsdtar qemu-system-x86 bzr xz-utils cpio
* Download a desktop image from *http//cdimage.ubuntu.com/*
* Execute the command:
$ ./ubiquity-autopilot-runner/run-ubiquity-test <ISO>
e.g
$ ./ubiquity-autopilot-runner/run-ubiquity-test ~/iso/ubuntu/trusty-desktop-amd64.iso
* If you want to watch what is running use option `--sdl`
* If your system have enough memory you can run tests in memory (in /dev/shm
actually) with option `-s|--shm`
* And of course `-h|--help` for a list of available options
* At the end of the run, results are collected in `/tmp/ubiquity.tests`
Executing in Jenkins
====================
Execute the tasks below on the slave:
* Create a user called `ubiquity`. This account will pull all the code
required to execute the tests. It can update the code without any specific
jenkins privileges. You can then import ssh keys of the person you want to
be able to manage this account.
* Create a directory `$HOME/bin` for this user.
* Branch ubiquity trunk:
$ bzr branch lp:ubiquity
* Branch ubuntu-qa-tools (to get the script `dl-ubuntu-test-iso`)
$ bzr branch lp:ubuntu-qa-tools
* Get the script `download-latest-test-iso.py` from the project
`lp:ubuntu-server-iso-testing`. This script is a wrapper around
dl-ubuntu-test-iso that adds additional checksums validation, cache and lock
handling to run multiple time in parallel.
$ bzr cat lp:ubuntu-server-iso-testing/download-latest-test-iso.py > ~/bin/download-latest-test-iso.py
$ chmod 755 ~/bin/download-latest-test-iso.py
* Create the following links in ~/bin/
$ ln -s ~/ubuntu-qa-tools/dl-ubuntu-test-iso/dl-ubuntu-test-iso ~/bin
$ ln -s ~/ubiquity/autopilot/ubiquity-autopilot-runner/jenkins/publish2jenkins
* On **jenkins server** create an account in jenkins with job management
privileges.
* On the slave create a credential file in ~/.jenkins.credentials for the
account create on the server with the following content:
<INSTANCE>: # Name of the instance, must match SERVER in publish2jenkins
username: <JENKINS USER NAME>
password: <JENKINS PASSWORD>
url: <JENKINS SERVER URL>
token: <REMOTE JOBS TOKEN>
* Login as **user jenkins** and create the directory `bin`
* Create the same symlinks than symlinks in `/home/ubiquity/bin/`
$ find /home/ubiquity/bin/ ! -type d -exec ln -s {} $HOME/bin/ \;
$ ln -s /home/ubiquity/ubiquity/autopilot/ubiquity-autopilot-runner/ ubiquity-autopilot-runner
* **Logout** from user jenkins
* As **user ubiquity** deploy jenkins jobs:
$ publish2jenkins
* If it runs without error, verify that jenkins jobs are created or updated if
they already existed on `<JENKINS SERVER URL>`
* On **jenkins server** create the following jenkins views:
* **Main** view is a `nested view` that defaults to `All`
* **All** view is a `dashboard view` with the following configuration:
* Regular expression to include jobs: `ubiquity_ap.*`
* Left portlet: `Unstable jobs` and `Jobs statistics`
* Right portlet: `Latest builds`
* Bottom portlet: `Jenkins jobs list`
* Each **flavor view** is a list view with a regular expression to include
jobs set to `ubiquity_ap-<FLAVOR>_.*` . Replace `<FLAVOR>` by the name of
the flavor.
The hierachy of jobs is:
* `ubiquity_ap-<flavor>_devel_daily-download`: Monitor the publication of new
images, download the image and start a new run
* `ubiquity_ap-<flavor>_devel_daily-run`: Run all the tests for a given
flavor.
* `ubiquity_ap-flavor_devel_daily-<test>`: Executes a test for a flavor.
This is a matrix job with an architecture axis. Supported architectures
are amd64 and i386.
There is one job for each test defined in the flavor
configuration file in `ubiquity-autopilot-runner/jenkins/config/`
The tests will start automatically when a new image is available. This is done
a URL content's change trigger that monitor the URL defined in the flavor
configuration file; for example
`http://cdimage.ubuntu.com/daily-live/pending/MD5SUMS`
You can run a specific flavor manually from jenkins by running the job
`ubiquity_ap-ubuntu_devel_daily-download` if there is no ISO already downloaded
on the system, or `ubiquity_ap-ubuntu_devel_daily-run` is an ISO is already
present.
To modifiy jenkins configuration, do not do it directly from the UI but instead
either modify the template in
`ubiquity/autopilot/ubiquity-autopilot-runner/jenkins/templates/` or the configuration
of the flavor in `ubiquity/autopilot/ubiquity-autopilot-runner/jenkins/config/`
To add a new flavor, add a configuration file for the flavor in
`ubiquity/autopilot/ubiquity-autopilot-runner/jenkins/config/`.
To add/remove/modify the list of autopilot tests to run for a flavor, change it
in its configuration file.
Contact Information
-------------------
Authors:
Dan Chapman <dpniel@ubuntu.com>
Jean-Baptiste Lallement <jean-baptiste.lallement@ubuntu.com>
Report Autopilot tests or test runner bugs at: http://bugs.launchpad.net/ubiquity
|