1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
|
Introduction
============
The purpose of this document is to describe the workflow auto package test
running in the QA Lab. General description of the project, usage and syntax
of the tools is described in the documents README.md, USAGE.md and tools CLI
help.
Auto package testing workflow
=============================
Testbed Setup
-------------
Testbed is prepared with the script *bin/prepare-testbed*. It builds a pristine
VM from a cloud-image using cloud-init.
By default it uses the latest server daily build from cloud-images.ubuntu.com.
This script is triggered by Jenkins when new images are available from
cloud-images.ubuntu.com i.e when the URL
http://cloud-images.ubuntu.com/quantal/current/MD5SUMS changes.
It takes approximately 20 minutes to creates fresh amd64 and i386 testing
environments.
Once the environment is setup, pristine VM are created in
${DISKDIR}/pristine-${RELEASE}-${ARCH}.img
This file is a symbolic link to the latest environment provisioned and the
previous is not destroyed to keep a possibility of manually reverting to the
last known good image.
Build trigger
-------------
The master script to trigger builds is *jenkins/trigger-adt-test*.
This script is triggered when the archive indexes are refreshed, i.e when one of
the following file changes:
* http://archive.ubuntu.com/dist/<RELEASE>/Release
* http://archive.ubuntu.com/dist/<RELEASE>-update/Release
* http://archive.ubuntu.com/dist/<RELEASE>-proposed/Release (Only used if
-proposed testing is enabled with -P)
This jobs execute the following tasks:
* Refresh the archive indices
* Check Sources index for packages with an XS-Testsuite header and a value of
autopkgtest
* If a new package is found, adds it to Jenkins. The job is created dynamically
from the XML template file *jenkins/jenkins_config.xml.tmpl* . This template
file uses Jinja2 syntax.
* If the script runs in update mode (with -U) existing jobs are also updated
from the template file.
* Loads the state of the last run (state files are located in
*$APTROOT/var/cache/adt/<RELEASE>/<PACKAGE>.state*) and triggers a build if:
* New package with tests.
* New version of the package is available.
* New version of a dependency of a binary built from the source package is
available. This is limited to first level dependencies.
* If testing from -proposed is enabled:
* A new version of the package is available in -proposed
* The package is in -release or -update and a new version of a binary
dependency is available from -proposed.
* The package is in -proposed and a new version of a binary dependency
is available from -release or -updates.
* Remotely trigger a jenkins jobs. Jenkins credentials and secret token are
defined in *jenkins/crendentials*.
* Update the state file of the package with the version of the package under
test, the list of dependencies and their version, the cause of the build
list of source packages and versions)
autopkgtest job
---------------
The script called to run autopkgtest is *bin/run-adt-test*. It is called
remotely by *jenkins/trigger-adt-test*.
This script executes the following tasks:
* Prepares a snapshot of the pristine VM to run the test and start it.
* Upload the runner to the testbed. The runner is the script
*bin/testbed/run-adt*.
* Call the runner inside the VM and wait until end of execution.
* Collect the results and copy them locally.
* Call *jenkins/update-adt-result* to update the state file of the package
with the result of the test.
* exit with *adt-run* exit code.
* On exit, destroy the testing environment and lock files.
The runner inside the VM executes the following tasks:
* Prepares the execution environment
* Add proposed and ppa apt sources if required
* update package cache and apply updates
* Downloads the source package in download mode only
* Run adt-run against the .dsc
* For reference this script can also run adt against:
* A package installed from a PPA
* A pacakge installed and a packaging branch in bzr
* A build of a package from a bzr branch. To build the package just pass the
branch name as argument without the source package name.
* Exit with the exit code of adt-run
Results
-------
The state file is updated at the end of the run by the script
*jenkins/update-adt-result*.
This script updates the result for a given architecture. The possible values for
a result are PASS or FAIL.
When all the architectures are tested, it updates the global result of the tests
in the state file. The status can be:
* PASS: If tests succeeded on all architectures
* FAIL: If tests failed on at least 1 architecture
It then creates a summary file
*$APTROOT/var/cache/<RELEASE>/<PACKAGE>_<TIMESTAMP>* which can be consumed by a
machine to automatically proceed with the publication of the package. The format
of the summary file is:
<src pkg> <version> <results> <causes>
<causes> is a list of [<scrpkg:vesion> ...]
The results are then collected by a central server for publication. This central
server periodically:
* Copies the summary files from the test server with rsync in move mode.
* Aggregates the summary files in a single file following timestamp order. This
file can contain more than one record for a given package.
* Archives the summary file in
*/var/local/adt/<RELEASE>archive/<YYYY>/<mm>/<dd>/*.
* Creates a symbolic link to the latest summary file for every package in
*/var/local/adt/latest/*.
|