1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
|
Introduction
============
The tool/infrasctructure described in this document is part of the deliverables
of UDS-P blueprint:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary
This tool determines what tests need to be run via depends, setup a clean
testing environment, drives said tests, and outputs in a format that another
tool can consumes and a human can understand.
The tool is made of the following components:
* prepare-testbed: Create and setup a clean testing VM from a cloud image.
* run-adt-test: Drives run of autopkgtest in the testbed from the host.
* trigger-adt-test: Check periodically the changes in the archive and trigger
the tests.
The project on Launchpad is https://launchpad.net/auto-package-testing
DEP8 documentation:
* http://dep.debian.net/deps/dep8/
* http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.package-tests;hb=HEAD
Testing infrastructure
======================
Pre-requistes
-------------
The host running this project requires the following features:
* kvm enabled - the testing framework using hardware virtualisation features; these need to be enabled (normally done in the BIOS setup screens and be done by anyone via the kvm console)
* http access to cloud-images.ubuntu.com
* http access to archive.ubuntu.com and external resources required by the tests
Configuration
-------------
There are 2 configuration files:
* config: General setup. The settings defined in this file can be overloaded
per user by the file ~/.adtrc. It defines various locations and default
settings and must be a valid shell script.
* credentials: Credentials file for jenkins. This file can be located anywhere
and passed as argument to trigger-adt-test. It contains Jenkins' username,
password, server URL, and secret token to remotely trigger builds.
Testing environment setup
-------------------------
The testing environment is the VM created by the script '''prepare-testbed'''
The script downloads the latest cloud image for a given release and uses
cloud-init to initialize it with the required components to run autopkgtest. A
symlink is created to the most recent testing environment available.
Running the test
----------------
The script responsible for running the test is run-adt-test.
It creates a new VM using the pristine VM as backing store, update it to the
latest version of the packages, download the source package of the package to
be tested and run auto-pkg-test for this package.
The results are copied to the local server and the environment is destroyed.
For debugging purpose, you can pass the argument '-k' and the testing
environment will be preserved after the run so you can login and manually
debug. You'll have to manually clean-up the file when you're done debugging.
The snapshot file can be created in memory to speed up the run with '-s'.
The result files are all the files generated by autopkgtest, a summary file,
and a json file that saves the status of the run. The structure of the json
file is:
{
'status':None, # Status of the run PASS,FAIL,RUNNING
'release':None, # Release being tested
'arch':None, # Architecture
'package':None, # Name of the package
'version':None, # Version of the package
'depends':{}, # List of the dependencies package:version
'causes':{}, # List of the dependencies that triggered the build
}
Trigerring the test
-------------------
The tests are triggered by the script trigger-adt-test.This script periodically
updates its package cache and compares the status of the packages with a dep8
control file to their version and dependencies in the archive. A run is
triggered if a new version of the package has been uploaded or if a new
dependency or a new version of a dependency is available.
Currently, this script triggers a remote jenkins job.
Packages with a header "XS-Testsuite: autopkgtest" are auto-discovered and added to jenkins automatically.
See document USAGE.md for further examples.
Integration with Jenkins
------------------------
Tests are run through jenkins and results are reported to there. There is one
matrix job per source package. This job triggers a test for 2 architectures
i386 and amd64.
The jenkins job is started remotely by the trigger script.
Jenkins jobs for Raring:
https://jenkins.qa.ubuntu.com/view/Raring/view/AutoPkg%20Test/
See document WORKFLOW.md for a full description.
Lab Configuration
-----------------
In the lab the tests run on albali as user 'usit' and scripts are located in
'$HOME/auto-package-testing'
Adding support to a package with an existing testsuite
======================================================
* Prepare a fresh testbed:
$ bin/prepare-testbed amd64
* Start a run for the package you want to add support to. It will fail
because there is no dep8 control file, but will setup the environment for this
package:
$ ./bin/run-adt-test -a amd64 -Sdk aptdaemon
* SSH to your testbed. You can use the command line displayed at the ends of
the invocation of run-adt-test
$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /tmp/adt/disks/adtkey -p 54322 -l ubuntu localhost
* cd to aptdaemon*/debian/ and create a tests directory
* in the test directory create a control file with the following content:
Tests: run-test
Depends: @, python3-nose, python3-setuptools, python3-distutils-extra, python3-mock, gir1.2-packagekitglib-1.0
* Create the file debian/tests/run-test with the following content, and make
it executable:
#!/bin/sh
set -e
# clean up old crash reports
rm -rf /var/crash/*
# succeeding test must not write anything to stderr, as per DEP-8
# work around LP #972324
env -u TMPDIR python3 setup.py test 2>&1
* Run the test:
$ export ADTLOG_PATH=$HOME/adt-log
$ export ADTOPTS="-d --summary $ADTLOG_PATH/summary.log --tmp-dir=$ADTLOG_PATH/ --leave-lang"
$ sudo adt-run -d --summary $ADTLOG_PATH/summary.log --tmp-dir=$ADTLOG_PATH/ --leave-lang --unbuilt-tree . --- adt-virt-null
This command will build the package, install the test dependencies and run the
test script run-test.
Check the everything is right after the run, add any missing test dependencies
to the test control file and run again until the test passes.
* Add a XS-Testsuite header to the source record for autodiscovery. Edit the
file debian/control and add the entry "XS-Testsuite: autopkgtest" to the source
record. It should look like:
Source: aptdaemon
Section: admin
Priority: extra
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
XSBC-Original-Maintainer: Julian Andres Klode <jak@debian.org>
Build-Depends: debhelper (>= 7.3),
debconf-i18n,
dbus,
lintian,
dh-translations,
python-all,
python-setuptools,
python-distutils-extra,
python-nose,
python3-all,
python3-setuptools,
python3-distutils-extra,
python3-nose,
gir1.2-glib-2.0,
gir1.2-packagekitglib-1.0,
python3-apt (>= 0.8.5~ubuntu1),
python3-defer (>= 1.0.6),
python3-dbus,
python3-gi,
python3-mock,
python3-pkg-resources,
python-apt (>= 0.8.5~ubuntu1),
python-defer (>= 1.0.6),
python-dbus,
python-gi,
python-mock,
python-pkg-resources,
iso-codes,
Standards-Version: 3.9.1
Homepage: https://launchpad.net/aptdaemon
Vcs-Bzr: lp:~aptdaemon-developers/aptdaemon/ubuntu-quantal
Vcs-Browser: https://code.launchpad.net/~aptdaemon-developers/aptdaemon/ubuntu-quantal
XS-Original-Vcs-Bzr: nosmart+http://bzr.debian.org/bzr/apt/aptdaemon/debian-sid
XS-Original-Vcs-Browser: http://bzr.debian.org/loggerhead/apt/aptdaemon/debian-sid
X-Python-Version: >= 2.7
X-Python3-Version: >= 3.2
XS-Testsuite: autopkgtest
And you're all set
|