2
BuildBot: build/test automation
3
http://buildbot.sourceforge.net/
4
Brian Warner <warner-buildbot @ lothar . com>
9
The BuildBot is a system to automate the compile/test cycle required by most
10
software projects to validate code changes. By automatically rebuilding and
11
testing the tree each time something has changed, build problems are
12
pinpointed quickly, before other developers are inconvenienced by the
13
failure. The guilty developer can be identified and harassed without human
14
intervention. By running the builds on a variety of platforms, developers
15
who do not have the facilities to test their changes everywhere before
16
checkin will at least know shortly afterwards whether they have broken the
17
build or not. Warning counts, lint checks, image size, compile time, and
18
other build parameters can be tracked over time, are more visible, and
19
are therefore easier to improve.
21
The overall goal is to reduce tree breakage and provide a platform to run
22
tests or code-quality checks that are too annoying or pedantic for any human
23
to waste their time with. Developers get immediate (and potentially public)
24
feedback about their changes, encouraging them to be more careful about
25
testing before checkin.
30
* run builds on a variety of slave platforms
31
* arbitrary build process: handles projects using C, Python, whatever
32
* minimal host requirements: python and Twisted
33
* slaves can be behind a firewall if they can still do checkout
34
* status delivery through web page, email, IRC, other protocols
35
* track builds in progress, provide estimated completion time
36
* flexible configuration by subclassing generic build process classes
37
* debug tools to force a new build, submit fake Changes, query slave status
38
* released under the GPL
43
The PyCon paper has a good description of the overall architecture. It is
44
available in HTML form in docs/PyCon-2003/buildbot.html, or on the web page.
46
docs/buildbot.info contains the beginnings of the User's Manual, and the
47
Installation chapter is the best guide to use for setup instructions. The
48
.texinfo source can also be turned into printed documentation.
52
Python: http://www.python.org
54
Buildbot requires python-2.2 or later, and is primarily developed against
55
python-2.3. The buildmaster uses generators, a feature which is not
56
available in python-2.1, and both master and slave require a version of
57
Twisted which only works with python-2.2 or later. Certain features (like
58
the inclusion of build logs in status emails) require python-2.2.2 or
59
later, while the IRC 'force' command requires python-2.3 .
61
Twisted: http://twistedmatrix.com
63
Both the buildmaster and the buildslaves require Twisted-1.3.0 or later.
64
It has been mainly developed against Twisted-2.0.1, but has been tested
65
against Twisted-2.1.0 (the most recent as this time), and might even work
66
on versions as old as Twisted-1.1.0, but as always the most recent version
69
When using the split subpackages of Twisted-2.x.x, you'll need at least
70
"Twisted" (the core package), and you'll also want TwistedMail,
71
TwistedWeb, and TwistedWords (for sending email, serving a web status
72
page, and delivering build status via IRC, respectively).
74
CVSToys: http://purl.net/net/CVSToys
76
If your buildmaster uses FreshCVSSource to receive change notification
77
from a cvstoys daemon, it will require CVSToys be installed (tested with
78
CVSToys-1.0.10). If the it doesn't use that source (i.e. if you only use
79
a mail-parsing change source, or the SVN notification script), you will
84
Please read the User's Manual in docs/buildbot.info (or in HTML form on the
85
buildbot web site) for complete instructions. This file only contains a brief
88
RUNNING THE UNIT TESTS
90
If you would like to run the unit test suite, use a command like this:
92
PYTHONPATH=. trial -v buildbot.test
94
This should run up to 175 tests, depending upon what VC tools you have
95
installed. On my desktop machine it takes about four minutes to complete.
96
Nothing should fail, a few might be skipped. If any of the tests fail, you
97
should stop and investigate the cause before continuing the installation
98
process, as it will probably be easier to track down the bug early.
101
INSTALLING THE LIBRARIES:
103
The first step is to install the python libraries. This package uses the
104
standard 'distutils' module, so installing them is usually a matter of
105
doing something like:
107
python ./setup.py install
109
To test this, shift to a different directory (like /tmp), and run:
113
If it shows you a brief description of the package and its contents, the
114
install went ok. If it says "no Python documentation found for 'buildbot'",
115
then something went wrong.
118
SETTING UP A BUILD SLAVE:
120
If you want to run a build slave, you need to obtain the following pieces of
121
information from the administrator of the buildmaster you intend to connect
124
your buildslave's name
125
the password assigned to your buildslave
126
the hostname and port number of the buildmaster, i.e. example.com:8007
128
You also need to pick a working directory for the buildslave. All commands
129
will be run inside this directory.
131
Now run the 'buildbot' command as follows:
133
buildbot slave WORKDIR MASTERHOST:PORT SLAVENAME PASSWORD
135
This will create a file called "buildbot.tac", which bundles up all the state
136
needed by the build slave application. Twisted has a tool called "twistd"
137
which knows how to load these saved applications and start running them.
138
twistd takes care of logging and daemonization (running the program in the
139
background). /usr/bin/buildbot is a front end which runs twistd for you.
141
Once you've set up the directory with the .tac file, you start it running
144
buildbot start WORKDIR
146
This will start the build slave in the background and finish, so you don't
147
need to put it in the background yourself with "&". The process ID of the
148
background task is written to a file called "twistd.pid", and all output from
149
the program is written to a log file named "twistd.log". Look in twistd.log
150
to make sure the buildslave has started.
152
To shut down the build slave, use:
154
buildbot stop WORKDIR
157
RUNNING BEHIND A NAT BOX:
159
Some network environments will not properly maintain a TCP connection that
160
appears to be idle. NAT boxes which do some form of connection tracking may
161
drop the port mapping if it looks like the TCP session has been idle for too
162
long. The buildslave attempts to turn on TCP "keepalives" (supported by
163
Twisted 1.0.6 and later), and if these cannot be activated, it uses
164
application level keepalives (which send a dummy message to the build master
165
on a periodic basis). The TCP keepalive is typically sent at intervals of
166
about 2 hours, and is configurable through the kernel. The application-level
167
keepalive defaults to running once every 10 minutes.
169
To manually turn on application-level keepalives, or to set them to use some
170
other interval, add "--keepalive NNN" to the 'buildbot slave' command line.
171
NNN is the number of seconds between keepalives. Use as large a value as your
172
NAT box allows to reduce the amount of unnecessary traffic on the wire. 600
173
seconds (10 minutes) is a reasonable value.
176
SETTING UP A BUILD MASTER:
178
Please read the user's manual for instructions. The short form is that you
179
use 'buildbot master MASTERDIR' to create the base directory, then you edit
180
the 'master.cfg' file to configure the buildmaster. Once this is ready, you
181
use 'buildbot START MASTERDIR' to launch it.
183
A sample configuration file will be created for you in WORKDIR/master.cfg .
184
There are more examples in docs/examples/, and plenty of documentation in the
185
user's manual. Everything is controlled by the config file.
190
Please send questions, bugs, patches, etc, to the buildbot-devel mailing
191
list reachable through http://buildbot.sourceforge.net/, so that everyone
added
removed
README
