« back to all changes in this revision
Viewing changes to docs/buildbot.info
- browse files at revision 1
- compare with another revision
- download diff
- download tarball
- view history from revision 1
- Committer: Bazaar Package Importer
- Author(s): Matthias Klose
- Date: 2006-04-15 21:20:08 UTC
- Revision ID: james.westby@ubuntu.com-20060415212008-jfj53u29zl30jqi1
Tags: upstream-0.7.2
Import upstream version 0.7.2
- files added:
- bin
- buildbot
- buildbot/changes
- buildbot/clients
- buildbot/process
- buildbot/scripts
- buildbot/slave
- buildbot/status
- buildbot/test
- buildbot/test/mail
- buildbot/test/subdir
- contrib
- contrib/windows
- docs
- docs/PyCon-2003
- docs/examples
added
removed
docs/buildbot.info
1
This is buildbot.info, produced by makeinfo version 4.8 from
2
buildbot.texinfo.
3
4
This is the BuildBot manual.
5
6
Copyright (C) 2005,2006 Brian Warner
7
8
Copying and distribution of this file, with or without
9
modification, are permitted in any medium without royalty provided
10
the copyright notice and this notice are preserved.
11
12
13
File: buildbot.info, Node: Top, Next: Introduction, Prev: (dir), Up: (dir)
14
15
BuildBot
16
********
17
18
This is the BuildBot manual.
19
20
Copyright (C) 2005,2006 Brian Warner
21
22
Copying and distribution of this file, with or without
23
modification, are permitted in any medium without royalty provided
24
the copyright notice and this notice are preserved.
25
26
* Menu:
27
28
* Introduction:: What the BuildBot does.
29
* Installation:: Creating a buildmaster and buildslaves,
30
running them.
31
* Concepts:: What goes on in the buildbot's little mind.
32
* Configuration:: Controlling the buildbot.
33
* Getting Source Code Changes:: Discovering when to run a build.
34
* Build Process:: Controlling how each build is run.
35
* Status Delivery:: Telling the world about the build's results.
36
* Command-line tool::
37
* Resources:: Getting help.
38
* Developer's Appendix::
39
* Index:: Complete index.
40
41
--- The Detailed Node Listing ---
42
43
Introduction
44
45
* History and Philosophy::
46
* System Architecture::
47
* Control Flow::
48
49
Installation
50
51
* Requirements::
52
* Installing the code::
53
* Creating a buildmaster::
54
* Creating a buildslave::
55
* Launching the daemons::
56
* Logfiles::
57
* Shutdown::
58
* Maintenance::
59
* Troubleshooting::
60
61
Creating a buildslave
62
63
* Buildslave Options::
64
65
Troubleshooting
66
67
* Starting the buildslave::
68
* Connecting to the buildmaster::
69
* Forcing Builds::
70
71
Concepts
72
73
* Version Control Systems::
74
* Schedulers::
75
* BuildSet::
76
* BuildRequest::
77
* Builder::
78
* Users::
79
80
Version Control Systems
81
82
* Generalizing VC Systems::
83
* Source Tree Specifications::
84
* How Different VC Systems Specify Sources::
85
* Attributes of Changes::
86
87
Users
88
89
* Doing Things With Users::
90
* Email Addresses::
91
* IRC Nicknames::
92
* Live Status Clients::
93
94
Configuration
95
96
* Config File Format::
97
* Loading the Config File::
98
* Defining the Project::
99
* Listing Change Sources and Schedulers::
100
* Setting the slaveport::
101
* Buildslave Specifiers::
102
* Defining Builders::
103
* Defining Status Targets::
104
* Debug options::
105
106
Listing Change Sources and Schedulers
107
108
* Scheduler Types::
109
* Build Dependencies::
110
111
Getting Source Code Changes
112
113
* Change Sources::
114
115
Change Sources
116
117
* Choosing ChangeSources::
118
* CVSToys - PBService::
119
* CVSToys - mail notification::
120
* Other mail notification ChangeSources::
121
* PBChangeSource::
122
123
Build Process
124
125
* Build Steps::
126
* Interlocks::
127
* Build Factories::
128
129
Build Steps
130
131
* Common Parameters::
132
* Source Checkout::
133
* ShellCommand::
134
* Simple ShellCommand Subclasses::
135
136
Source Checkout
137
138
* CVS::
139
* SVN::
140
* Darcs::
141
* Arch::
142
* Bazaar::
143
* P4Sync::
144
145
Simple ShellCommand Subclasses
146
147
* Configure::
148
* Compile::
149
* Test::
150
151
Build Factories
152
153
* BuildStep Objects::
154
* BuildFactory::
155
* Process-Specific build factories::
156
157
BuildFactory
158
159
* BuildFactory Attributes::
160
* Quick builds::
161
162
Process-Specific build factories
163
164
* GNUAutoconf::
165
* CPAN::
166
* Python distutils::
167
* Python/Twisted/trial projects::
168
169
Status Delivery
170
171
* HTML Waterfall::
172
* IRC Bot::
173
* PBListener::
174
175
Command-line tool
176
177
* Administrator Tools::
178
* Developer Tools::
179
* Other Tools::
180
* .buildbot config directory::
181
182
Developer Tools
183
184
* statuslog::
185
* statusgui::
186
* try::
187
188
Other Tools
189
190
* sendchange::
191
* debugclient::
192
193
194
File: buildbot.info, Node: Introduction, Next: Installation, Prev: Top, Up: Top
195
196
1 Introduction
197
**************
198
199
The BuildBot is a system to automate the compile/test cycle required
200
by most software projects to validate code changes. By automatically
201
rebuilding and testing the tree each time something has changed,
202
build problems are pinpointed quickly, before other developers are
203
inconvenienced by the failure. The guilty developer can be identified
204
and harassed without human intervention. By running the builds on a
205
variety of platforms, developers who do not have the facilities to
206
test their changes everywhere before checkin will at least know
207
shortly afterwards whether they have broken the build or not. Warning
208
counts, lint checks, image size, compile time, and other build
209
parameters can be tracked over time, are more visible, and are
210
therefore easier to improve.
211
212
The overall goal is to reduce tree breakage and provide a platform
213
to run tests or code-quality checks that are too annoying or pedantic
214
for any human to waste their time with. Developers get immediate (and
215
potentially public) feedback about their changes, encouraging them to
216
be more careful about testing before checkin.
217
218
Features:
219
220
* run builds on a variety of slave platforms
221
222
* arbitrary build process: handles projects using C, Python,
223
whatever
224
225
* minimal host requirements: python and Twisted
226
227
* slaves can be behind a firewall if they can still do checkout
228
229
* status delivery through web page, email, IRC, other protocols
230
231
* track builds in progress, provide estimated completion time
232
233
* flexible configuration by subclassing generic build process
234
classes
235
236
* debug tools to force a new build, submit fake Changes, query
237
slave status
238
239
* released under the GPL
240
241
* Menu:
242
243
* History and Philosophy::
244
* System Architecture::
245
* Control Flow::
246
247
248
File: buildbot.info, Node: History and Philosophy, Next: System Architecture, Prev: Introduction, Up: Introduction
249
250
1.1 History and Philosophy
251
==========================
252
253
The Buildbot was inspired by a similar project built for a development
254
team writing a cross-platform embedded system. The various components
255
of the project were supposed to compile and run on several flavors of
256
unix (linux, solaris, BSD), but individual developers had their own
257
preferences and tended to stick to a single platform. From time to
258
time, incompatibilities would sneak in (some unix platforms want to
259
use `string.h', some prefer `strings.h'), and then the tree would
260
compile for some developers but not others. The buildbot was written
261
to automate the human process of walking into the office, updating a
262
tree, compiling (and discovering the breakage), finding the developer
263
at fault, and complaining to them about the problem they had
264
introduced. With multiple platforms it was difficult for developers to
265
do the right thing (compile their potential change on all platforms);
266
the buildbot offered a way to help.
267
268
Another problem was when programmers would change the behavior of a
269
library without warning its users, or change internal aspects that
270
other code was (unfortunately) depending upon. Adding unit tests to
271
the codebase helps here: if an application's unit tests pass despite
272
changes in the libraries it uses, you can have more confidence that
273
the library changes haven't broken anything. Many developers
274
complained that the unit tests were inconvenient or took too long to
275
run: having the buildbot run them reduces the developer's workload to
276
a minimum.
277
278
In general, having more visibility into the project is always good,
279
and automation makes it easier for developers to do the right thing.
280
When everyone can see the status of the project, developers are
281
encouraged to keep the tree in good working order. Unit tests that
282
aren't run on a regular basis tend to suffer from bitrot just like
283
code does: exercising them on a regular basis helps to keep them
284
functioning and useful.
285
286
The current version of the Buildbot is additionally targeted at
287
distributed free-software projects, where resources and platforms are
288
only available when provided by interested volunteers. The buildslaves
289
are designed to require an absolute minimum of configuration, reducing
290
the effort a potential volunteer needs to expend to be able to
291
contribute a new test environment to the project. The goal is for
292
anyone who wishes that a given project would run on their favorite
293
platform should be able to offer that project a buildslave, running on
294
that platform, where they can verify that their portability code
295
works, and keeps working.
296
297
298
File: buildbot.info, Node: System Architecture, Next: Control Flow, Prev: History and Philosophy, Up: Introduction
299
300
1.2 System Architecture
301
=======================
302
303
The Buildbot consists of a single `buildmaster' and one or more
304
`buildslaves', connected in a star topology. The buildmaster makes
305
all decisions about what and when to build. It sends commands to be
306
run on the build slaves, which simply execute the commands and return
307
the results. (certain steps involve more local decision making, where
308
the overhead of sending a lot of commands back and forth would be
309
inappropriate, but in general the buildmaster is responsible for
310
everything).
311
312
The buildmaster is usually fed `Changes' by some sort of version
313
control system *Note Change Sources::, which may cause builds to be
314
run. As the builds are performed, various status messages are
315
produced, which are then sent to any registered Status Targets *Note
316
Status Delivery::.
317
318
TODO: picture of change sources, master, slaves, status targets
319
should look like docs/PyCon-2003/sources/overview.svg
320
321
The buildmaster is configured and maintained by the "buildmaster
322
admin", who is generally the project team member responsible for
323
build process issues. Each buildslave is maintained by a "buildslave
324
admin", who do not need to be quite as involved. Generally slaves are
325
run by anyone who has an interest in seeing the project work well on
326
their platform.
327
328
329
File: buildbot.info, Node: Control Flow, Prev: System Architecture, Up: Introduction
330
331
1.3 Control Flow
332
================
333
334
A day in the life of the buildbot:
335
336
* A developer commits some source code changes to the repository.
337
A hook script or commit trigger of some sort sends information
338
about this change to the buildmaster through one of its
339
configured Change Sources. This notification might arrive via
340
email, or over a network connection (either initiated by the
341
buildmaster as it "subscribes" to changes, or by the commit
342
trigger as it pushes Changes towards the buildmaster). The
343
Change contains information about who made the change, what
344
files were modified, which revision contains the change, and any
345
checkin comments.
346
347
* The buildmaster distributes this change to all of its configured
348
Schedulers. Any "important" changes cause the "tree-stable-timer"
349
to be started, and the Change is added to a list of those that
350
will go into a new Build. When the timer expires, a Build is
351
started on each of a set of configured Builders, all
352
compiling/testing the same source code. Unless configured
353
otherwise, all Builds run in parallel on the various buildslaves.
354
355
* The Build consists of a series of Steps. Each Step causes some
356
number of commands to be invoked on the remote buildslave
357
associated with that Builder. The first step is almost always to
358
perform a checkout of the appropriate revision from the same VC
359
system that produced the Change. The rest generally perform a
360
compile and run unit tests. As each Step runs, the buildslave
361
reports back command output and return status to the buildmaster.
362
363
* As the Build runs, status messages like "Build Started", "Step
364
Started", "Build Finished", etc, are published to a collection of
365
Status Targets. One of these targets is usually the HTML
366
"Waterfall" display, which shows a chronological list of events,
367
and summarizes the results of the most recent build at the top
368
of each column. Developers can periodically check this page to
369
see how their changes have fared. If they see red, they know
370
that they've made a mistake and need to fix it. If they see
371
green, they know that they've done their duty and don't need to
372
worry about their change breaking anything.
373
374
* If a MailNotifier status target is active, the completion of a
375
build will cause email to be sent to any developers whose
376
Changes were incorporated into this Build. The MailNotifier can
377
be configured to only send mail upon failing builds, or for
378
builds which have just transitioned from passing to failing.
379
Other status targets can provide similar real-time notification
380
via different communication channels, like IRC.
381
382
383
384
File: buildbot.info, Node: Installation, Next: Concepts, Prev: Introduction, Up: Top
385
386
2 Installation
387
**************
388
389
* Menu:
390
391
* Requirements::
392
* Installing the code::
393
* Creating a buildmaster::
394
* Creating a buildslave::
395
* Launching the daemons::
396
* Logfiles::
397
* Shutdown::
398
* Maintenance::
399
* Troubleshooting::
400
401
402
File: buildbot.info, Node: Requirements, Next: Installing the code, Prev: Installation, Up: Installation
403
404
2.1 Requirements
405
================
406
407
At a bare minimum, you'll need the following (for both the buildmaster
408
and a buildslave):
409
410
* Python: http://www.python.org
411
412
Buildbot requires python-2.2 or later, and is primarily developed
413
against python-2.3. The buildmaster uses generators, a feature
414
which is not available in python-2.1, and both master and slave
415
require a version of Twisted which only works with python-2.2 or
416
later. Certain features (like the inclusion of build logs in
417
status emails) require python-2.2.2 or later. The IRC "force
418
build" command requires python-2.3 (for the shlex.split
419
function).
420
421
* Twisted: http://twistedmatrix.com
422
423
Both the buildmaster and the buildslaves require Twisted-1.3.0 or
424
later. It has been mainly developed against Twisted-2.0.1, but
425
has been tested against Twisted-2.1.0 (the most recent as of this
426
writing), and might even work on versions as old as
427
Twisted-1.1.0, but as always the most recent version is
428
recommended.
429
430
Twisted-1.3.0 and earlier were released as a single monolithic
431
package. When you run Buildbot against Twisted-2.0.0 or later
432
(which are split into a number of smaller subpackages), you'll
433
need at least "Twisted" (the core package), and you'll also want
434
TwistedMail, TwistedWeb, and TwistedWords (for sending email,
435
serving a web status page, and delivering build status via IRC,
436
respectively).
437
438
Certain other packages may be useful on the system running the
439
buildmaster:
440
441
* CVSToys: http://purl.net/net/CVSToys
442
443
If your buildmaster uses FreshCVSSource to receive change
444
notification from a cvstoys daemon, it will require CVSToys be
445
installed (tested with CVSToys-1.0.10). If the it doesn't use
446
that source (i.e. if you only use a mail-parsing change source,
447
or the SVN notification script), you will not need CVSToys.
448
449
450
And of course, your project's build process will impose additional
451
requirements on the buildslaves. These hosts must have all the tools
452
necessary to compile and test your project's source code.
453
454
455
File: buildbot.info, Node: Installing the code, Next: Creating a buildmaster, Prev: Requirements, Up: Installation
456
457
2.2 Installing the code
458
=======================
459
460
The Buildbot is installed using the standard python `distutils'
461
module. After unpacking the tarball, the process is:
462
463
python setup.py build
464
python setup.py install
465
466
where the install step may need to be done as root. This will put
467
the bulk of the code in somewhere like
468
/usr/lib/python2.3/site-packages/buildbot . It will also install the
469
`buildbot' command-line tool in /usr/bin/buildbot.
470
471
To test this, shift to a different directory (like /tmp), and run:
472
473
buildbot --version
474
475
If it shows you the versions of Buildbot and Twisted, the install
476
went ok. If it says `no such command' or it gets an `ImportError'
477
when it tries to load the libaries, then something went wrong.
478
`pydoc buildbot' is another useful diagnostic tool.
479
480
Windows users will find these files in other places. You will need
481
to make sure that python can find the libraries, and will probably
482
find it convenient to have `buildbot' on your PATH.
483
484
If you wish, you can run the buildbot unit test suite like this:
485
486
PYTHONPATH=. trial -v buildbot.test
487
488
This should run up to 175 tests, depending upon what VC tools you
489
have installed. On my desktop machine it takes about four minutes to
490
complete. Nothing should fail, a few might be skipped. If any of the
491
tests fail, you should stop and investigate the cause before
492
continuing the installation process, as it will probably be easier to
493
track down the bug early.
494
495
If you cannot or do not wish to install the buildbot into a
496
site-wide location like `/usr' or `/usr/local', you can also install
497
it into the account's home directory. Do the install command like
498
this:
499
500
python setup.py install --home=~
501
502
That will populate `~/lib/python' and create `~/bin/buildbot'.
503
Make sure this lib directory is on your `PYTHONPATH'.
504
505
506
File: buildbot.info, Node: Creating a buildmaster, Next: Creating a buildslave, Prev: Installing the code, Up: Installation
507
508
2.3 Creating a buildmaster
509
==========================
510
511
As you learned earlier (*note System Architecture::), the buildmaster
512
runs on a central host (usually one that is publically visible, so
513
everybody can check on the status of the project), and controls all
514
aspects of the buildbot system. Let us call this host
515
`buildbot.example.org'.
516
517
You may wish to create a separate user account for the buildmaster,
518
perhaps named `buildmaster'. This can help keep your personal
519
configuration distinct from that of the buildmaster and is useful if
520
you have to use a mail-based notification system (*note Change
521
Sources::). However, the Buildbot will work just fine with your
522
regular user account.
523
524
You need to choose a directory for the buildmaster, called the
525
`basedir'. This directory will be owned by the buildmaster, which
526
will use configuration files therein, and create status files as it
527
runs. `~/Buildbot' is a likely value. If you run multiple
528
buildmasters in the same account, or if you run both masters and
529
slaves, you may want a more distinctive name like
530
`~/Buildbot/master/gnomovision' or `~/Buildmasters/fooproject'. If
531
you are using a separate user account, this might just be
532
`~buildmaster/masters/fooprojects'.
533
534
Once you've picked a directory, use the `buildbot master' command
535
to create the directory and populate it with startup files:
536
537
buildbot master BASEDIR
538
539
You will need to create a configuration file (*note
540
Configuration::) before starting the buildmaster. Most of the rest of
541
this manual is dedicated to explaining how to do this. A sample
542
configuration file is placed in the working directory, named
543
`master.cfg.sample', which can be copied to `master.cfg' and edited
544
to suit your purposes.
545
546
(Internal details: This command creates a file named
547
`buildbot.tac' that contains all the state necessary to create the
548
buildmaster. Twisted has a tool called `twistd' which can use this
549
.tac file to create and launch a buildmaster instance. twistd takes
550
care of logging and daemonization (running the program in the
551
background). `/usr/bin/buildbot' is a front end which runs twistd for
552
you.)
553
554
In addition to `buildbot.tac', a small `Makefile.sample' is
555
installed. This can be used as the basis for customized daemon
556
startup, *Note Launching the daemons::.
557
558
559
File: buildbot.info, Node: Creating a buildslave, Next: Launching the daemons, Prev: Creating a buildmaster, Up: Installation
560
561
2.4 Creating a buildslave
562
=========================
563
564
Typically, you will be adding a buildslave to an existing buildmaster,
565
to provide additional architecture coverage. The buildbot
566
administrator will give you several pieces of information necessary to
567
connect to the buildmaster. You should also be somewhat familiar with
568
the project being tested, so you can troubleshoot build problems
569
locally.
570
571
The buildbot exists to make sure that the project's stated "how to
572
build it" process actually works. To this end, the buildslave should
573
run in an environment just like that of your regular developers.
574
Typically the project build process is documented somewhere
575
(`README', `INSTALL', etc), in a document that should mention all
576
library dependencies and contain a basic set of build instructions.
577
This document will be useful as you configure the host and account in
578
which the buildslave runs.
579
580
Here's a good checklist for setting up a buildslave:
581
582
1. Set up the account
583
584
It is recommended (although not mandatory) to set up a separate
585
user account for the buildslave. This account is frequently named
586
`buildbot' or `buildslave'. This serves to isolate your personal
587
working environment from that of the slave's, and helps to
588
minimize the security threat posed by letting possibly-unknown
589
contributors run arbitrary code on your system. The account
590
should have a minimum of fancy init scripts.
591
592
2. Install the buildbot code
593
594
Follow the instructions given earlier (*note Installing the
595
code::). If you use a separate buildslave account, and you
596
didn't install the buildbot code to a shared location, then you
597
will need to install it with `--home=~' for each account that
598
needs it.
599
600
3. Set up the host
601
602
Make sure the host can actually reach the buildmaster. Usually
603
the buildmaster is running a status webserver on the same
604
machine, so simply point your web browser at it and see if you
605
can get there. Install whatever additional packages or
606
libraries the project's INSTALL document advises. (or not: if
607
your buildslave is supposed to make sure that building without
608
optional libraries still works, then don't install those
609
libraries).
610
611
Again, these libraries don't necessarily have to be installed to
612
a site-wide shared location, but they must be available to your
613
build process. Accomplishing this is usually very specific to
614
the build process, so installing them to `/usr' or `/usr/local'
615
is usually the best approach.
616
617
4. Test the build process
618
619
Follow the instructions in the INSTALL document, in the
620
buildslave's account. Perform a full CVS (or whatever) checkout,
621
configure, make, run tests, etc. Confirm that the build works
622
without manual fussing. If it doesn't work when you do it by
623
hand, it will be unlikely to work when the buildbot attempts to
624
do it in an automated fashion.
625
626
5. Choose a base directory
627
628
This should be somewhere in the buildslave's account, typically
629
named after the project which is being tested. The buildslave
630
will not touch any file outside of this directory. Something
631
like `~/Buildbot' or `~/Buildslaves/fooproject' is appropriate.
632
633
6. Get the buildmaster host/port, botname, and password
634
635
When the buildbot admin configures the buildmaster to accept and
636
use your buildslave, they will provide you with the following
637
pieces of information:
638
639
* your buildslave's name
640
641
* the password assigned to your buildslave
642
643
* the hostname and port number of the buildmaster, i.e.
644
buildbot.example.org:8007
645
646
7. Create the buildslave
647
648
Now run the 'buildbot' command as follows:
649
650
buildbot slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD
651
652
This will create the base directory and a collection of files
653
inside, including the `buildbot.tac' file that contains all the
654
information you passed to the `buildbot' command.
655
656
8. Fill in the hostinfo files
657
658
When it first connects, the buildslave will send a few files up
659
to the buildmaster which describe the host that it is running
660
on. These files are presented on the web status display so that
661
developers have more information to reproduce any test failures
662
that are witnessed by the buildbot. There are sample files in
663
the `info' subdirectory of the buildbot's base directory. You
664
should edit these to correctly describe you and your host.
665
666
`BASEDIR/info/admin' should contain your name and email address.
667
This is the "buildslave admin address", and will be visible from
668
the build status page (so you may wish to munge it a bit if
669
address-harvesting spambots are a concern).
670
671
`BASEDIR/info/host' should be filled with a brief description of
672
the host: OS, version, memory size, CPU speed, versions of
673
relevant libraries installed, and finally the version of the
674
buildbot code which is running the buildslave.
675
676
If you run many buildslaves, you may want to create a single
677
`~buildslave/info' file and share it among all the buildslaves
678
with symlinks.
679
680
681
* Menu:
682
683
* Buildslave Options::
684
685
686
File: buildbot.info, Node: Buildslave Options, Prev: Creating a buildslave, Up: Creating a buildslave
687
688
2.4.1 Buildslave Options
689
------------------------
690
691
There are a handful of options you might want to use when creating the
692
buildslave with the `buildbot slave <options> DIR <params>' command.
693
You can type `buildbot slave --help' for a summary. To use these,
694
just include them on the `buildbot slave' command line, like this:
695
696
buildbot slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd
697
698
`--usepty'
699
This is a boolean flag that tells the buildslave whether to
700
launch child processes in a PTY (the default) or with regular
701
pipes. The advantage of using a PTY is that "grandchild"
702
processes are more likely to be cleaned up if the build is
703
interrupted or times out (since it enables the use of a "process
704
group" in which all child processes will be placed). The
705
disadvantages: some forms of Unix have problems with PTYs, some
706
of your unit tests may behave differently when run under a PTY
707
(generally those which check to see if they are being run
708
interactively), and PTYs will merge the stdout and stderr
709
streams into a single output stream (which means the red-vs-black
710
coloring in the logfiles will be lost). If you encounter
711
problems, you can add `--usepty=0' to disable the use of PTYs.
712
Note that windows buildslaves never use PTYs.
713
714
`--umask'
715
This is a string (generally an octal representation of an
716
integer) which will cause the buildslave process' "umask" value
717
to be set shortly after initialization. The "twistd"
718
daemonization utility forces the umask to 077 at startup (which
719
means that all files created by the buildslave or its child
720
processes will be unreadable by any user other than the
721
buildslave account). If you want build products to be readable
722
by other accounts, you can add `--umask=022' to tell the
723
buildslave to fix the umask after twistd clobbers it. If you want
724
build products to be _writable_ by other accounts too, use
725
`--umask=000', but this is likely to be a security problem.
726
727
`--keepalive'
728
This is a number that indicates how frequently "keepalive"
729
messages should be sent from the buildslave to the buildmaster,
730
expressed in seconds. The default (600) causes a message to be
731
sent to the buildmaster at least once every 10 minutes. To set
732
this to a lower value, use e.g. `--keepalive=120'.
733
734
If the buildslave is behind a NAT box or stateful firewall, these
735
messages may help to keep the connection alive: some NAT boxes
736
tend to forget about a connection if it has not been used in a
737
while. When this happens, the buildmaster will think that the
738
buildslave has disappeared, and builds will time out. Meanwhile
739
the buildslave will not realize than anything is wrong.
740
741
742
743
File: buildbot.info, Node: Launching the daemons, Next: Logfiles, Prev: Creating a buildslave, Up: Installation
744
745
2.5 Launching the daemons
746
=========================
747
748
Both the buildmaster and the buildslave run as daemon programs. To
749
launch them, pass the working directory to the `buildbot' command:
750
751
buildbot start BASEDIR
752
753
This command will start the daemon and then return, so normally it
754
will not produce any output. To verify that the programs are indeed
755
running, look for a pair of files named `twistd.log' and `twistd.pid'
756
that should be created in the working directory. `twistd.pid'
757
contains the process ID of the newly-spawned daemon.
758
759
When the buildslave connects to the buildmaster, new directories
760
will start appearing in its base directory. The buildmaster tells the
761
slave to create a directory for each Builder which will be using that
762
slave. All build operations are performed within these directories:
763
CVS checkouts, compiles, and tests.
764
765
Once you get everything running, you will want to arrange for the
766
buildbot daemons to be started at boot time. One way is to use
767
`cron', by putting them in a @reboot crontab entry:
768
769
@reboot buildbot BASEDIR
770
771
It is important to remember that the environment provided to cron
772
jobs and init scripts can be quite different that your normal runtime.
773
There may be fewer environment variables specified, and the PATH may
774
be shorter than usual. It is a good idea to test out this method of
775
launching the buildslave by using a cron job with a time in the near
776
future, with the same command, and then check `twistd.log' to make
777
sure the slave actually started correctly. Common problems here are
778
for `/usr/local' or `~/bin' to not be on your `PATH', or for
779
`PYTHONPATH' to not be set correctly. Sometimes `HOME' is messed up
780
too.
781
782
To modify the way the daemons are started (perhaps you want to set
783
some environment variables first, or perform some cleanup each time),
784
you can create a file named `Makefile.buildbot' in the base
785
directory. When the `buildbot' front-end tool is told to `start' the
786
daemon, and it sees this file (and `/usr/bin/make' exists), it will
787
do `make -f Makefile.buildbot start' instead of its usual action
788
(which involves running `twistd'). When the buildmaster or buildslave
789
is installed, a `Makefile.sample' is created which implements the
790
same behavior as the the `buildbot' tool uses, so if you want to
791
customize the process, just copy `Makefile.sample' to
792
`Makefile.buildbot' and edit it as necessary.
793
794
795
File: buildbot.info, Node: Logfiles, Next: Shutdown, Prev: Launching the daemons, Up: Installation
796
797
2.6 Logfiles
798
============
799
800
While a buildbot daemon runs, it emits text to a logfile, named
801
`twistd.log'. A command like `tail -f twistd.log' is useful to watch
802
the command output as it runs.
803
804
The buildmaster will announce any errors with its configuration
805
file in the logfile, so it is a good idea to look at the log at
806
startup time to check for any problems. Most buildmaster activities
807
will cause lines to be added to the log.
808
809
810
File: buildbot.info, Node: Shutdown, Next: Maintenance, Prev: Logfiles, Up: Installation
811
812
2.7 Shutdown
813
============
814
815
To stop a buildmaster or buildslave manually, use:
816
817
buildbot stop BASEDIR
818
819
This simply looks for the `twistd.pid' file and kills whatever
820
process is identified within.
821
822
At system shutdown, all processes are sent a `SIGKILL'. The
823
buildmaster and buildslave will respond to this by shutting down
824
normally.
825
826
The buildmaster will respond to a `SIGHUP' by re-reading its
827
config file. The following shortcut is available:
828
829
buildbot sighup BASEDIR
830
831
832
File: buildbot.info, Node: Maintenance, Next: Troubleshooting, Prev: Shutdown, Up: Installation
833
834
2.8 Maintenance
835
===============
836
837
It is a good idea to check the buildmaster's status page every once in
838
a while, to see if your buildslave is still online. Eventually the
839
buildbot will probably be enhanced to send you email (via the
840
`info/admin' email address) when the slave has been offline for more
841
than a few hours.
842
843
If you find you can no longer provide a buildslave to the project,
844
please let the project admins know, so they can put out a call for a
845
replacement.
846
847
The Buildbot records status and logs output continually, each time
848
a build is performed. The status tends to be small, but the build logs
849
can become quite large. Each build and log are recorded in a separate
850
file, arranged hierarchically under the buildmaster's base directory.
851
To prevent these files from growing without bound, you should
852
periodically delete old build logs. A simple cron job to delete
853
anything older than, say, two weeks should do the job. The only trick
854
is to leave the `buildbot.tac' and other support files alone, for
855
which find's `-mindepth' argument helps skip everything in the top
856
directory. You can use something like the following:
857
858
@weekly cd BASEDIR && find . -mindepth 2 -type f -mtime +14 -exec rm {} \;
859
@weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \;
860
861
862
File: buildbot.info, Node: Troubleshooting, Prev: Maintenance, Up: Installation
863
864
2.9 Troubleshooting
865
===================
866
867
Here are a few hints on diagnosing common problems.
868
869
* Menu:
870
871
* Starting the buildslave::
872
* Connecting to the buildmaster::
873
* Forcing Builds::
874
875
876
File: buildbot.info, Node: Starting the buildslave, Next: Connecting to the buildmaster, Prev: Troubleshooting, Up: Troubleshooting
877
878
2.9.1 Starting the buildslave
879
-----------------------------
880
881
Cron jobs are typically run with a minimal shell (`/bin/sh', not
882
`/bin/bash'), and tilde expansion is not always performed in such
883
commands. You may want to use explicit paths, because the `PATH' is
884
usually quite short and doesn't include anything set by your shell's
885
startup scripts (`.profile', `.bashrc', etc). If you've installed
886
buildbot (or other python libraries) to an unusual location, you may
887
need to add a `PYTHONPATH' specification (note that python will do
888
tilde-expansion on `PYTHONPATH' elements by itself). Sometimes it is
889
safer to fully-specify everything:
890
891
@reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir
892
893
Take the time to get the @reboot job set up. Otherwise, things
894
will work fine for a while, but the first power outage or system
895
reboot you have will stop the buildslave with nothing but the cries
896
of sorrowful developers to remind you that it has gone away.
897
898
899
File: buildbot.info, Node: Connecting to the buildmaster, Next: Forcing Builds, Prev: Starting the buildslave, Up: Troubleshooting
900
901
2.9.2 Connecting to the buildmaster
902
-----------------------------------
903
904
If the buildslave cannot connect to the buildmaster, the reason should
905
be described in the `twistd.log' logfile. Some common problems are an
906
incorrect master hostname or port number, or a mistyped bot name or
907
password. If the buildslave loses the connection to the master, it is
908
supposed to attempt to reconnect with an exponentially-increasing
909
backoff. Each attempt (and the time of the next attempt) will be
910
logged. If you get impatient, just manually stop and re-start the
911
buildslave.
912
913
When the buildmaster is restarted, all slaves will be disconnected,
914
and will attempt to reconnect as usual. The reconnect time will depend
915
upon how long the buildmaster is offline (i.e. how far up the
916
exponential backoff curve the slaves have travelled). Again,
917
`buildbot stop BASEDIR; buildbot start BASEDIR' will speed up the
918
process.
919
920
921
File: buildbot.info, Node: Forcing Builds, Prev: Connecting to the buildmaster, Up: Troubleshooting
922
923
2.9.3 Forcing Builds
924
--------------------
925
926
From the buildmaster's main status web page, you can force a build to
927
be run on your build slave. Figure out which column is for a builder
928
that runs on your slave, click on that builder's name, and the page
929
that comes up will have a "Force Build" button. Fill in the form, hit
930
the button, and a moment later you should see your slave's
931
`twistd.log' filling with commands being run. Using `pstree' or `top'
932
should also reveal the cvs/make/gcc/etc processes being run by the
933
buildslave. Note that the same web page should also show the `admin'
934
and `host' information files that you configured earlier.
935
936
937
File: buildbot.info, Node: Concepts, Next: Configuration, Prev: Installation, Up: Top
938
939
3 Concepts
940
**********
941
942
This chapter defines some of the basic concepts that the Buildbot
943
uses. You'll need to understand how the Buildbot sees the world to
944
configure it properly.
945
946
* Menu:
947
948
* Version Control Systems::
949
* Schedulers::
950
* BuildSet::
951
* BuildRequest::
952
* Builder::
953
* Users::
954
955
956
File: buildbot.info, Node: Version Control Systems, Next: Schedulers, Prev: Concepts, Up: Concepts
957
958
3.1 Version Control Systems
959
===========================
960
961
These source trees come from a Version Control System of some kind.
962
CVS and Subversion are two popular ones, but the Buildbot supports
963
others. All VC systems have some notion of an upstream `repository'
964
which acts as a server(1), from which clients can obtain source trees
965
according to various parameters. The VC repository provides source
966
trees of various projects, for different branches, and from various
967
points in time. The first thing we have to do is to specify which
968
source tree we want to get.
969
970
* Menu:
971
972
* Generalizing VC Systems::
973
* Source Tree Specifications::
974
* How Different VC Systems Specify Sources::
975
* Attributes of Changes::
976
977
---------- Footnotes ----------
978
979
(1) except Darcs, but since the Buildbot never modifies its local
980
source tree we can ignore the fact that Darcs uses a less centralized
981
model
982
983
984
File: buildbot.info, Node: Generalizing VC Systems, Next: Source Tree Specifications, Prev: Version Control Systems, Up: Version Control Systems
985
986
3.1.1 Generalizing VC Systems
987
-----------------------------
988
989
For the purposes of the Buildbot, we will try to generalize all VC
990
systems as having repositories that each provide sources for a variety
991
of projects. Each project is defined as a directory tree with source
992
files. The individual files may each have revisions, but we ignore
993
that and treat the project as a whole as having a set of revisions.
994
Each time someone commits a change to the project, a new revision
995
becomes available. These revisions can be described by a tuple with
996
two items: the first is a branch tag, and the second is some kind of
997
timestamp or revision stamp. Complex projects may have multiple branch
998
tags, but there is always a default branch. The timestamp may be an
999
actual timestamp (such as the -D option to CVS), or it may be a
1000
monotonically-increasing transaction number (such as the change number
1001
used by SVN and P4, or the revision number used by Arch, or a labeled
1002
tag used in CVS)(1). The SHA1 revision ID used by Monotone is also a
1003
kind of revision stamp, in that it specifies a unique copy of the
1004
source tree, as does a Darcs "context" file.
1005
1006
When we aren't intending to make any changes to the sources we
1007
check out (at least not any that need to be committed back upstream),
1008
there are two basic ways to use a VC system:
1009
1010
* Retrieve a specific set of source revisions: some tag or key is
1011
used to index this set, which is fixed and cannot be changed by
1012
subsequent developers committing new changes to the tree.
1013
Releases are built from tagged revisions like this, so that they
1014
can be rebuilt again later (probably with controlled
1015
modifications).
1016
1017
* Retrieve the latest sources along a specific branch: some tag is
1018
used to indicate which branch is to be used, but within that
1019
constraint we want to get the latest revisions.
1020
1021
Build personnel or CM staff typically use the first approach: the
1022
build that results is (ideally) completely specified by the two
1023
parameters given to the VC system: repository and revision tag. This
1024
gives QA and end-users something concrete to point at when reporting
1025
bugs. Release engineers are also reportedly fond of shipping code that
1026
can be traced back to a concise revision tag of some sort.
1027
1028
Developers are more likely to use the second approach: each morning
1029
the developer does an update to pull in the changes committed by the
1030
team over the last day. These builds are not easy to fully specify: it
1031
depends upon exactly when you did a checkout, and upon what local
1032
changes the developer has in their tree. Developers do not normally
1033
tag each build they produce, because there is usually significant
1034
overhead involved in creating these tags. Recreating the trees used by
1035
one of these builds can be a challenge. Some VC systems may provide
1036
implicit tags (like a revision number), while others may allow the use
1037
of timestamps to mean "the state of the tree at time X" as opposed to
1038
a tree-state that has been explicitly marked.
1039
1040
The Buildbot is designed to help developers, so it usually works in
1041
terms of _the latest_ sources as opposed to specific tagged
1042
revisions. However, it would really prefer to build from reproducible
1043
source trees, so implicit revisions are used whenever possible.
1044
1045
---------- Footnotes ----------
1046
1047
(1) many VC systems provide more complexity than this: in
1048
particular the local views that P4 and ClearCase can assemble out of
1049
various source directories are more complex than we're prepared to
1050
take advantage of here
1051
1052
1053
File: buildbot.info, Node: Source Tree Specifications, Next: How Different VC Systems Specify Sources, Prev: Generalizing VC Systems, Up: Version Control Systems
1054
1055
3.1.2 Source Tree Specifications
1056
--------------------------------
1057
1058
So for the Buildbot's purposes we treat each VC system as a server
1059
which can take a list of specifications as input and produce a source
1060
tree as output. Some of these specifications are static: they are
1061
attributes of the builder and do not change over time. Others are more
1062
variable: each build will have a different value. The repository is
1063
changed over time by a sequence of Changes, each of which represents a
1064
single developer making changes to some set of files. These Changes
1065
are cumulative(1).
1066
1067
For normal builds, the Buildbot wants to get well-defined source
1068
trees that contain specific Changes, and exclude other Changes that
1069
may have occurred after the desired ones. We assume that the Changes
1070
arrive at the buildbot (through one of the mechanisms described in
1071
*note Change Sources::) in the same order in which they are committed
1072
to the repository. The Buildbot waits for the tree to become "stable"
1073
before initiating a build, for two reasons. The first is that
1074
developers frequently make multiple related commits in quick
1075
succession, even when the VC system provides ways to make atomic
1076
transactions involving multiple files at the same time. Running a
1077
build in the middle of these sets of changes would use an inconsistent
1078
set of source files, and is likely to fail (and is certain to be less
1079
useful than a build which uses the full set of changes). The
1080
tree-stable-timer is intended to avoid these useless builds that
1081
include some of the developer's changes but not all. The second reason
1082
is that some VC systems (i.e. CVS) do not provide repository-wide
1083
transaction numbers, so that timestamps are the only way to refer to
1084
a specific repository state. These timestamps may be somewhat
1085
ambiguous, due to processing and notification delays. By waiting until
1086
the tree has been stable for, say, 10 minutes, we can choose a
1087
timestamp from the middle of that period to use for our source
1088
checkout, and then be reasonably sure that any clock-skew errors will
1089
not cause the build to be performed on an inconsistent set of source
1090
files.
1091
1092
The Schedulers always use the tree-stable-timer, with a timeout
1093
that is configured to reflect a reasonable tradeoff between build
1094
latency and change frequency. When the VC system provides coherent
1095
repository-wide revision markers (such as Subversion's revision
1096
numbers, or in fact anything other than CVS's timestamps), the
1097
resulting Build is simply performed against a source tree defined by
1098
that revision marker. When the VC system does not provide this, a
1099
timestamp from the middle of the tree-stable period is used to
1100
generate the source tree(2).
1101
1102
---------- Footnotes ----------
1103
1104
(1) Monotone's _multiple heads_ feature violates this assumption
1105
of cumulative Changes, but in most situations the changes don't occur
1106
frequently enough for this to be a significant problem
1107
1108
(2) this `checkoutDelay' defaults to half the tree-stable timer,
1109
but it can be overridden with an argument to the Source Step
1110
1111
1112
File: buildbot.info, Node: How Different VC Systems Specify Sources, Next: Attributes of Changes, Prev: Source Tree Specifications, Up: Version Control Systems
1113
1114
3.1.3 How Different VC Systems Specify Sources
1115
----------------------------------------------
1116
1117
For CVS, the static specifications are `repository' and `module'. In
1118
addition to those, each build uses a timestamp (or omits the
1119
timestamp to mean `the latest') and `branch tag' (which defaults to
1120
HEAD). These parameters collectively specify a set of sources from
1121
which a build may be performed.
1122
1123
Subversion (http://subversion.tigris.org) combines the repository,
1124
module, and branch into a single `Subversion URL' parameter. Within
1125
that scope, source checkouts can be specified by a numeric `revision
1126
number' (a repository-wide monotonically-increasing marker, such that
1127
each transaction that changes the repository is indexed by a
1128
different revision number), or a revision timestamp. When branches
1129
are used, the repository and module form a static `baseURL', while
1130
each build has a `revision number' and a `branch' (which defaults to a
1131
statically-specified `defaultBranch'). The `baseURL' and `branch' are
1132
simply concatenated together to derive the `svnurl' to use for the
1133
checkout.
1134
1135
Arch (http://wiki.gnuarch.org/) and Bazaar
1136
(http://bazaar.canonical.com/) specify a repository by URL, as well
1137
as a `version' which is kind of like a branch name. Arch uses the
1138
word `archive' to represent the repository. Arch lets you push
1139
changes from one archive to another, removing the strict
1140
centralization required by CVS and SVN. It retains the distinction
1141
between repository and working directory that most other VC systems
1142
use. For complex multi-module directory structures, Arch has a
1143
built-in `build config' layer with which the checkout process has two
1144
steps. First, an initial bootstrap checkout is performed to retrieve
1145
a set of build-config files. Second, one of these files is used to
1146
figure out which archives/modules should be used to populate
1147
subdirectories of the initial checkout.
1148
1149
Builders which use Arch and Bazaar therefore have a static archive
1150
`url', and a default "branch" (which is a string that specifies a
1151
complete category-branch-version triple). Each build can have its own
1152
branch (the category-branch-version string) to override the default,
1153
as well as a revision number (which is turned into a -patch-NN suffix
1154
when performing the checkout).
1155
1156
Darcs (http://abridgegame.org/darcs/) doesn't really have the
1157
notion of a single master repository. Nor does it really have
1158
branches. In Darcs, each working directory is also a repository, and
1159
there are operations to push and pull patches from one of these
1160
`repositories' to another. For the Buildbot's purposes, all you need
1161
to do is specify the URL of a repository that you want to build from.
1162
The build slave will then pull the latest patches from that
1163
repository and build them. Multiple branches are implemented by using
1164
multiple repositories (possibly living on the same server).
1165
1166
Builders which use Darcs therefore have a static `repourl' which
1167
specifies the location of the repository. If branches are being used,
1168
the source Step is instead configured with a `baseURL' and a
1169
`defaultBranch', and the two strings are simply concatenated together
1170
to obtain the repository's URL. Each build then has a specific branch
1171
which replaces `defaultBranch', or just uses the default one. Instead
1172
of a revision number, each build can have a "context", which is a
1173
string that records all the patches that are present in a given tree
1174
(this is the output of `darcs changes --context', and is considerably
1175
less concise than, e.g. Subversion's revision number, but the
1176
patch-reordering flexibility of Darcs makes it impossible to provide
1177
a shorter useful specification).
1178
1179
1180
File: buildbot.info, Node: Attributes of Changes, Prev: How Different VC Systems Specify Sources, Up: Version Control Systems
1181
1182
3.1.4 Attributes of Changes
1183
---------------------------
1184
1185
Who
1186
===
1187
1188
Each Change has a `who' attribute, which specifies which developer is
1189
responsible for the change. This is a string which comes from a
1190
namespace controlled by the VC repository. Frequently this means it
1191
is a username on the host which runs the repository, but not all VC
1192
systems require this (Arch, for example, uses a fully-qualified `Arch
1193
ID', which looks like an email address, as does Darcs). Each
1194
StatusNotifier will map the `who' attribute into something
1195
appropriate for their particular means of communication: an email
1196
address, an IRC handle, etc.
1197
1198
Files
1199
=====
1200
1201
It also has a list of `files', which are just the tree-relative
1202
filenames of any files that were added, deleted, or modified for this
1203
Change. These filenames are used by the `isFileImportant' function
1204
(in the Scheduler) to decide whether it is worth triggering a new
1205
build or not, e.g. the function could use `filename.endswith(".c")'
1206
to only run a build if a C file were checked in. Certain BuildSteps
1207
can also use the list of changed files to run a more targeted series
1208
of tests, e.g. the `step_twisted.Trial' step can run just the unit
1209
tests that provide coverage for the modified .py files instead of
1210
running the full test suite.
1211
1212
Comments
1213
========
1214
1215
The Change also has a `comments' attribute, which is a string
1216
containing any checkin comments.
1217
1218
Revision
1219
========
1220
1221
Each Change can have a `revision' attribute, which describes how to
1222
get a tree with a specific state: a tree which includes this Change
1223
(and all that came before it) but none that come after it. If this
1224
information is unavailable, the `.revision' attribute will be `None'.
1225
These revisions are provided by the ChangeSource, and consumed by the
1226
`computeSourceRevision' method in the appropriate `step.Source' class.
1227
1228
`CVS'
1229
`revision' is an int, seconds since the epoch
1230
1231
`SVN'
1232
`revision' is an int, a transation number (r%d)
1233
1234
`Darcs'
1235
`revision' is a large string, the output of `darcs changes
1236
--context'
1237
1238
`Arch/Bazaar'
1239
`revision' is the full revision ID (ending in -patch-%d)
1240
1241
`P4'
1242
`revision' is an int, the transaction number
1243
1244
Branches
1245
========
1246
1247
The Change might also have a `branch' attribute. This indicates that
1248
all of the Change's files are in the same named branch. The
1249
Schedulers get to decide whether the branch should be built or not.
1250
1251
For VC systems like CVS, Arch, and Monotone, the `branch' name is
1252
unrelated to the filename. (that is, the branch name and the filename
1253
inhabit unrelated namespaces). For SVN, branches are expressed as
1254
subdirectories of the repository, so the file's "svnurl" is a
1255
combination of some base URL, the branch name, and the filename within
1256
the branch. (In a sense, the branch name and the filename inhabit the
1257
same namespace). Darcs branches are subdirectories of a base URL just
1258
like SVN.
1259
1260
`CVS'
1261
branch='warner-newfeature', files=['src/foo.c']
1262
1263
`SVN'
1264
branch='branches/trunk', files=['src/foo.c']
1265
1266
`Darcs'
1267
branch='branches/trunk', files=['src/foo.c']
1268
1269
`Arch/Bazaar'
1270
branch='buildbot-usebranches-0', files=['buildbot/master.py']
1271
1272
Links
1273
=====
1274
1275
Finally, the Change might have a `links' list, which is intended to
1276
provide a list of URLs to a _viewcvs_-style web page that provides
1277
more detail for this Change, perhaps including the full file diffs.
1278
1279
1280
File: buildbot.info, Node: Schedulers, Next: BuildSet, Prev: Version Control Systems, Up: Concepts
1281
1282
3.2 Schedulers
1283
==============
1284
1285
Each Buildmaster has a set of `Scheduler' objects, each of which gets
1286
a copy of every incoming Change. The Schedulers are responsible for
1287
deciding when Builds should be run. Some Buildbot installations might
1288
have a single Scheduler, while others may have several, each for a
1289
different purpose.
1290
1291
For example, a "quick" scheduler might exist to give immediate
1292
feedback to developers, hoping to catch obvious problems in the code
1293
that can be detected quickly. These typically do not run the full test
1294
suite, nor do they run on a wide variety of platforms. They also
1295
usually do a VC update rather than performing a brand-new checkout
1296
each time. You could have a "quick" scheduler which used a 30 second
1297
timeout, and feeds a single "quick" Builder that uses a VC
1298
`mode='update'' setting.
1299
1300
A separate "full" scheduler would run more comprehensive tests a
1301
little while later, to catch more subtle problems. This scheduler
1302
would have a longer tree-stable-timer, maybe 30 minutes, and would
1303
feed multiple Builders (with a `mode=' of `'copy'', `'clobber'', or
1304
`'export'').
1305
1306
The `tree-stable-timer' and `isFileImportant' decisions are made
1307
by the Scheduler. Dependencies are also implemented here. Periodic
1308
builds (those which are run every N seconds rather than after new
1309
Changes arrive) are triggered by a special `Periodic' Scheduler
1310
subclass. The default Scheduler class can also be told to watch for
1311
specific branches, ignoring Changes on other branches. This may be
1312
useful if you have a trunk and a few release branches which should be
1313
tracked, but when you don't want to have the Buildbot pay attention
1314
to several dozen private user branches.
1315
1316
Some Schedulers may trigger builds for other reasons, other than
1317
recent Changes. For example, a Scheduler subclass could connect to a
1318
remote buildmaster and watch for builds of a library to succeed before
1319
triggering a local build that uses that library.
1320
1321
Each Scheduler creates and submits `BuildSet' objects to the
1322
`BuildMaster', which is then responsible for making sure the
1323
individual `BuildRequests' are delivered to the target `Builders'.
1324
1325
`Scheduler' instances are activated by placing them in the
1326
`c['schedulers']' list in the buildmaster config file. Each Scheduler
1327
has a unique name.
1328
1329
1330
File: buildbot.info, Node: BuildSet, Next: BuildRequest, Prev: Schedulers, Up: Concepts
1331
1332
3.3 BuildSet
1333
============
1334
1335
A `BuildSet' is the name given to a set of Builds that all
1336
compile/test the same version of the tree on multiple Builders. In
1337
general, all these component Builds will perform the same sequence of
1338
Steps, using the same source code, but on different platforms or
1339
against a different set of libraries.
1340
1341
The `BuildSet' is tracked as a single unit, which fails if any of
1342
the component Builds have failed, and therefore can succeed only if
1343
_all_ of the component Builds have succeeded. There are two kinds of
1344
status notification messages that can be emitted for a BuildSet: the
1345
`firstFailure' type (which fires as soon as we know the BuildSet will
1346
fail), and the `Finished' type (which fires once the BuildSet has
1347
completely finished, regardless of whether the overall set passed or
1348
failed).
1349
1350
A `BuildSet' is created with a _source stamp_ tuple of (branch,
1351
revision, changes, patch), some of which may be None, and a list of
1352
Builders on which it is to be run. They are then given to the
1353
BuildMaster, which is responsible for creating a separate
1354
`BuildRequest' for each Builder.
1355
1356
There are a couple of different likely values for the
1357
`SourceStamp':
1358
1359
`(revision=None, changes=[CHANGES], patch=None)'
1360
This is a `SourceStamp' used when a series of Changes have
1361
triggered a build. The VC step will attempt to check out a tree
1362
that contains CHANGES (and any changes that occurred before
1363
CHANGES, but not any that occurred after them).
1364
1365
`(revision=None, changes=None, patch=None)'
1366
This builds the most recent code on the default branch. This is
1367
the sort of `SourceStamp' that would be used on a Build that was
1368
triggered by a user request, or a Periodic scheduler. It is also
1369
possible to configure the VC Source Step to always check out the
1370
latest sources rather than paying attention to the Changes in the
1371
SourceStamp, which will result in same behavior as this.
1372
1373
`(branch=BRANCH, revision=None, changes=None, patch=None)'
1374
This builds the most recent code on the given BRANCH. Again,
1375
this is generally triggered by a user request or Periodic build.
1376
1377
`(revision=REV, changes=None, patch=(LEVEL, DIFF))'
1378
This checks out the tree at the given revision REV, then applies
1379
a patch (using `diff -pLEVEL <DIFF'). The *Note try:: feature
1380
uses this kind of `SourceStamp'. If `patch' is None, the patching
1381
step is bypassed.
1382
1383
1384
The buildmaster is responsible for turning the `BuildSet' into a
1385
set of `BuildRequest' objects and queueing them on the appropriate
1386
Builders.
1387
1388
1389
File: buildbot.info, Node: BuildRequest, Next: Builder, Prev: BuildSet, Up: Concepts
1390
1391
3.4 BuildRequest
1392
================
1393
1394
A `BuildRequest' is a request to build a specific set of sources on a
1395
single specific Builder. Each Builder runs the `BuildRequest' as soon
1396
as it can (i.e. when an associated buildslave becomes free).
1397
1398
The `BuildRequest' contains the `SourceStamp' specification. The
1399
actual process of running the build (the series of Steps that will be
1400
executed) is implemented by the `Build' object. In this future this
1401
might be changed, to have the `Build' define _what_ gets built, and a
1402
separate `BuildProcess' (provided by the Builder) to define _how_ it
1403
gets built.
1404
1405
The `BuildRequest' may be mergeable with other compatible
1406
`BuildRequest's. Builds that are triggered by incoming Changes will
1407
generally be mergeable. Builds that are triggered by user requests
1408
are generally not, unless they are multiple requests to build the
1409
_latest sources_ of the same branch.
1410
1411
1412
File: buildbot.info, Node: Builder, Next: Users, Prev: BuildRequest, Up: Concepts
1413
1414
3.5 Builder
1415
===========
1416
1417
The `Builder' is a long-lived object which controls all Builds of a
1418
given type. Each one is created when the config file is first parsed,
1419
and lives forever (or rather until it is removed from the config
1420
file). It mediates the connections to the buildslaves that do all the
1421
work, and is responsible for creating the `Build' objects that decide
1422
_how_ a build is performed (i.e., which steps are executed in what
1423
order).
1424
1425
Each `Builder' gets a unique name, and the path name of a
1426
directory where it gets to do all its work (there is a
1427
buildmaster-side directory for keeping status information, as well as
1428
a buildslave-side directory where the actual checkout/compile/test
1429
commands are executed). It also gets a `BuildFactory', which is
1430
responsible for creating new `Build' instances: because the `Build'
1431
instance is what actually performs each build, choosing the
1432
`BuildFactory' is the way to specify what happens each time a build
1433
is done.
1434
1435
Each `Builder' is associated with one of more `BuildSlaves'. A
1436
`Builder' which is used to perform OS-X builds (as opposed to Linux
1437
or Solaris builds) should naturally be associated with an OS-X-based
1438
buildslave.
1439
1440
1441
File: buildbot.info, Node: Users, Prev: Builder, Up: Concepts
1442
1443
3.6 Users
1444
=========
1445
1446
Buildbot has a somewhat limited awareness of _users_. It assumes the
1447
world consists of a set of developers, each of whom can be described
1448
by a couple of simple attributes. These developers make changes to
1449
the source code, causing builds which may succeed or fail.
1450
1451
Each developer is primarily known through the source control
1452
system. Each Change object that arrives is tagged with a `who' field
1453
that typically gives the account name (on the repository machine) of
1454
the user responsible for that change. This string is the primary key
1455
by which the User is known, and is displayed on the HTML status pages
1456
and in each Build's "blamelist".
1457
1458
To do more with the User than just refer to them, this username
1459
needs to be mapped into an address of some sort. The responsibility
1460
for this mapping is left up to the status module which needs the
1461
address. The core code knows nothing about email addresses or IRC
1462
nicknames, just user names.
1463
1464
* Menu:
1465
1466
* Doing Things With Users::
1467
* Email Addresses::
1468
* IRC Nicknames::
1469
* Live Status Clients::
1470
1471
1472
File: buildbot.info, Node: Doing Things With Users, Next: Email Addresses, Prev: Users, Up: Users
1473
1474
3.6.1 Doing Things With Users
1475
-----------------------------
1476
1477
Each Change has a single User who is responsible for that Change. Most
1478
Builds have a set of Changes: the Build represents the first time
1479
these Changes have been built and tested by the Buildbot. The build
1480
has a "blamelist" that consists of a simple union of the Users
1481
responsible for all the Build's Changes.
1482
1483
The Build provides (through the IBuildStatus interface) a list of
1484
Users who are "involved" in the build. For now this is equal to the
1485
blamelist, but in the future it will be expanded to include a "build
1486
sheriff" (a person who is "on duty" at that time and responsible for
1487
watching over all builds that occur during their shift), as well as
1488
per-module owners who simply want to keep watch over their domain
1489
(chosen by subdirectory or a regexp matched against the filenames
1490
pulled out of the Changes). The Involved Users are those who probably
1491
have an interest in the results of any given build.
1492
1493
In the future, Buildbot will acquire the concept of "Problems",
1494
which last longer than builds and have beginnings and ends. For
1495
example, a test case which passed in one build and then failed in the
1496
next is a Problem. The Problem lasts until the test case starts
1497
passing again, at which point the Problem is said to be "resolved".
1498
1499
If there appears to be a code change that went into the tree at the
1500
same time as the test started failing, that Change is marked as being
1501
resposible for the Problem, and the user who made the change is added
1502
to the Problem's "Guilty" list. In addition to this user, there may
1503
be others who share responsibility for the Problem (module owners,
1504
sponsoring developers). In addition to the Responsible Users, there
1505
may be a set of Interested Users, who take an interest in the fate of
1506
the Problem.
1507
1508
Problems therefore have sets of Users who may want to be kept
1509
aware of the condition of the problem as it changes over time. If
1510
configured, the Buildbot can pester everyone on the Responsible list
1511
with increasing harshness until the problem is resolved, with the
1512
most harshness reserved for the Guilty parties themselves. The
1513
Interested Users may merely be told when the problem starts and
1514
stops, as they are not actually responsible for fixing anything.
1515
1516
1517
File: buildbot.info, Node: Email Addresses, Next: IRC Nicknames, Prev: Doing Things With Users, Up: Users
1518
1519
3.6.2 Email Addresses
1520
---------------------
1521
1522
The `buildbot.status.mail.MailNotifier' class provides a status
1523
target which can send email about the results of each build. It
1524
accepts a static list of email addresses to which each message should
1525
be delivered, but it can also be configured to send mail to the
1526
Build's Interested Users. To do this, it needs a way to convert User
1527
names into email addresses.
1528
1529
For many VC systems, the User Name is actually an account name on
1530
the system which hosts the repository. As such, turning the name into
1531
an email address is a simple matter of appending
1532
"@repositoryhost.com". Some projects use other kinds of mappings (for
1533
example the preferred email address may be at "project.org" despite
1534
the repository host being named "cvs.project.org"), and some VC
1535
systems have full separation between the concept of a user and that
1536
of an account on the repository host (like Perforce). Some systems
1537
(like Arch) put a full contact email address in every change.
1538
1539
To convert these names to addresses, the MailNotifier uses an
1540
EmailLookup object. This provides a .getAddress method which accepts
1541
a name and (eventually) returns an address. The default `MailNotifier'
1542
module provides an EmailLookup which simply appends a static string,
1543
configurable when the notifier is created. To create more complex
1544
behaviors (perhaps using an LDAP lookup, or using "finger" on a
1545
central host to determine a preferred address for the developer),
1546
provide a different object as the `lookup' argument.
1547
1548
In the future, when the Problem mechanism has been set up, the
1549
Buildbot will need to send mail to arbitrary Users. It will do this
1550
by locating a MailNotifier-like object among all the buildmaster's
1551
status targets, and asking it to send messages to various Users. This
1552
means the User-to-address mapping only has to be set up once, in your
1553
MailNotifier, and every email message the buildbot emits will take
1554
advantage of it.
1555
1556
1557
File: buildbot.info, Node: IRC Nicknames, Next: Live Status Clients, Prev: Email Addresses, Up: Users
1558
1559
3.6.3 IRC Nicknames
1560
-------------------
1561
1562
Like MailNotifier, the `buildbot.status.words.IRC' class provides a
1563
status target which can announce the results of each build. It also
1564
provides an interactive interface by responding to online queries
1565
posted in the channel or sent as private messages.
1566
1567
In the future, the buildbot can be configured map User names to IRC
1568
nicknames, to watch for the recent presence of these nicknames, and to
1569
deliver build status messages to the interested parties. Like
1570
`MailNotifier' does for email addresses, the `IRC' object will have
1571
an `IRCLookup' which is responsible for nicknames. The mapping can be
1572
set up statically, or it can be updated by online users themselves
1573
(by claiming a username with some kind of "buildbot: i am user
1574
warner" commands).
1575
1576
Once the mapping is established, the rest of the buildbot can ask
1577
the `IRC' object to send messages to various users. It can report on
1578
the likelihood that the user saw the given message (based upon how
1579
long the user has been inactive on the channel), which might prompt
1580
the Problem Hassler logic to send them an email message instead.
1581
1582
1583
File: buildbot.info, Node: Live Status Clients, Prev: IRC Nicknames, Up: Users
1584
1585
3.6.4 Live Status Clients
1586
-------------------------
1587
1588
The Buildbot also offers a PB-based status client interface which can
1589
display real-time build status in a GUI panel on the developer's
1590
desktop. This interface is normally anonymous, but it could be
1591
configured to let the buildmaster know _which_ developer is using the
1592
status client. The status client could then be used as a
1593
message-delivery service, providing an alternative way to deliver
1594
low-latency high-interruption messages to the developer (like "hey,
1595
you broke the build").
1596
1597
1598
File: buildbot.info, Node: Configuration, Next: Getting Source Code Changes, Prev: Concepts, Up: Top
1599
1600
4 Configuration
1601
***************
1602
1603
The buildbot's behavior is defined by the "config file", which
1604
normally lives in the `master.cfg' file in the buildmaster's base
1605
directory (but this can be changed with an option to the `buildbot
1606
master' command). This file completely specifies which Builders are
1607
to be run, which slaves they should use, how Changes should be
1608
tracked, and where the status information is to be sent. The
1609
buildmaster's `buildbot.tac' file names the base directory;
1610
everything else comes from the config file.
1611
1612
A sample config file was installed for you when you created the
1613
buildmaster, but you will need to edit it before your buildbot will do
1614
anything useful.
1615
1616
This chapter gives an overview of the format of this file and the
1617
various sections in it. You will need to read the later chapters to
1618
understand how to fill in each section properly.
1619
1620
* Menu:
1621
1622
* Config File Format::
1623
* Loading the Config File::
1624
* Defining the Project::
1625
* Listing Change Sources and Schedulers::
1626
* Setting the slaveport::
1627
* Buildslave Specifiers::
1628
* Defining Builders::
1629
* Defining Status Targets::
1630
* Debug options::
1631
1632
1633
File: buildbot.info, Node: Config File Format, Next: Loading the Config File, Prev: Configuration, Up: Configuration
1634
1635
4.1 Config File Format
1636
======================
1637
1638
The config file is, fundamentally, just a piece of Python code which
1639
defines a dictionary named `BuildmasterConfig', with a number of keys
1640
that are treated specially. You don't need to know Python to do basic
1641
configuration, though, you can just copy the syntax of the sample
1642
file. If you _are_ comfortable writing Python code, however, you can
1643
use all the power of a full programming language to achieve more
1644
complicated configurations.
1645
1646
The `BuildmasterConfig' name is the only one which matters: all
1647
other names defined during the execution of the file are discarded.
1648
When parsing the config file, the Buildmaster generally compares the
1649
old configuration with the new one and performs the minimum set of
1650
actions necessary to bring the buildbot up to date: Builders which are
1651
not changed are left untouched, and Builders which are modified get to
1652
keep their old event history.
1653
1654
Basic Python syntax: comments start with a hash character ("#"),
1655
tuples are defined with `(parenthesis, pairs)', arrays are defined
1656
with `[square, brackets]', tuples and arrays are mostly
1657
interchangeable. Dictionaries (data structures which map "keys" to
1658
"values") are defined with curly braces: `{'key1': 'value1', 'key2':
1659
'value2'} '. Function calls (and object instantiation) can use named
1660
parameters, like `w = html.Waterfall(http_port=8010)'.
1661
1662
The config file starts with a series of `import' statements, which
1663
make various kinds of Steps and Status targets available for later
1664
use. The main `BuildmasterConfig' dictionary is created, then it is
1665
populated with a variety of keys. These keys are broken roughly into
1666
the following sections, each of which is documented in the rest of
1667
this chapter:
1668
1669
* Project Definitions
1670
1671
* Change Sources / Schedulers
1672
1673
* Slaveport
1674
1675
* Buildslave Configuration
1676
1677
* Builders / Interlocks
1678
1679
* Status Targets
1680
1681
* Debug options
1682
1683
The config file can use a few names which are placed into its
1684
namespace:
1685
1686
`basedir'
1687
the base directory for the buildmaster. This string has not been
1688
expanded, so it may start with a tilde. It needs to be expanded
1689
before use. The config file is located in
1690
`os.path.expanduser(os.path.join(basedir, 'master.cfg'))'
1691
1692
1693
1694
File: buildbot.info, Node: Loading the Config File, Next: Defining the Project, Prev: Config File Format, Up: Configuration
1695
1696
4.2 Loading the Config File
1697
===========================
1698
1699
The config file is only read at specific points in time. It is first
1700
read when the buildmaster is launched. Once it is running, there are
1701
various ways to ask it to reload the config file. If you are on the
1702
system hosting the buildmaster, you can send a `SIGHUP' signal to it:
1703
the `buildbot' tool has a shortcut for this:
1704
1705
buildbot sighup BASEDIR
1706
1707
The debug tool (`buildbot debugclient --master HOST:PORT') has a
1708
"Reload .cfg" button which will also trigger a reload. In the future,
1709
there will be other ways to accomplish this step (probably a
1710
password-protected button on the web page, as well as a privileged IRC
1711
command).
1712
1713
1714
File: buildbot.info, Node: Defining the Project, Next: Listing Change Sources and Schedulers, Prev: Loading the Config File, Up: Configuration
1715
1716
4.3 Defining the Project
1717
========================
1718
1719
There are a couple of basic settings that you use to tell the buildbot
1720
what project it is working on. This information is used by status
1721
reporters to let users find out more about the codebase being
1722
exercised by this particular Buildbot installation.
1723
1724
c['projectName'] = "Buildbot"
1725
c['projectURL'] = "http://buildbot.sourceforge.net/"
1726
c['buildbotURL'] = "http://localhost:8010/"
1727
1728
`projectName' is a short string will be used to describe the
1729
project that this buildbot is working on. For example, it is used as
1730
the title of the waterfall HTML page.
1731
1732
`projectURL' is a string that gives a URL for the project as a
1733
whole. HTML status displays will show `projectName' as a link to
1734
`projectURL', to provide a link from buildbot HTML pages to your
1735
project's home page.
1736
1737
The `buildbotURL' string should point to the location where the
1738
buildbot's internal web server (usually the `html.Waterfall' page) is
1739
visible. This typically uses the port number set when you create the
1740
`Waterfall' object: the buildbot needs your help to figure out a
1741
suitable externally-visible host name.
1742
1743
When status notices are sent to users (either by email or over
1744
IRC), `buildbotURL' will be used to create a URL to the specific build
1745
or problem that they are being notified about. It will also be made
1746
available to queriers (over IRC) who want to find out where to get
1747
more information about this buildbot.
1748
1749
1750
File: buildbot.info, Node: Listing Change Sources and Schedulers, Next: Setting the slaveport, Prev: Defining the Project, Up: Configuration
1751
1752
4.4 Listing Change Sources and Schedulers
1753
=========================================
1754
1755
The `c['sources']' key is a list of ChangeSource instances(1). This
1756
defines how the buildmaster learns about source code changes. More
1757
information about what goes here is available in *Note Getting Source
1758
Code Changes::.
1759
1760
import buildbot.changes.pb
1761
c['sources'] = [buildbot.changes.pb.PBChangeSource()]
1762
1763
`c['schedulers']' is a list of Scheduler instances, each of which
1764
causes builds to be started on a particular set of Builders. The two
1765
basic Scheduler classes you are likely to start with are `Scheduler'
1766
and `Periodic', but you can write a customized subclass to implement
1767
more complicated build scheduling.
1768
1769
The docstring for `buildbot.scheduler.Scheduler' is the best place
1770
to see all the options that can be used. Type `pydoc
1771
buildbot.scheduler.Scheduler' to see it, or look in
1772
`buildbot/scheduler.py' directly.
1773
1774
The basic Scheduler takes four arguments:
1775
1776
`name'
1777
Each Scheduler must have a unique name. This is only used in
1778
status displays.
1779
1780
`branch'
1781
This Scheduler will pay attention to a single branch, ignoring
1782
Changes that occur on other branches. Setting `branch' equal to
1783
the special value of `None' means it should only pay attention
1784
to the default branch. Note that `None' is a keyword, not a
1785
string, so you want to use `None' and not `"None"'.
1786
1787
`treeStableTimer'
1788
The Scheduler will wait for this many seconds before starting the
1789
build. If new changes are made during this interval, the timer
1790
will be restarted, so really the build will be started after a
1791
change and then after this many seconds of inactivity.
1792
1793
`builderNames'
1794
When the tree-stable-timer finally expires, builds will be
1795
started on these Builders. Each Builder gets a unique name:
1796
these strings must match.
1797
1798
1799
from buildbot import scheduler
1800
quick = scheduler.Scheduler("quick", None, 60,
1801
["quick-linux", "quick-netbsd"])
1802
full = scheduler.Scheduler("full", None, 5*60,
1803
["full-linux", "full-netbsd", "full-OSX"])
1804
nightly = scheduler.Periodic("nightly", ["full-solaris"], 24*60*60)
1805
c['schedulers'] = [quick, full, nightly]
1806
1807
In this example, the two "quick" builds are triggered 60 seconds
1808
after the tree has been changed. The "full" builds do not run quite
1809
so quickly (they wait 5 minutes), so hopefully if the quick builds
1810
fail due to a missing file or really simple typo, the developer can
1811
discover and fix the problem before the full builds are started. Both
1812
Schedulers only pay attention to the default branch: any changes on
1813
other branches are ignored by these Schedulers. Each Scheduler
1814
triggers a different set of Builders, referenced by name.
1815
1816
The third Scheduler in this example just runs the full solaris
1817
build once per day. (note that this Scheduler only lets you control
1818
the time between builds, not the absolute time-of-day of each Build,
1819
so this could easily wind up a "daily" or "every afternoon" scheduler
1820
depending upon when it was first activated).
1821
1822
* Menu:
1823
1824
* Scheduler Types::
1825
* Build Dependencies::
1826
1827
---------- Footnotes ----------
1828
1829
(1) To be precise, it is a list of objects which all implement the
1830
`buildbot.interfaces.IChangeSource' Interface
1831
1832
1833
File: buildbot.info, Node: Scheduler Types, Next: Build Dependencies, Prev: Listing Change Sources and Schedulers, Up: Listing Change Sources and Schedulers
1834
1835
4.4.1 Scheduler Types
1836
---------------------
1837
1838
Here is a brief catalog of the available Scheduler types. All these
1839
Schedulers are classes in `buildbot.scheduler', and the docstrings
1840
there are the best source of documentation on the arguments taken by
1841
each one.
1842
1843
`Scheduler'
1844
This is the default Scheduler class. It follows exactly one
1845
branch, and starts a configurable tree-stable-timer after each
1846
change on that branch. When the timer expires, it starts a build
1847
on some set of Builders. The Scheduler accepts a
1848
`fileIsImportant' function which can be used to ignore some
1849
Changes if they do not affect any "important" files.
1850
1851
`AnyBranchScheduler'
1852
This scheduler uses a tree-stable-timer like the default one, but
1853
follows multiple branches at once. Each branch gets a separate
1854
timer.
1855
1856
`Dependent'
1857
This scheduler watches an "upstream" Builder. When that Builder
1858
successfully builds a particular set of Changes, it triggers
1859
builds of the same code on a configured set of "downstream"
1860
builders. The next section (*note Build Dependencies::)
1861
describes this scheduler in more detail.
1862
1863
`Periodic'
1864
This simple scheduler just triggers a build every N seconds.
1865
1866
`Nightly'
1867
This is highly configurable periodic build scheduler, which
1868
triggers a build at particular times of day, week, month, or
1869
year. The configuration syntax is very similar to the well-known
1870
`crontab' format, in which you provide values for minute, hour,
1871
day, and month (some of which can be wildcards), and a build is
1872
triggered whenever the current time matches the given
1873
constraints. This can run a build every night, every morning,
1874
every weekend, alternate Thursdays, on your boss's birthday, etc.
1875
1876
`Try_Jobdir / Try_Userpass'
1877
This scheduler allows developers to use the `buildbot try'
1878
command to trigger builds of code they have not yet committed.
1879
See *Note try:: for complete details.
1880
1881
1882
1883
File: buildbot.info, Node: Build Dependencies, Prev: Scheduler Types, Up: Listing Change Sources and Schedulers
1884
1885
4.4.2 Build Dependencies
1886
------------------------
1887
1888
It is common to wind up with one kind of build which should only be
1889
performed if the same source code was successfully handled by some
1890
other kind of build first. An example might be a packaging step: you
1891
might only want to produce .deb or RPM packages from a tree that was
1892
known to compile successfully and pass all unit tests. You could put
1893
the packaging step in the same Build as the compile and testing steps,
1894
but there might be other reasons to not do this (in particular you
1895
might have several Builders worth of compiles/tests, but only wish to
1896
do the packaging once). Another example is if you want to skip the
1897
"full" builds after a failing "quick" build of the same source code.
1898
Or, if one Build creates a product (like a compiled library) that is
1899
used by some other Builder, you'd want to make sure the consuming
1900
Build is run _after_ the producing one.
1901
1902
You can use `Dependencies' to express this relationship to the
1903
Buildbot. There is a special kind of Scheduler named
1904
`scheduler.Dependent' that will watch an "upstream" Scheduler for
1905
builds to complete successfully (on all of its Builders). Each time
1906
that happens, the same source code (i.e. the same `SourceStamp') will
1907
be used to start a new set of builds, on a different set of Builders.
1908
This "downstream" scheduler doesn't pay attention to Changes at all,
1909
it only pays attention to the upstream scheduler.
1910
1911
If the SourceStamp fails on any of the Builders in the upstream
1912
set, the downstream builds will not fire.
1913
1914
from buildbot import scheduler
1915
tests = scheduler.Scheduler("tests", None, 5*60,
1916
["full-linux", "full-netbsd", "full-OSX"])
1917
package = scheduler.Dependent("package",
1918
tests, # upstream scheduler
1919
["make-tarball", "make-deb", "make-rpm"])
1920
c['schedulers'] = [tests, package]
1921
1922
Note that `Dependent''s upstream scheduler argument is given as a
1923
`Scheduler' _instance_, not a name. This makes it impossible to
1924
create circular dependencies in the config file.
1925
1926
1927
File: buildbot.info, Node: Setting the slaveport, Next: Buildslave Specifiers, Prev: Listing Change Sources and Schedulers, Up: Configuration
1928
1929
4.5 Setting the slaveport
1930
=========================
1931
1932
The buildmaster will listen on a TCP port of your choosing for
1933
connections from buildslaves. It can also use this port for
1934
connections from remote Change Sources, status clients, and debug
1935
tools. This port should be visible to the outside world, and you'll
1936
need to tell your buildslave admins about your choice.
1937
1938
It does not matter which port you pick, as long it is externally
1939
visible, however you should probably use something larger than 1024,
1940
since most operating systems don't allow non-root processes to bind to
1941
low-numbered ports. If your buildmaster is behind a firewall or a NAT
1942
box of some sort, you may have to configure your firewall to permit
1943
inbound connections to this port.
1944
1945
c['slavePortnum'] = 10000
1946
1947
`c['slavePortnum']' is a _strports_ specification string, defined
1948
in the `twisted.application.strports' module (try `pydoc
1949
twisted.application.strports' to get documentation on the format).
1950
This means that you can have the buildmaster listen on a
1951
localhost-only port by doing:
1952
1953
c['slavePortnum'] = "tcp:10000:interface=127.0.0.1"
1954
1955
This might be useful if you only run buildslaves on the same
1956
machine, and they are all configured to contact the buildmaster at
1957
`localhost:10000'.
1958
1959
1960
File: buildbot.info, Node: Buildslave Specifiers, Next: Defining Builders, Prev: Setting the slaveport, Up: Configuration
1961
1962
4.6 Buildslave Specifiers
1963
=========================
1964
1965
The `c['bots']' key is a list of known buildslaves. Each buildslave
1966
is defined by a tuple of (slavename, slavepassword). These are the
1967
same two values that need to be provided to the buildslave
1968
administrator when they create the buildslave.
1969
1970
c['bots'] = [('bot-solaris', 'solarispasswd'),
1971
('bot-bsd', 'bsdpasswd'),
1972
]
1973
1974
The slavenames must be unique, of course. The password exists to
1975
prevent evildoers from interfering with the buildbot by inserting
1976
their own (broken) buildslaves into the system and thus displacing the
1977
real ones.
1978
1979
Buildslaves with an unrecognized slavename or a non-matching
1980
password will be rejected when they attempt to connect, and a message
1981
describing the problem will be put in the log file (see *Note
1982
Logfiles::).
1983
1984
1985
File: buildbot.info, Node: Defining Builders, Next: Defining Status Targets, Prev: Buildslave Specifiers, Up: Configuration
1986
1987
4.7 Defining Builders
1988
=====================
1989
1990
The `c['builders']' key is a list of dictionaries which specify the
1991
Builders. The Buildmaster runs a collection of Builders, each of
1992
which handles a single type of build (e.g. full versus quick), on a
1993
single build slave. A Buildbot which makes sure that the latest code
1994
("HEAD") compiles correctly across four separate architecture will
1995
have four Builders, each performing the same build but on different
1996
slaves (one per platform).
1997
1998
Each Builder gets a separate column in the waterfall display. In
1999
general, each Builder runs independently (although various kinds of
2000
interlocks can cause one Builder to have an effect on another).
2001
2002
Each Builder specification dictionary has several required keys:
2003
2004
`name'
2005
This specifies the Builder's name, which is used in status
2006
reports.
2007
2008
`slavename'
2009
This specifies which buildslave will be used by this Builder.
2010
`slavename' must appear in the `c['bots']' list. Each buildslave
2011
can accomodate multiple Builders.
2012
2013
`slavenames'
2014
If you provide `slavenames' instead of `slavename', you can give
2015
a list of buildslaves which are capable of running this Builder.
2016
If multiple buildslaves are available for any given Builder, you
2017
will have some measure of redundancy: in case one slave goes
2018
offline, the others can still keep the Builder working. In
2019
addition, multiple buildslaves will allow multiple simultaneous
2020
builds for the same Builder, which might be useful if you have a
2021
lot of forced or "try" builds taking place.
2022
2023
If you use this feature, it is important to make sure that the
2024
buildslaves are all, in fact, capable of running the given
2025
build. The slave hosts should be configured similarly, otherwise
2026
you will spend a lot of time trying (unsuccessfully) to
2027
reproduce a failure that only occurs on some of the buildslaves
2028
and not the others. Different platforms, operating systems,
2029
versions of major programs or libraries, all these things mean
2030
you should use separate Builders.
2031
2032
`builddir'
2033
This specifies the name of a subdirectory (under the base
2034
directory) in which everything related to this builder will be
2035
placed. On the buildmaster, this holds build status information.
2036
On the buildslave, this is where checkouts, compiles, and tests
2037
are run.
2038
2039
`factory'
2040
This is a `buildbot.process.factory.BuildFactory' instance which
2041
controls how the build is performed. Full details appear in
2042
their own chapter, *Note Build Process::. Parameters like the
2043
location of the CVS repository and the compile-time options used
2044
for the build are generally provided as arguments to the
2045
factory's constructor.
2046
2047
2048
Other optional keys may be set on each Builder:
2049
2050
`category'
2051
If provided, this is a string that identifies a category for the
2052
builder to be a part of. Status clients can limit themselves to a
2053
subset of the available categories. A common use for this is to
2054
add new builders to your setup (for a new module, or for a new
2055
buildslave) that do not work correctly yet and allow you to
2056
integrate them with the active builders. You can put these new
2057
builders in a test category, make your main status clients
2058
ignore them, and have only private status clients pick them up.
2059
As soon as they work, you can move them over to the active
2060
category.
2061
2062
2063
2064
File: buildbot.info, Node: Defining Status Targets, Next: Debug options, Prev: Defining Builders, Up: Configuration
2065
2066
4.8 Defining Status Targets
2067
===========================
2068
2069
The Buildmaster has a variety of ways to present build status to
2070
various users. Each such delivery method is a "Status Target" object
2071
in the configuration's `status' list. To add status targets, you just
2072
append more objects to this list:
2073
2074
c['status'] = []
2075
2076
from buildbot.status import html
2077
c['status'].append(html.Waterfall(http_port=8010))
2078
2079
from buildbot.status import mail
2080
m = mail.MailNotifier(fromaddr="buildbot@localhost",
2081
extraRecipients=["builds@lists.example.com"],
2082
sendToInterestedUsers=False)
2083
c['status'].append(m)
2084
2085
from buildbot.status import words
2086
c['status'].append(words.IRC(host="irc.example.com", nick="bb",
2087
channels=["#example"]))
2088
2089
Status delivery has its own chapter, *Note Status Delivery::, in
2090
which all the built-in status targets are documented.
2091
2092
2093
File: buildbot.info, Node: Debug options, Prev: Defining Status Targets, Up: Configuration
2094
2095
4.9 Debug options
2096
=================
2097
2098
If you set `c['debugPassword']', then you can connect to the
2099
buildmaster with the diagnostic tool launched by `buildbot
2100
debugclient MASTER:PORT'. From this tool, you can reload the config
2101
file, manually force builds, and inject changes, which may be useful
2102
for testing your buildmaster without actually commiting changes to
2103
your repository (or before you have the Change Sources set up). The
2104
debug tool uses the same port number as the slaves do:
2105
`c['slavePortnum']', and is authenticated with this password.
2106
2107
c['debugPassword'] = "debugpassword"
2108
2109
If you set `c['manhole']' to an instance of the
2110
`buildbot.master.Manhole' class, you can telnet into the buildmaster
2111
and get an interactive Python shell, which may be useful for
2112
debugging buildbot internals. It is probably only useful for buildbot
2113
developers. It exposes full access to the buildmaster's account
2114
(including the ability to modify and delete files), so it should not
2115
be enabled with a weak or easily guessable password.
2116
2117
The `Manhole' instance can be configured to listen on a specific
2118
port. You may wish to have this listening port bind to the loopback
2119
interface (sometimes known as "lo0", "localhost", or 127.0.0.1) to
2120
restrict access to clients which are running on the same host.
2121
2122
from buildbot.master import Manhole
2123
c['manhole'] = Manhole("tcp:9999:interface=127.0.0.1", "admin", "password")
2124
2125
To have the `Manhole' listen on all interfaces, use `"tcp:9999"'.
2126
This port specification uses `twisted.application.strports', so you
2127
can make it listen on SSL or even UNIX-domain sockets if you want.
2128
2129
2130
File: buildbot.info, Node: Getting Source Code Changes, Next: Build Process, Prev: Configuration, Up: Top
2131
2132
5 Getting Source Code Changes
2133
*****************************
2134
2135
The most common way to use the Buildbot is centered around the idea of
2136
`Source Trees': a directory tree filled with source code of some form
2137
which can be compiled and/or tested. Some projects use languages that
2138
don't involve any compilation step: nevertheless there may be a
2139
`build' phase where files are copied or rearranged into a form that
2140
is suitable for installation. Some projects do not have unit tests,
2141
and the Buildbot is merely helping to make sure that the sources can
2142
compile correctly. But in all of these cases, the thing-being-tested
2143
is a single source tree.
2144
2145
A Version Control System mantains a source tree, and tells the
2146
buildmaster when it changes. The first step of each Build is typically
2147
to acquire a copy of some version of this tree.
2148
2149
This chapter describes how the Buildbot learns about what Changes
2150
have occurred. For more information on VC systems and Changes, see
2151
*Note Version Control Systems::.
2152
2153
* Menu:
2154
2155
* Change Sources::
2156
2157
2158
File: buildbot.info, Node: Change Sources, Prev: Getting Source Code Changes, Up: Getting Source Code Changes
2159
2160
5.1 Change Sources
2161
==================
2162
2163
Each Buildmaster watches a single source tree. Changes can be provided
2164
by a variety of ChangeSource types, however any given project will
2165
typically have only a single ChangeSource active. This section
2166
provides a description of all available ChangeSource types and
2167
explains how to set up each of them.
2168
2169
* Menu:
2170
2171
* Choosing ChangeSources::
2172
* CVSToys - PBService::
2173
* CVSToys - mail notification::
2174
* Other mail notification ChangeSources::
2175
* PBChangeSource::
2176
2177
2178
File: buildbot.info, Node: Choosing ChangeSources, Next: CVSToys - PBService, Prev: Change Sources, Up: Change Sources
2179
2180
5.1.1 Choosing ChangeSources
2181
----------------------------
2182
2183
The `master.cfg' configuration file has a dictionary key named
2184
`BuildmasterConfig['sources']', which holds a list of `IChangeSource'
2185
objects. The config file will typically create an object from one of
2186
the classes described below and stuff it into the list.
2187
2188
s = FreshCVSSourceNewcred(host="host", port=4519,
2189
user="alice", passwd="secret",
2190
prefix="Twisted")
2191
BuildmasterConfig['sources'] = [s]
2192
2193
Each source tree has a nominal `top'. Each Change has a list of
2194
filenames, which are all relative to this top location. The
2195
ChangeSource is responsible for doing whatever is necessary to
2196
accomplish this. Most sources have a `prefix' argument: a partial
2197
pathname which is stripped from the front of all filenames provided to
2198
that `ChangeSource'. Files which are outside this sub-tree are
2199
ignored by the changesource: it does not generate Changes for those
2200
files.
2201
2202
2203
File: buildbot.info, Node: CVSToys - PBService, Next: CVSToys - mail notification, Prev: Choosing ChangeSources, Up: Change Sources
2204
2205
5.1.2 CVSToys - PBService
2206
-------------------------
2207
2208
The CVSToys (http://purl.net/net/CVSToys) package provides a server
2209
which runs on the machine that hosts the CVS repository it watches.
2210
It has a variety of ways to distribute commit notifications, and
2211
offers a flexible regexp-based way to filter out uninteresting
2212
changes. One of the notification options is named `PBService' and
2213
works by listening on a TCP port for clients. These clients subscribe
2214
to hear about commit notifications.
2215
2216
The buildmaster has a CVSToys-compatible `PBService' client built
2217
in. There are two versions of it, one for old versions of CVSToys
2218
(1.0.9 and earlier) which used the `oldcred' authentication
2219
framework, and one for newer versions (1.0.10 and later) which use
2220
`newcred'. Both are classes in the `buildbot.changes.freshcvs'
2221
package.
2222
2223
`FreshCVSSourceNewcred' objects are created with the following
2224
parameters:
2225
2226
``host' and `port''
2227
these specify where the CVSToys server can be reached
2228
2229
``user' and `passwd''
2230
these specify the login information for the CVSToys server
2231
(`freshcvs'). These must match the server's values, which are
2232
defined in the `freshCfg' configuration file (which lives in the
2233
CVSROOT directory of the repository).
2234
2235
``prefix''
2236
this is the prefix to be found and stripped from filenames
2237
delivered by the CVSToys server. Most projects live in
2238
sub-directories of the main repository, as siblings of the
2239
CVSROOT sub-directory, so typically this prefix is set to that
2240
top sub-directory name.
2241
2242
2243
Example
2244
=======
2245
2246
To set up the freshCVS server, add a statement like the following to
2247
your `freshCfg' file:
2248
2249
pb = ConfigurationSet([
2250
(None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
2251
])
2252
2253
This will announce all changes to a client which connects to port
2254
4519 using a username of 'foo' and a password of 'bar'.
2255
2256
Then add a clause like this to your buildmaster's `master.cfg':
2257
2258
BuildmasterConfig['sources'] = [FreshCVSSource("cvs.example.com", 4519,
2259
"foo", "bar",
2260
prefix="glib/")]
2261
2262
where "cvs.example.com" is the host that is running the FreshCVS
2263
daemon, and "glib" is the top-level directory (relative to the
2264
repository's root) where all your source code lives. Most projects
2265
keep one or more projects in the same repository (along with CVSROOT/
2266
to hold admin files like loginfo and freshCfg); the prefix= argument
2267
tells the buildmaster to ignore everything outside that directory,
2268
and to strip that common prefix from all pathnames it handles.
2269
2270
2271
File: buildbot.info, Node: CVSToys - mail notification, Next: Other mail notification ChangeSources, Prev: CVSToys - PBService, Up: Change Sources
2272
2273
5.1.3 CVSToys - mail notification
2274
---------------------------------
2275
2276
CVSToys also provides a `MailNotification' action which will send
2277
email to a list of recipients for each commit. This tends to work
2278
better than using `/bin/mail' from within the CVSROOT/loginfo file
2279
directly, as CVSToys will batch together all files changed during the
2280
same CVS invocation, and can provide more information (like creating
2281
a ViewCVS URL for each file changed).
2282
2283
The Buildbot's `FCMaildirSource' is a ChangeSource which knows how
2284
to parse these CVSToys messages and turn them into Change objects.
2285
It watches a Maildir for new messages. The usually installation
2286
process looks like:
2287
2288
1. Create a mailing list, `projectname-commits'.
2289
2290
2. In CVSToys' freshCfg file, use a `MailNotification' action to
2291
send commit mail to this mailing list.
2292
2293
3. Subscribe the buildbot user to the mailing list.
2294
2295
4. Configure your .qmail or .forward file to deliver these messages
2296
into a maildir.
2297
2298
5. In the Buildbot's master.cfg file, use a `FCMaildirSource' to
2299
watch the maildir for commit messages.
2300
2301
The `FCMaildirSource' is created with two parameters: the
2302
directory name of the maildir root, and the prefix to strip.
2303
2304
2305
File: buildbot.info, Node: Other mail notification ChangeSources, Next: PBChangeSource, Prev: CVSToys - mail notification, Up: Change Sources
2306
2307
5.1.4 Other mail notification ChangeSources
2308
-------------------------------------------
2309
2310
There are other types of maildir-watching ChangeSources, which only
2311
differ in the function used to parse the message body.
2312
2313
`SyncmailMaildirSource' knows how to parse the message format used
2314
in mail sent by Syncmail.
2315
2316
`BonsaiMaildirSource' parses messages sent out by Bonsai.
2317
2318
2319
File: buildbot.info, Node: PBChangeSource, Prev: Other mail notification ChangeSources, Up: Change Sources
2320
2321
5.1.5 PBChangeSource
2322
--------------------
2323
2324
The last kind of ChangeSource actually listens on a TCP port for
2325
clients to connect and push change notices _into_ the Buildmaster.
2326
This is used by the built-in `buildbot sendchange' notification tool,
2327
as well as the VC-specific `contrib/svn_buildbot.py' and
2328
`contrib/arch_buildbot.py' tools. These tools are run by the
2329
repository (in a commit hook script), and connect to the buildmaster
2330
directly each time a file is comitted. This is also useful for
2331
creating new kinds of change sources that work on a `push' model
2332
instead of some kind of subscription scheme, for example a script
2333
which is run out of an email .forward file.
2334
2335
This ChangeSource can be configured to listen on its own TCP port,
2336
or it can share the port that the buildmaster is already using for the
2337
buildslaves to connect. (This is possible because the
2338
`PBChangeSource' uses the same protocol as the buildslaves, and they
2339
can be distinguished by the `username' attribute used when the
2340
initial connection is established). It might be useful to have it
2341
listen on a different port if, for example, you wanted to establish
2342
different firewall rules for that port. You could allow only the SVN
2343
repository machine access to the `PBChangeSource' port, while
2344
allowing only the buildslave machines access to the slave port. Or you
2345
could just expose one port and run everything over it. _Note: this
2346
feature is not yet implemented, the PBChangeSource will always share
2347
the slave port and will always have a `user' name of `change', and a
2348
passwd of `changepw'. These limitations will be removed in the
2349
future._.
2350
2351
The `PBChangeSource' is created with the following arguments:
2352
2353
``port''
2354
which port to listen on. If `None' (which is the default), it
2355
shares the port used for buildslave connections. _Not
2356
Implemented, always set to `None'_.
2357
2358
``user' and `passwd''
2359
the user/passwd account information that the client program must
2360
use to connect. Defaults to `change' and `changepw'. _Not
2361
Implemented, `user' is currently always set to `change',
2362
`passwd' is always set to `changepw'_.
2363
2364
``prefix''
2365
the prefix to be found and stripped from filenames delivered
2366
over the connection.
2367
2368
2369
File: buildbot.info, Node: Build Process, Next: Status Delivery, Prev: Getting Source Code Changes, Up: Top
2370
2371
6 Build Process
2372
***************
2373
2374
A `Build' object is responsible for actually performing a build. It
2375
gets access to a remote `SlaveBuilder' where it may run commands, and
2376
a `BuildStatus' object where it must emit status events. The `Build'
2377
is created by the Builder's `BuildFactory'.
2378
2379
The default `Build' class is made up of a fixed sequence of
2380
`BuildSteps', executed one after another until all are complete (or
2381
one of them indicates that the build should be halted early). The
2382
default `BuildFactory' creates instances of this `Build' class with a
2383
list of `BuildSteps', so the basic way to configure the build is to
2384
provide a list of `BuildSteps' to your `BuildFactory'.
2385
2386
More complicated `Build' subclasses can make other decisions:
2387
execute some steps only if certain files were changed, or if certain
2388
previous steps passed or failed. The base class has been written to
2389
allow users to express basic control flow without writing code, but
2390
you can always subclass and customize to achieve more specialized
2391
behavior.
2392
2393
* Menu:
2394
2395
* Build Steps::
2396
* Interlocks::
2397
* Build Factories::
2398
2399
2400
File: buildbot.info, Node: Build Steps, Next: Interlocks, Prev: Build Process, Up: Build Process
2401
2402
6.1 Build Steps
2403
===============
2404
2405
`BuildStep's are usually specified in the buildmaster's configuration
2406
file, in a list of "step specifications" that is used to create the
2407
`BuildFactory'. These "step specifications" are not actual steps, but
2408
rather a tuple of the `BuildStep' subclass to be created and a
2409
dictionary of arguments. (the actual `BuildStep' instances are not
2410
created until the Build is started, so that each Build gets an
2411
independent copy of each BuildStep). There is a convenience function
2412
named "`s'" in the `buildbot.process.factory' module for creating
2413
these specification tuples. It allows you to create a
2414
`BuildFactory'-ready list like this:
2415
2416
from buildbot.process import step, factory
2417
from buildbot.process.factory import s
2418
2419
steps = [s(step.SVN, svnurl="http://svn.example.org/Trunk/"),
2420
s(step.ShellCommand, command=["make", "all"]),
2421
s(step.ShellCommand, command=["make", "test"]),
2422
]
2423
f = factory.BuildFactory(steps)
2424
2425
The rest of this section lists all the standard BuildStep objects
2426
available for use in a Build, and the parameters which can be used to
2427
control each.
2428
2429
* Menu:
2430
2431
* Common Parameters::
2432
* Source Checkout::
2433
* ShellCommand::
2434
* Simple ShellCommand Subclasses::
2435
2436
2437
File: buildbot.info, Node: Common Parameters, Next: Source Checkout, Prev: Build Steps, Up: Build Steps
2438
2439
6.1.1 Common Parameters
2440
-----------------------
2441
2442
The standard `Build' runs a series of `BuildStep's in order, only
2443
stopping when it runs out of steps or if one of them requests that
2444
the build be halted. It collects status information from each one to
2445
create an overall build status (of SUCCESS, WARNINGS, or FAILURE).
2446
2447
All BuildSteps accept some common parameters. Some of these control
2448
how their individual status affects the overall build. Others are used
2449
to specify which `Locks' (see *note Interlocks::) should be acquired
2450
before allowing the step to run.
2451
2452
Arguments common to all `BuildStep' subclasses:
2453
2454
`name'
2455
the name used to describe the step on the status display. It is
2456
also used to give a name to any LogFiles created by this step.
2457
2458
`haltOnFailure'
2459
if True, a FAILURE of this build step will cause the build to
2460
halt immediately with an overall result of FAILURE.
2461
2462
`flunkOnWarnings'
2463
when True, a WARNINGS or FAILURE of this build step will mark the
2464
overall build as FAILURE. The remaining steps will still be
2465
executed.
2466
2467
`flunkOnFailure'
2468
when True, a FAILURE of this build step will mark the overall
2469
build as a FAILURE. The remaining steps will still be executed.
2470
2471
`warnOnWarnings'
2472
when True, a WARNINGS or FAILURE of this build step will mark the
2473
overall build as having WARNINGS. The remaining steps will still
2474
be executed.
2475
2476
`warnOnFailure'
2477
when True, a FAILURE of this build step will mark the overall
2478
build as having WARNINGS. The remaining steps will still be
2479
executed.
2480
2481
`locks'
2482
a list of Locks (instances of `buildbot.locks.SlaveLock' or
2483
`buildbot.locks.MasterLock') that should be acquired before
2484
starting this Step. The Locks will be released when the step is
2485
complete. Note that this is a list of actual Lock instances, not
2486
names. Also note that all Locks must have unique names.
2487
2488
2489
2490
File: buildbot.info, Node: Source Checkout, Next: ShellCommand, Prev: Common Parameters, Up: Build Steps
2491
2492
6.1.2 Source Checkout
2493
---------------------
2494
2495
The first step of any build is typically to acquire the source code
2496
from which the build will be performed. There are several classes to
2497
handle this, one for each of the different source control system that
2498
Buildbot knows about. For a description of how Buildbot treats source
2499
control in general, see *Note Version Control Systems::.
2500
2501
All source checkout steps accept some common parameters to control
2502
how they get the sources and where they should be placed. The
2503
remaining per-VC-system parameters are mostly to specify where
2504
exactly the sources are coming from.
2505
2506
`mode'
2507
a string describing the kind of VC operation that is desired.
2508
Defaults to `update'.
2509
2510
`update'
2511
specifies that the CVS checkout/update should be performed
2512
directly into the workdir. Each build is performed in the
2513
same directory, allowing for incremental builds. This
2514
minimizes disk space, bandwidth, and CPU time. However, it
2515
may encounter problems if the build process does not handle
2516
dependencies properly (sometimes you must do a "clean
2517
build" to make sure everything gets compiled), or if source
2518
files are deleted but generated files can influence test
2519
behavior (e.g. python's .pyc files), or when source
2520
directories are deleted but generated files prevent CVS
2521
from removing them. Builds ought to be correct regardless
2522
of whether they are done "from scratch" or incrementally,
2523
but it is useful to test both kinds: this mode exercises the
2524
incremental-build style.
2525
2526
`copy'
2527
specifies that the CVS workspace should be maintained in a
2528
separate directory (called the 'copydir'), using checkout
2529
or update as necessary. For each build, a new workdir is
2530
created with a copy of the source tree (rm -rf workdir; cp
2531
-r copydir workdir). This doubles the disk space required,
2532
but keeps the bandwidth low (update instead of a full
2533
checkout). A full 'clean' build is performed each time. This
2534
avoids any generated-file build problems, but is still
2535
occasionally vulnerable to CVS problems such as a
2536
repository being manually rearranged, causing CVS errors on
2537
update which are not an issue with a full checkout.
2538
2539
`clobber'
2540
specifes that the working directory should be deleted each
2541
time, necessitating a full checkout for each build. This
2542
insures a clean build off a complete checkout, avoiding any
2543
of the problems described above. This mode exercises the
2544
"from-scratch" build style.
2545
2546
`export'
2547
this is like `clobber', except that the 'cvs export'
2548
command is used to create the working directory. This
2549
command removes all CVS metadata files (the CVS/
2550
directories) from the tree, which is sometimes useful for
2551
creating source tarballs (to avoid including the metadata
2552
in the tar file).
2553
2554
`workdir'
2555
like all Steps, this indicates the directory where the build
2556
will take place. Source Steps are special in that they perform
2557
some operations outside of the workdir (like creating the
2558
workdir itself).
2559
2560
`alwaysUseLatest'
2561
if True, bypass the usual "update to the last Change" behavior,
2562
and always update to the latest changes instead.
2563
2564
`retry'
2565
If set, this specifies a tuple of `(delay, repeats)' which means
2566
that when a full VC checkout fails, it should be retried up to
2567
REPEATS times, waiting DELAY seconds between attempts. If you
2568
don't provide this, it defaults to `None', which means VC
2569
operations should not be retried. This is provided to make life
2570
easier for buildslaves which are stuck behind poor network
2571
connections.
2572
2573
2574
My habit as a developer is to do a `cvs update' and `make' each
2575
morning. Problems can occur, either because of bad code being checked
2576
in, or by incomplete dependencies causing a partial rebuild to fail
2577
where a complete from-scratch build might succeed. A quick Builder
2578
which emulates this incremental-build behavior would use the
2579
`mode='update'' setting.
2580
2581
On the other hand, other kinds of dependency problems can cause a
2582
clean build to fail where a partial build might succeed. This
2583
frequently results from a link step that depends upon an object file
2584
that was removed from a later version of the tree: in the partial
2585
tree, the object file is still around (even though the Makefiles no
2586
longer know how to create it).
2587
2588
"official" builds (traceable builds performed from a known set of
2589
source revisions) are always done as clean builds, to make sure it is
2590
not influenced by any uncontrolled factors (like leftover files from a
2591
previous build). A "full" Builder which behaves this way would want
2592
to use the `mode='clobber'' setting.
2593
2594
Each VC system has a corresponding source checkout class: their
2595
arguments are described on the following pages.
2596
2597
* Menu:
2598
2599
* CVS::
2600
* SVN::
2601
* Darcs::
2602
* Arch::
2603
* Bazaar::
2604
* P4Sync::
2605
2606
2607
File: buildbot.info, Node: CVS, Next: SVN, Prev: Source Checkout, Up: Source Checkout
2608
2609
6.1.2.1 CVS
2610
...........
2611
2612
The `CVS' build step performs a CVS (http://www.nongnu.org/cvs/)
2613
checkout or update. It takes the following arguments:
2614
2615
`cvsroot'
2616
(required): specify the CVSROOT value, which points to a CVS
2617
repository, probably on a remote machine. For example, the
2618
cvsroot value you would use to get a copy of the Buildbot source
2619
code is
2620
`:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot'
2621
2622
`cvsmodule'
2623
(required): specify the cvs `module', which is generally a
2624
subdirectory of the CVSROOT. The cvsmodule for the Buildbot
2625
source code is `buildbot'.
2626
2627
`branch'
2628
a string which will be used in a `-r' argument. This is most
2629
useful for specifying a branch to work on. Defaults to `HEAD'.
2630
2631
`global_options'
2632
a list of flags to be put before the verb in the CVS command.
2633
2634
`checkoutDelay'
2635
if set, the number of seconds to put between the timestamp of
2636
the last known Change and the value used for the `-D' option.
2637
Defaults to half of the parent Build's treeStableTimer.
2638
2639
2640
2641
File: buildbot.info, Node: SVN, Next: Darcs, Prev: CVS, Up: Source Checkout
2642
2643
6.1.2.2 SVN
2644
...........
2645
2646
The `SVN' build step performs a Subversion
2647
(http://subversion.tigris.org) checkout or update. There are two
2648
basic ways of setting up the checkout step, depending upon whether
2649
you are using multiple branches or not.
2650
2651
If all of your builds use the same branch, then you should create
2652
the `SVN' step with the `svnurl' argument:
2653
2654
`svnurl'
2655
(required): this specifies the `URL' argument that will be given
2656
to the `svn checkout' command. It dictates both where the
2657
repository is located and which sub-tree should be extracted. In
2658
this respect, it is like a combination of the CVS `cvsroot' and
2659
`cvsmodule' arguments. For example, if you are using a remote
2660
Subversion repository which is accessible through HTTP at a URL
2661
of `http://svn.example.com/repos', and you wanted to check out
2662
the `trunk/calc' sub-tree, you would use
2663
`svnurl="http://svn.example.com/repos/trunk/calc"' as an argument
2664
to your `SVN' step.
2665
2666
If, on the other hand, you are building from multiple branches,
2667
then you should create the `SVN' step with the `baseURL' and
2668
`defaultBranch' arguments instead:
2669
2670
`baseURL'
2671
(required): this specifies the base repository URL, to which a
2672
branch name will be appended. It should probably end in a slash.
2673
2674
`defaultBranch'
2675
this specifies the name of the branch to use when a Build does
2676
not provide one of its own. This will be appended to `baseURL' to
2677
create the string that will be passed to the `svn checkout'
2678
command.
2679
2680
If you are using branches, you must also make sure your
2681
`ChangeSource' will report the correct branch names.
2682
2683
branch example
2684
==============
2685
2686
Let's suppose that the "MyProject" repository uses branches for the
2687
trunk, for various users' individual development efforts, and for
2688
several new features that will require some amount of work (involving
2689
multiple developers) before they are ready to merge onto the trunk.
2690
Such a repository might be organized as follows:
2691
2692
svn://svn.example.org/MyProject/trunk
2693
svn://svn.example.org/MyProject/branches/User1/foo
2694
svn://svn.example.org/MyProject/branches/User1/bar
2695
svn://svn.example.org/MyProject/branches/User2/baz
2696
svn://svn.example.org/MyProject/features/newthing
2697
svn://svn.example.org/MyProject/features/otherthing
2698
2699
Further assume that we want the Buildbot to run tests against the
2700
trunk and against all the feature branches (i.e., do a
2701
checkout/compile/build of branch X when a file has been changed on
2702
branch X, when X is in the set [trunk, features/newthing,
2703
features/otherthing]). We do not want the Buildbot to automatically
2704
build any of the user branches, but it should be willing to build a
2705
user branch when explicitly requested (most likely by the user who
2706
owns that branch).
2707
2708
There are three things that need to be set up to accomodate this
2709
system. The first is a ChangeSource that is capable of identifying the
2710
branch which owns any given file. This depends upon a user-supplied
2711
function, in an external program that runs in the SVN commit hook and
2712
connects to the buildmaster's `PBChangeSource' over a TCP connection.
2713
(you can use the "`buildbot sendchange'" utility for this purpose,
2714
but you will still need an external program to decide what value
2715
should be passed to the `--branch=' argument). For example, a change
2716
to a file with the SVN url of
2717
"svn://svn.example.org/MyProject/features/newthing/src/foo.c" should
2718
be broken down into a Change instance with
2719
`branch='features/newthing'' and `file='src/foo.c''.
2720
2721
The second piece is an `AnyBranchScheduler' which will pay
2722
attention to the desired branches. It will not pay attention to the
2723
user branches, so it will not automatically start builds in response
2724
to changes there. The AnyBranchScheduler class requires you to
2725
explicitly list all the branches you want it to use, but it would not
2726
be difficult to write a subclass which used
2727
`branch.startswith('features/'' to remove the need for this explicit
2728
list. Or, if you want to build user branches too, you can use
2729
AnyBranchScheduler with `branches=None' to indicate that you want it
2730
to pay attention to all branches.
2731
2732
The third piece is an `SVN' checkout step that is configured to
2733
handle the branches correctly, with a `baseURL' value that matches
2734
the way the ChangeSource splits each file's URL into base, branch,
2735
and file.
2736
2737
from buildbot.changes.pb import PBChangeSource
2738
from buildbot.scheduler import AnyBranchScheduler
2739
from buildbot.process import step, factory
2740
from buildbot.process.factory import s
2741
2742
c['sources'] = [PBChangeSource()]
2743
s1 = AnyBranchScheduler('main',
2744
['trunk', 'features/newthing', 'features/otherthing'],
2745
10*60, ['test-i386', 'test-ppc'])
2746
c['schedulers'] = [s1]
2747
source = s(step.SVN, mode='update',
2748
baseURL='svn://svn.example.org/MyProject/',
2749
defaultBranch='trunk')
2750
f = factory.BuildFactory([source,
2751
s(step.Compile, command="make all"),
2752
s(step.Test, command="make test")])
2753
c['builders'] = [
2754
{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
2755
'factory':f },
2756
{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
2757
'factory':f },
2758
]
2759
2760
In this example, when a change arrives with a `branch' attribute
2761
of "trunk", the resulting build will have an SVN step that
2762
concatenates "svn://svn.example.org/MyProject/" (the baseURL) with
2763
"trunk" (the branch name) to get the correct svn command. If the
2764
"newthing" branch has a change to "src/foo.c", then the SVN step will
2765
concatenate "svn://svn.example.org/MyProject/" with
2766
"features/newthing" to get the svnurl for checkout.
2767
2768
2769
File: buildbot.info, Node: Darcs, Next: Arch, Prev: SVN, Up: Source Checkout
2770
2771
6.1.2.3 Darcs
2772
.............
2773
2774
The `Darcs' build step performs a Darcs
2775
(http://abridgegame.org/darcs/) checkout or update.
2776
2777
Like *Note SVN::, this step can either be configured to always
2778
check out a specific tree, or set up to pull from a particular branch
2779
that gets specified separately for each build. Also like SVN, the
2780
repository URL given to Darcs is created by concatenating a `baseURL'
2781
with the branch name, and if no particular branch is requested, it
2782
uses a `defaultBranch'. The only difference in usage is that each
2783
potential Darcs repository URL must point to a fully-fledged
2784
repository, whereas SVN URLs usually point to sub-trees of the main
2785
Subversion repository. In other words, doing an SVN checkout of
2786
`baseURL' is legal, but silly, since you'd probably wind up with a
2787
copy of every single branch in the whole repository. Doing a Darcs
2788
checkout of `baseURL' is just plain wrong, since the parent directory
2789
of a collection of Darcs repositories is not itself a valid
2790
repository.
2791
2792
The Darcs step takes the following arguments:
2793
2794
`repourl'
2795
(required unless `baseURL' is provided): the URL at which the
2796
Darcs source repository is available.
2797
2798
`baseURL'
2799
(required unless `repourl' is provided): the base repository URL,
2800
to which a branch name will be appended. It should probably end
2801
in a slash.
2802
2803
`defaultBranch'
2804
(allowed if and only if `baseURL' is provided): this specifies
2805
the name of the branch to use when a Build does not provide one
2806
of its own. This will be appended to `baseURL' to create the
2807
string that will be passed to the `darcs get' command.
2808
2809
2810
File: buildbot.info, Node: Arch, Next: Bazaar, Prev: Darcs, Up: Source Checkout
2811
2812
6.1.2.4 Arch
2813
............
2814
2815
The `Arch' build step performs an Arch (http://gnuarch.org/) checkout
2816
or update using the `tla' client. It takes the following arguments:
2817
2818
`url'
2819
(required): this specifies the URL at which the Arch source
2820
archive is available.
2821
2822
`version'
2823
(required): this specifies which "development line" (like a
2824
branch) should be used. This provides the default branch name,
2825
but individual builds may specify a different one.
2826
2827
`archive'
2828
(optional): Each repository knows its own archive name. If this
2829
parameter is provided, it must match the repository's archive
2830
name. The parameter is accepted for compatibility with the
2831
`Bazaar' step, below.
2832
2833
2834
2835
File: buildbot.info, Node: Bazaar, Next: P4Sync, Prev: Arch, Up: Source Checkout
2836
2837
6.1.2.5 Bazaar
2838
..............
2839
2840
`Bazaar' is an alternate implementation of the Arch VC system, which
2841
uses a client named `baz'. The checkout semantics are just different
2842
enough from `tla' that there is a separate BuildStep for it.
2843
2844
It takes exactly the same arguments as `Arch', except that the
2845
`archive=' parameter is required. (baz does not emit the archive name
2846
when you do `baz register-archive', so we must provide it ourselves).
2847
2848
2849
File: buildbot.info, Node: P4Sync, Prev: Bazaar, Up: Source Checkout
2850
2851
6.1.2.6 P4Sync
2852
..............
2853
2854
The `P4Sync' build step performs a Perforce
2855
(http://www.perforce.com/) update. It is a temporary facility: a more
2856
complete P4 checkout step (named `P4') will eventually replace it.
2857
This step requires significant manual setup on each build slave. It
2858
takes the following arguments.
2859
2860
`p4port'
2861
(required): the host:port string describing how to get to the P4
2862
Depot (repository), used as the P4PORT environment variable for
2863
all p4 commands
2864
2865
2866
File: buildbot.info, Node: ShellCommand, Next: Simple ShellCommand Subclasses, Prev: Source Checkout, Up: Build Steps
2867
2868
6.1.3 ShellCommand
2869
------------------
2870
2871
This is a useful base class for just about everything you might want
2872
to do during a build (except for the initial source checkout). It runs
2873
a single command in a child shell on the buildslave. All stdout/stderr
2874
is recorded into a LogFile. The step finishes with a status of FAILURE
2875
if the command's exit code is non-zero, otherwise it has a status of
2876
SUCCESS.
2877
2878
The preferred way to specify the command is with a list of argv
2879
strings, since this allows for spaces in filenames and avoids doing
2880
any fragile shell-escaping. You can also specify the command with a
2881
single string, in which case the string is given to '/bin/sh -c
2882
COMMAND' for parsing.
2883
2884
All ShellCommands are run by default in the "workdir", which
2885
defaults to the "`build'" subdirectory of the slave builder's base
2886
directory. The absolute path of the workdir will thus be the slave's
2887
basedir (set as an option to `buildbot slave', *note Creating a
2888
buildslave::) plus the builder's basedir (set in the builder's
2889
`c['builddir']' key in master.cfg) plus the workdir itself (a
2890
class-level attribute of the BuildFactory, defaults to "`build'").
2891
2892
`ShellCommand' arguments:
2893
2894
`command'
2895
a list of strings (preferred) or single string (discouraged)
2896
which specifies the command to be run
2897
2898
`env'
2899
a dictionary of environment strings which will be added to the
2900
child command's environment.
2901
2902
`want_stdout'
2903
if False, stdout from the child process is discarded rather than
2904
being sent to the buildmaster for inclusion in the step's
2905
LogFile.
2906
2907
`want_stderr'
2908
like `want_stdout' but for stderr. Note that commands run through
2909
a PTY do not have separate stdout/stderr streams: both are
2910
merged into stdout.
2911
2912
`timeout'
2913
if the command fails to produce any output for this many
2914
seconds, it is assumed to be locked up and will be killed.
2915
2916
2917
File: buildbot.info, Node: Simple ShellCommand Subclasses, Prev: ShellCommand, Up: Build Steps
2918
2919
6.1.4 Simple ShellCommand Subclasses
2920
------------------------------------
2921
2922
Several subclasses of ShellCommand are provided as starting points for
2923
common build steps. These are all very simple: they just override a
2924
few parameters so you don't have to specify them yourself, making the
2925
master.cfg file less verbose.
2926
2927
* Menu:
2928
2929
* Configure::
2930
* Compile::
2931
* Test::
2932
2933
2934
File: buildbot.info, Node: Configure, Next: Compile, Prev: Simple ShellCommand Subclasses, Up: Simple ShellCommand Subclasses
2935
2936
6.1.4.1 Configure
2937
.................
2938
2939
This is intended to handle the `./configure' step from autoconf-style
2940
projects, or the `perl Makefile.PL' step from perl MakeMaker.pm-style
2941
modules. The default command is `./configure' but you can change this
2942
by providing a `command=' parameter.
2943
2944
2945
File: buildbot.info, Node: Compile, Next: Test, Prev: Configure, Up: Simple ShellCommand Subclasses
2946
2947
6.1.4.2 Compile
2948
...............
2949
2950
This is meant to handle compiling or building a project written in C.
2951
The default command is `make all'. When the compile is finished, the
2952
log file is scanned for GCC error/warning messages and a summary log
2953
is created with any problems that were seen (TODO: the summary is not
2954
yet created).
2955
2956
2957
File: buildbot.info, Node: Test, Prev: Compile, Up: Simple ShellCommand Subclasses
2958
2959
6.1.4.3 Test
2960
............
2961
2962
This is meant to handle unit tests. The default command is `make
2963
test', and the `warnOnFailure' flag is set.
2964
2965
2966
File: buildbot.info, Node: Interlocks, Next: Build Factories, Prev: Build Steps, Up: Build Process
2967
2968
6.2 Interlocks
2969
==============
2970
2971
For various reasons, you may want to prevent certain Steps (or perhaps
2972
entire Builds) from running simultaneously. Limited CPU speed or
2973
network bandwidth to the VC server, problems with simultaneous access
2974
to a database server used by unit tests, or multiple Builds which
2975
access shared state may all require some kind of interlock to prevent
2976
corruption, confusion, or resource overload.
2977
2978
`Locks' are the mechanism used to express these kinds of
2979
constraints on when Builds or Steps can be run. There are two kinds of
2980
`Locks', each with their own scope: `SlaveLock's are scoped to a
2981
single buildslave, while `MasterLock' instances are scoped to the
2982
buildbot as a whole. Each `Lock' is created with a unique name.
2983
2984
To use a lock, simply include it in the `locks=' argument of the
2985
`BuildStep' object that should obtain the lock before it runs. This
2986
argument accepts a list of `Lock' objects: the Step will acquire all
2987
of them before it runs.
2988
2989
To claim a lock for the whole Build, add a `'locks'' key to the
2990
builder specification dictionary with the same list of `Lock'
2991
objects. (This is the dictionary that has the `'name'',
2992
`'slavename'', `'builddir'', and `'factory'' keys). The `Build'
2993
object also accepts a `locks=' argument, but unless you are writing
2994
your own `BuildFactory' subclass then it will be easier to set the
2995
locks in the builder dictionary.
2996
2997
Note that there are no partial-acquire or partial-release
2998
semantics: this prevents deadlocks caused by two Steps each waiting
2999
for a lock held by the other. This also means that waiting to acquire
3000
a `Lock' can take an arbitrarily long time: if the buildmaster is
3001
very busy, a Step or Build which requires only one `Lock' may starve
3002
another that is waiting for that `Lock' plus some others. (1)
3003
3004
In the following example, we run the same build on three different
3005
platforms. The unit-test steps of these builds all use a common
3006
database server, and would interfere with each other if allowed to run
3007
simultaneously. The `Lock' prevents more than one of these builds
3008
from happening at the same time.
3009
3010
from buildbot import locks
3011
from buildbot.process import s, step, factory
3012
3013
db_lock = locks.MasterLock("database")
3014
steps = [s(step.SVN, svnurl="http://example.org/svn/Trunk"),
3015
s(step.ShellCommand, command="make all"),
3016
s(step.ShellCommand, command="make test", locks=[db_lock]),
3017
]
3018
f = factory.BuildFactory(steps)
3019
b1 = {'name': 'full1', 'slavename': 'bot-1, builddir='f1', 'factory': f}
3020
b2 = {'name': 'full2', 'slavename': 'bot-2, builddir='f2', 'factory': f}
3021
b3 = {'name': 'full3', 'slavename': 'bot-3, builddir='f3', 'factory': f}
3022
c['builders'] = [b1, b2, b3]
3023
3024
In the next example, we have one buildslave hosting three separate
3025
Builders (each running tests against a different version of Python).
3026
The machine which hosts this buildslave is not particularly fast, so
3027
we want to prevent the builds from all happening at the same time. We
3028
use a `SlaveLock' because the builds happening on the slow slave do
3029
not affect builds running on other slaves, and we use the lock on the
3030
build as a whole because the slave is so slow that even multiple SVN
3031
checkouts would be taxing.
3032
3033
from buildbot import locks
3034
from buildbot.process import s, step, factory
3035
3036
slow_lock = locks.SlaveLock("cpu")
3037
source = s(step.SVN, svnurl="http://example.org/svn/Trunk")
3038
f22 = factory.Trial(source, trialpython=["python2.2"])
3039
f23 = factory.Trial(source, trialpython=["python2.3"])
3040
f24 = factory.Trial(source, trialpython=["python2.4"])
3041
b1 = {'name': 'p22', 'slavename': 'bot-1, builddir='p22', 'factory': f22,
3042
'locks': [slow_lock] }
3043
b2 = {'name': 'p23', 'slavename': 'bot-1, builddir='p23', 'factory': f23,
3044
'locks': [slow_lock] }
3045
b3 = {'name': 'p24', 'slavename': 'bot-1, builddir='p24', 'factory': f24,
3046
'locks': [slow_lock] }
3047
c['builders'] = [b1, b2, b3]
3048
3049
In the last example, we use two Locks at the same time. In this
3050
case, we're concerned about both of the previous constraints, but
3051
we'll say that only the tests are computationally intensive, and that
3052
they have been split into those which use the database and those
3053
which do not. In addition, two of the Builds run on a fast machine
3054
which does not need to worry about the cpu lock, but which still must
3055
be prevented from simultaneous database access.
3056
3057
from buildbot import locks
3058
from buildbot.process import s, step, factory
3059
3060
db_lock = locks.MasterLock("database")
3061
cpu_lock = locks.SlaveLock("cpu")
3062
slow_steps = [s(step.SVN, svnurl="http://example.org/svn/Trunk"),
3063
s(step.ShellCommand, command="make all", locks=[cpu_lock]),
3064
s(step.ShellCommand, command="make test", locks=[cpu_lock]),
3065
s(step.ShellCommand, command="make db-test",
3066
locks=[db_lock, cpu_lock]),
3067
]
3068
slow_f = factory.BuildFactory(slow_steps)
3069
fast_steps = [s(step.SVN, svnurl="http://example.org/svn/Trunk"),
3070
s(step.ShellCommand, command="make all", locks=[]),
3071
s(step.ShellCommand, command="make test", locks=[]),
3072
s(step.ShellCommand, command="make db-test",
3073
locks=[db_lock]),
3074
]
3075
fast_factory = factory.BuildFactory(fast_steps)
3076
b1 = {'name': 'full1', 'slavename': 'bot-slow, builddir='full1',
3077
'factory': slow_factory}
3078
b2 = {'name': 'full2', 'slavename': 'bot-slow, builddir='full2',
3079
'factory': slow_factory}
3080
b3 = {'name': 'full3', 'slavename': 'bot-fast, builddir='full3',
3081
'factory': fast_factory}
3082
b4 = {'name': 'full4', 'slavename': 'bot-fast, builddir='full4',
3083
'factory': fast_factory}
3084
c['builders'] = [b1, b2, b3, b4]
3085
3086
As a final note, remember that a unit test system which breaks when
3087
multiple people run it at the same time is fragile and should be
3088
fixed. Asking your human developers to serialize themselves when
3089
running unit tests will just discourage them from running the unit
3090
tests at all. Find a way to fix this: change the database tests to
3091
create a new (uniquely-named) user or table for each test run, don't
3092
use fixed listening TCP ports for network tests (instead listen on
3093
port 0 to let the kernel choose a port for you and then query the
3094
socket to find out what port was allocated). `MasterLock's can be
3095
used to accomodate broken test systems like this, but are really
3096
intended for other purposes: build processes that store or retrieve
3097
products in shared directories, or which do things that human
3098
developers would not (or which might slow down or break in ways that
3099
require human attention to deal with).
3100
3101
---------- Footnotes ----------
3102
3103
(1) Also note that a clever buildmaster admin could still create
3104
the opportunity for deadlock: Build A obtains Lock 1, inside which
3105
Step A.two tries to acquire Lock 2 at the Step level. Meanwhile Build
3106
B obtains Lock 2, and has a Step B.two which wants to acquire Lock 1
3107
at the Step level. Don't Do That.
3108
3109
3110
File: buildbot.info, Node: Build Factories, Prev: Interlocks, Up: Build Process
3111
3112
6.3 Build Factories
3113
===================
3114
3115
Each Builder is equipped with a "build factory", which is responsible
3116
for producing the actual `Build' objects that perform each build.
3117
This factory is created in the configuration file, and attached to a
3118
Builder through the `factory' element of its dictionary.
3119
3120
The standard `BuildFactory' object creates `Build' objects by
3121
default. These Builds will each execute a collection of BuildSteps in
3122
a fixed sequence. Each step can affect the results of the build, but
3123
in general there is little intelligence to tie the different steps
3124
together. You can create subclasses of `Build' to implement more
3125
sophisticated build processes, and then use a subclass of
3126
`BuildFactory' (or simply set the `buildClass' attribute) to create
3127
instances of your new Build subclass.
3128
3129
* Menu:
3130
3131
* BuildStep Objects::
3132
* BuildFactory::
3133
* Process-Specific build factories::
3134
3135
3136
File: buildbot.info, Node: BuildStep Objects, Next: BuildFactory, Prev: Build Factories, Up: Build Factories
3137
3138
6.3.1 BuildStep Objects
3139
-----------------------
3140
3141
The steps used by these builds are all subclasses of `BuildStep'.
3142
The standard ones provided with Buildbot are documented later, *Note
3143
Build Steps::. You can also write your own subclasses to use in
3144
builds.
3145
3146
The basic behavior for a `BuildStep' is to:
3147
3148
* run for a while, then stop
3149
3150
* possibly invoke some RemoteCommands on the attached build slave
3151
3152
* possibly produce a set of log files
3153
3154
* finish with a status described by one of four values defined in
3155
buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED
3156
3157
* provide a list of short strings to describe the step
3158
3159
* define a color (generally green, orange, or red) with which the
3160
step should be displayed
3161
3162
More sophisticated steps may produce additional information and
3163
provide it to later build steps, or store it in the factory to provide
3164
to later builds.
3165
3166
3167
File: buildbot.info, Node: BuildFactory, Next: Process-Specific build factories, Prev: BuildStep Objects, Up: Build Factories
3168
3169
6.3.2 BuildFactory
3170
------------------
3171
3172
The default `BuildFactory', provided in the
3173
`buildbot.process.factory' module, is constructed with a list of
3174
"BuildStep specifications": a list of `(step_class, kwargs)' tuples
3175
for each. When asked to create a Build, it loads the list of steps
3176
into the new Build object. When the Build is actually started, these
3177
step specifications are used to create the actual set of BuildSteps,
3178
which are then executed one at a time. For example, a build which
3179
consists of a CVS checkout followed by a `make build' would be
3180
constructed as follows:
3181
3182
from buildbot.process import step, factory
3183
from buildbot.factory import s
3184
# s is a convenience function, defined with:
3185
# def s(steptype, **kwargs): return (steptype, kwargs)
3186
3187
f = factory.BuildFactory([s(step.CVS,
3188
cvsroot=CVSROOT, cvsmodule="project",
3189
mode="update"),
3190
s(step.Compile, command=["make", "build"])])
3191
3192
Each step can affect the build process in the following ways:
3193
3194
* If the step's `haltOnFailure' attribute is True, then a failure
3195
in the step (i.e. if it completes with a result of FAILURE) will
3196
cause the whole build to be terminated immediately: no further
3197
steps will be executed. This is useful for setup steps upon
3198
which the rest of the build depends: if the CVS checkout or
3199
`./configure' process fails, there is no point in trying to
3200
compile or test the resulting tree.
3201
3202
* If the `flunkOnFailure' or `flunkOnWarnings' flag is set, then a
3203
result of FAILURE or WARNINGS will mark the build as a whole as
3204
FAILED. However, the remaining steps will still be executed.
3205
This is appropriate for things like multiple testing steps: a
3206
failure in any one of them will indicate that the build has
3207
failed, however it is still useful to run them all to completion.
3208
3209
* Similarly, if the `warnOnFailure' or `warnOnWarnings' flag is
3210
set, then a result of FAILURE or WARNINGS will mark the build as
3211
having WARNINGS, and the remaining steps will still be executed.
3212
This may be appropriate for certain kinds of optional build or
3213
test steps. For example, a failure experienced while building
3214
documentation files should be made visible with a WARNINGS
3215
result but not be serious enough to warrant marking the whole
3216
build with a FAILURE.
3217
3218
3219
In addition, each Step produces its own results, may create
3220
logfiles, etc. However only the flags described above have any effect
3221
on the build as a whole.
3222
3223
The pre-defined BuildSteps like `CVS' and `Compile' have
3224
reasonably appropriate flags set on them already. For example, without
3225
a source tree there is no point in continuing the build, so the `CVS'
3226
class has the `haltOnFailure' flag set to True. Look in
3227
`buildbot/process/step.py' to see how the other Steps are marked.
3228
3229
Each Step is created with an additional `workdir' argument that
3230
indicates where its actions should take place. This is specified as a
3231
subdirectory of the slave builder's base directory, with a default
3232
value of `build'. This is only implemented as a step argument (as
3233
opposed to simply being a part of the base directory) because the
3234
CVS/SVN steps need to perform their checkouts from the parent
3235
directory.
3236
3237
* Menu:
3238
3239
* BuildFactory Attributes::
3240
* Quick builds::
3241
3242
3243
File: buildbot.info, Node: BuildFactory Attributes, Next: Quick builds, Prev: BuildFactory, Up: BuildFactory
3244
3245
6.3.2.1 BuildFactory Attributes
3246
...............................
3247
3248
Some attributes from the BuildFactory are copied into each Build.
3249
3250
`useProgress'
3251
(defaults to True): if True, the buildmaster keeps track of how
3252
long each step takes, so it can provide estimates of how long
3253
future builds will take. If builds are not expected to take a
3254
consistent amount of time (such as incremental builds in which a
3255
random set of files are recompiled or tested each time), this
3256
should be set to False to inhibit progress-tracking.
3257
3258
3259
3260
File: buildbot.info, Node: Quick builds, Prev: BuildFactory Attributes, Up: BuildFactory
3261
3262
6.3.2.2 Quick builds
3263
....................
3264
3265
The difference between a "full build" and a "quick build" is that
3266
quick builds are generally done incrementally, starting with the tree
3267
where the previous build was performed. That simply means that the
3268
source-checkout step should be given a `mode='update'' flag, to do
3269
the source update in-place.
3270
3271
In addition to that, the `useProgress' flag should be set to
3272
False. Incremental builds will (or at least the ought to) compile as
3273
few files as necessary, so they will take an unpredictable amount of
3274
time to run. Therefore it would be misleading to claim to predict how
3275
long the build will take.
3276
3277
3278
File: buildbot.info, Node: Process-Specific build factories, Prev: BuildFactory, Up: Build Factories
3279
3280
6.3.3 Process-Specific build factories
3281
--------------------------------------
3282
3283
Many projects use one of a few popular build frameworks to simplify
3284
the creation and maintenance of Makefiles or other compilation
3285
structures. Buildbot provides several pre-configured BuildFactory
3286
subclasses which let you build these projects with a minimum of fuss.
3287
3288
* Menu:
3289
3290
* GNUAutoconf::
3291
* CPAN::
3292
* Python distutils::
3293
* Python/Twisted/trial projects::
3294
3295
3296
File: buildbot.info, Node: GNUAutoconf, Next: CPAN, Prev: Process-Specific build factories, Up: Process-Specific build factories
3297
3298
6.3.3.1 GNUAutoconf
3299
...................
3300
3301
GNU Autoconf (http://www.gnu.org/software/autoconf/) is a software
3302
portability tool, intended to make it possible to write programs in C
3303
(and other languages) which will run on a variety of UNIX-like
3304
systems. Most GNU software is built using autoconf. It is frequently
3305
used in combination with GNU automake. These tools both encourage a
3306
build process which usually looks like this:
3307
3308
% CONFIG_ENV=foo ./configure --with-flags
3309
% make all
3310
% make check
3311
# make install
3312
3313
(except of course the Buildbot always skips the `make install'
3314
part).
3315
3316
The Buildbot's `buildbot.process.factory.GNUAutoconf' factory is
3317
designed to build projects which use GNU autoconf and/or automake. The
3318
configuration environment variables, the configure flags, and command
3319
lines used for the compile and test are all configurable, in general
3320
the default values will be suitable.
3321
3322
Example:
3323
3324
# use the s() convenience function defined earlier
3325
f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"),
3326
flags=["--disable-nls"])
3327
3328
Required Arguments:
3329
3330
`source'
3331
This argument must be a step specification tuple that provides a
3332
BuildStep to generate the source tree.
3333
3334
Optional Arguments:
3335
3336
`configure'
3337
The command used to configure the tree. Defaults to
3338
`./configure'. Accepts either a string or a list of shell argv
3339
elements.
3340
3341
`configureEnv'
3342
The environment used for the initial configuration step. This
3343
accepts a dictionary which will be merged into the buildslave's
3344
normal environment. This is commonly used to provide things like
3345
`CFLAGS="-O2 -g"' (to turn off debug symbols during the compile).
3346
Defaults to an empty dictionary.
3347
3348
`configureFlags'
3349
A list of flags to be appended to the argument list of the
3350
configure command. This is commonly used to enable or disable
3351
specific features of the autoconf-controlled package, like
3352
`["--without-x"]' to disable windowing support. Defaults to an
3353
empty list.
3354
3355
`compile'
3356
this is a shell command or list of argv values which is used to
3357
actually compile the tree. It defaults to `make all'. If set to
3358
None, the compile step is skipped.
3359
3360
`test'
3361
this is a shell command or list of argv values which is used to
3362
run the tree's self-tests. It defaults to `make check'. If set to
3363
None, the test step is skipped.
3364
3365
3366
3367
File: buildbot.info, Node: CPAN, Next: Python distutils, Prev: GNUAutoconf, Up: Process-Specific build factories
3368
3369
6.3.3.2 CPAN
3370
............
3371
3372
Most Perl modules available from the CPAN (http://www.cpan.org/)
3373
archive use the `MakeMaker' module to provide configuration, build,
3374
and test services. The standard build routine for these modules looks
3375
like:
3376
3377
% perl Makefile.PL
3378
% make
3379
% make test
3380
# make install
3381
3382
(except again Buildbot skips the install step)
3383
3384
Buildbot provides a `CPAN' factory to compile and test these
3385
projects.
3386
3387
Arguments:
3388
`source'
3389
(required): A step specification tuple, that that used by
3390
GNUAutoconf.
3391
3392
`perl'
3393
A string which specifies the `perl' executable to use. Defaults
3394
to just `perl'.
3395
3396
3397
3398
File: buildbot.info, Node: Python distutils, Next: Python/Twisted/trial projects, Prev: CPAN, Up: Process-Specific build factories
3399
3400
6.3.3.3 Python distutils
3401
........................
3402
3403
Most Python modules use the `distutils' package to provide
3404
configuration and build services. The standard build process looks
3405
like:
3406
3407
% python ./setup.py build
3408
% python ./setup.py install
3409
3410
Unfortunately, although Python provides a standard unit-test
3411
framework named `unittest', to the best of my knowledge `distutils'
3412
does not provide a standardized target to run such unit tests. (please
3413
let me know if I'm wrong, and I will update this factory).
3414
3415
The `Distutils' factory provides support for running the build
3416
part of this process. It accepts the same `source=' parameter as the
3417
other build factories.
3418
3419
Arguments:
3420
`source'
3421
(required): A step specification tuple, that that used by
3422
GNUAutoconf.
3423
3424
`python'
3425
A string which specifies the `python' executable to use. Defaults
3426
to just `python'.
3427
3428
`test'
3429
Provides a shell command which runs unit tests. This accepts
3430
either a string or a list. The default value is None, which
3431
disables the test step (since there is no common default command
3432
to run unit tests in distutils modules).
3433
3434
3435
3436
File: buildbot.info, Node: Python/Twisted/trial projects, Prev: Python distutils, Up: Process-Specific build factories
3437
3438
6.3.3.4 Python/Twisted/trial projects
3439
.....................................
3440
3441
Twisted provides a unit test tool named `trial' which provides a few
3442
improvements over Python's built-in `unittest' module. Many python
3443
projects which use Twisted for their networking or application
3444
services also use trial for their unit tests. These modules are
3445
usually built and tested with something like the following:
3446
3447
% python ./setup.py build
3448
% PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test
3449
% python ./setup.py install
3450
3451
Unfortunately, the `build/lib' directory into which the
3452
built/copied .py files are placed is actually architecture-dependent,
3453
and I do not yet know of a simple way to calculate its value. For many
3454
projects it is sufficient to import their libraries "in place" from
3455
the tree's base directory (`PYTHONPATH=.').
3456
3457
In addition, the PROJECTNAME value where the test files are
3458
located is project-dependent: it is usually just the project's
3459
top-level library directory, as common practice suggests the unit test
3460
files are put in the `test' sub-module. This value cannot be guessed,
3461
the `Trial' class must be told where to find the test files.
3462
3463
The `Trial' class provides support for building and testing
3464
projects which use distutils and trial. If the test module name is
3465
specified, trial will be invoked. The library path used for testing
3466
can also be set.
3467
3468
One advantage of trial is that the Buildbot happens to know how to
3469
parse trial output, letting it identify which tests passed and which
3470
ones failed. The Buildbot can then provide fine-grained reports about
3471
how many tests have failed, when individual tests fail when they had
3472
been passing previously, etc.
3473
3474
Another feature of trial is that you can give it a series of source
3475
.py files, and it will search them for special `test-case-name' tags
3476
that indicate which test cases provide coverage for that file. Trial
3477
can then run just the appropriate tests. This is useful for quick
3478
builds, where you want to only run the test cases that cover the
3479
changed functionality.
3480
3481
Arguments:
3482
`source'
3483
(required): A step specification tuple, like that used by
3484
GNUAutoconf.
3485
3486
`buildpython'
3487
A list (argv array) of strings which specifies the `python'
3488
executable to use when building the package. Defaults to just
3489
`['python']'. It may be useful to add flags here, to supress
3490
warnings during compilation of extension modules. This list is
3491
extended with `['./setup.py', 'build']' and then executed in a
3492
ShellCommand.
3493
3494
`testpath'
3495
Provides a directory to add to `PYTHONPATH' when running the unit
3496
tests, if tests are being run. Defaults to `.' to include the
3497
project files in-place. The generated build library is frequently
3498
architecture-dependent, but may simply be `build/lib' for
3499
pure-python modules.
3500
3501
`trialpython'
3502
Another list of strings used to build the command that actually
3503
runs trial. This is prepended to the contents of the `trial'
3504
argument below. It may be useful to add `-W' flags here to
3505
supress warnings that occur while tests are being run. Defaults
3506
to an empty list, meaning `trial' will be run without an explicit
3507
interpreter, which is generally what you want if you're using
3508
`/usr/bin/trial' instead of, say, the `./bin/trial' that lives
3509
in the Twisted source tree.
3510
3511
`trial'
3512
provides the name of the `trial' command. It is occasionally
3513
useful to use an alternate executable, such as `trial2.2' which
3514
might run the tests under an older version of Python. Defaults to
3515
`trial'.
3516
3517
`tests'
3518
Provides a module name or names which contain the unit tests for
3519
this project. Accepts a string, typically `PROJECTNAME.test', or
3520
a list of strings. Defaults to None, indicating that no tests
3521
should be run. You must either set this or `useTestCaseNames' to
3522
do anyting useful with the Trial factory.
3523
3524
`useTestCaseNames'
3525
Tells the Step to provide the names of all changed .py files to
3526
trial, so it can look for test-case-name tags and run just the
3527
matching test cases. Suitable for use in quick builds. Defaults
3528
to False.
3529
3530
`randomly'
3531
If `True', tells Trial (with the `--random=0' argument) to run
3532
the test cases in random order, which sometimes catches subtle
3533
inter-test dependency bugs. Defaults to `False'.
3534
3535
`recurse'
3536
If `True', tells Trial (with the `--recurse' argument) to look
3537
in all subdirectories for additional test cases. It isn't clear
3538
to me how this works, but it may be useful to deal with the
3539
unknown-PROJECTNAME problem described above, and is currently
3540
used in the Twisted buildbot to accomodate the fact that test
3541
cases are now distributed through multiple
3542
twisted.SUBPROJECT.test directories.
3543
3544
3545
Unless one of `trialModule' or `useTestCaseNames' are set, no
3546
tests will be run.
3547
3548
Some quick examples follow. Most of these examples assume that the
3549
target python code (the "code under test") can be reached directly
3550
from the root of the target tree, rather than being in a `lib/'
3551
subdirectory.
3552
3553
# Trial(source, tests="toplevel.test") does:
3554
# python ./setup.py build
3555
# PYTHONPATH=. trial -to toplevel.test
3556
3557
# Trial(source, tests=["toplevel.test", "other.test"]) does:
3558
# python ./setup.py build
3559
# PYTHONPATH=. trial -to toplevel.test other.test
3560
3561
# Trial(source, useTestCaseNames=True) does:
3562
# python ./setup.py build
3563
# PYTHONPATH=. trial -to --testmodule=foo/bar.py.. (from Changes)
3564
3565
# Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"):
3566
# python2.3 -Wall ./setup.py build
3567
# PYTHONPATH=. trial -to foo.tests
3568
3569
# Trial(source, trialpython="python2.3", trial="/usr/bin/trial",
3570
# tests="foo.tests") does:
3571
# python2.3 -Wall ./setup.py build
3572
# PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests
3573
3574
# For running trial out of the tree being tested (only useful when the
3575
# tree being built is Twisted itself):
3576
# Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial",
3577
# tests="foo.tests") does:
3578
# python2.3 -Wall ./setup.py build
3579
# PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests
3580
3581
If the output directory of `./setup.py build' is known, you can
3582
pull the python code from the built location instead of the source
3583
directories. This should be able to handle variations in where the
3584
source comes from, as well as accomodating binary extension modules:
3585
3586
# Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3')
3587
# does:
3588
# python ./setup.py build
3589
# PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test
3590
3591
3592
File: buildbot.info, Node: Status Delivery, Next: Command-line tool, Prev: Build Process, Up: Top
3593
3594
7 Status Delivery
3595
*****************
3596
3597
More details are available in the docstrings for each class, use
3598
`pydoc buildbot.status.html.Waterfall' to see them. Most status
3599
delivery objects take a `categories=' argument, which can contain a
3600
list of "category" names: in this case, it will only show status for
3601
Builders that are in one of the named categories.
3602
3603
(implementor's note: each of these objects should be a
3604
service.MultiService which will be attached to the BuildMaster object
3605
when the configuration is processed. They should use
3606
`self.parent.getStatus()' to get access to the top-level IStatus
3607
object, either inside `startService' or later. They may call
3608
`status.subscribe()' in `startService' to receive notifications of
3609
builder events, in which case they must define `builderAdded' and
3610
related methods. See the docstrings in `buildbot/interfaces.py' for
3611
full details.)
3612
3613
* Menu:
3614
3615
* HTML Waterfall::
3616
* IRC Bot::
3617
* PBListener::
3618
3619
3620
File: buildbot.info, Node: HTML Waterfall, Next: IRC Bot, Prev: Status Delivery, Up: Status Delivery
3621
3622
7.0.1 HTML Waterfall
3623
--------------------
3624
3625
from buildbot.status import html
3626
w = html.Waterfall(http_port=8080)
3627
c['status'].append(w)
3628
3629
The `buildbot.status.html.Waterfall' status target creates an HTML
3630
"waterfall display", which shows a time-based chart of events. This
3631
display provides detailed information about all steps of all recent
3632
builds, and provides hyperlinks to look at individual build logs and
3633
source changes. If the `http_port' argument is provided, it provides
3634
a strports specification for the port that the web server should
3635
listen on. This can be a simple port number, or a string like
3636
`tcp:8080:interface=127.0.0.1' (to limit connections to the loopback
3637
interface, and therefore to clients running on the same host)(1).
3638
3639
If instead (or in addition) you provide the `distrib_port'
3640
argument, a twisted.web distributed server will be started either on a
3641
TCP port (if `distrib_port' is like `"tcp:12345"') or more likely on
3642
a UNIX socket (if `distrib_port' is like `"unix:/path/to/socket"').
3643
3644
The `distrib_port' option means that, on a host with a
3645
suitably-configured twisted-web server, you do not need to consume a
3646
separate TCP port for the buildmaster's status web page. When the web
3647
server is constructed with `mktap web --user', URLs that point to
3648
`http://host/~username/' are dispatched to a sub-server that is
3649
listening on a UNIX socket at `~username/.twisted-web-pb'. On such a
3650
system, it is convenient to create a dedicated `buildbot' user, then
3651
set `distrib_port' to
3652
`"unix:"+os.path.expanduser("~/.twistd-web-pb")'. This configuration
3653
will make the HTML status page available at `http://host/~buildbot/'
3654
. Suitable URL remapping can make it appear at
3655
`http://host/buildbot/', and the right virtual host setup can even
3656
place it at `http://buildbot.host/' .
3657
3658
In addition, the HTML page can have a favicon and custom CSS: see
3659
the docstring for details.
3660
3661
---------- Footnotes ----------
3662
3663
(1) It may even be possible to provide SSL access by using a
3664
specification like
3665
`"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"', but this is
3666
completely untested
3667
3668
3669
File: buildbot.info, Node: IRC Bot, Next: PBListener, Prev: HTML Waterfall, Up: Status Delivery
3670
3671
7.0.2 IRC Bot
3672
-------------
3673
3674
The `buildbot.status.words.IRC' status target creates an IRC bot
3675
which will attach to certain channels and be available for status
3676
queries. It can also be asked to announce builds as they occur, or be
3677
told to shut up.
3678
3679
from twisted.status import words
3680
irc = words.IRC("irc.example.org", "botnickname",
3681
channels=["channel1", "channel2"])
3682
c['status'].append(irc)
3683
3684
Take a look at the docstring for `words.IRC' for more details on
3685
configuring this service.
3686
3687
To use the service, you address messages at the buildbot, either
3688
normally (`botnickname: status') or with private messages (`/msg
3689
botnickname status'). The buildbot will respond in kind.
3690
3691
Some of the commands currently available:
3692
3693
`list builders'
3694
Emit a list of all configured builders
3695
3696
`status BUILDER'
3697
Announce the status of a specific Builder: what it is doing
3698
right now.
3699
3700
`status all'
3701
Announce the status of all Builders
3702
3703
`watch BUILDER'
3704
If the given Builder is currently running, wait until the Build
3705
is finished and then announce the results.
3706
3707
`last BUILDER'
3708
Return the results of the last build to run on the given Builder.
3709
3710
`help COMMAND'
3711
Describe a command. Use `help commands' to get a list of known
3712
commands.
3713
3714
`source'
3715
Announce the URL of the Buildbot's home page.
3716
3717
`version'
3718
Announce the version of this Buildbot.
3719
3720
If the `allowForce=True' option was used, some addtional commands
3721
will be available:
3722
3723
`force build BUILDER REASON'
3724
Tell the given Builder to start a build of the latest code. The
3725
user requesting the build and REASON are recorded in the Build
3726
status. The buildbot will announce the build's status when it
3727
finishes.
3728
3729
`stop build BUILDER REASON'
3730
Terminate any running build in the given Builder. REASON will be
3731
added to the build status to explain why it was stopped. You
3732
might use this if you committed a bug, corrected it right away,
3733
and don't want to wait for the first build (which is destined to
3734
fail) to complete before starting the second (hopefully fixed)
3735
build.
3736
3737
3738
File: buildbot.info, Node: PBListener, Prev: IRC Bot, Up: Status Delivery
3739
3740
7.0.3 PBListener
3741
----------------
3742
3743
`buildbot.status.client.PBListener(port=int, user=str, passwd=str)'
3744
3745
This sets up a PB listener on the given TCP port, to which a
3746
PB-based status client can connect and retrieve status information.
3747
`buildbot statusgui' is an example of such a status client. The
3748
`port' argument can also be a strports specification string.
3749
3750
3751
File: buildbot.info, Node: Command-line tool, Next: Resources, Prev: Status Delivery, Up: Top
3752
3753
8 Command-line tool
3754
*******************
3755
3756
The `buildbot' command-line tool can be used to start or stop a
3757
buildmaster or buildbot, and to interact with a running buildmaster.
3758
Some of its subcommands are intended for buildmaster admins, while
3759
some are for developers who are editing the code that the buildbot is
3760
monitoring.
3761
3762
* Menu:
3763
3764
* Administrator Tools::
3765
* Developer Tools::
3766
* Other Tools::
3767
* .buildbot config directory::
3768
3769
3770
File: buildbot.info, Node: Administrator Tools, Next: Developer Tools, Prev: Command-line tool, Up: Command-line tool
3771
3772
8.1 Administrator Tools
3773
=======================
3774
3775
The following `buildbot' sub-commands are intended for buildmaster
3776
administrators:
3777
3778
master
3779
======
3780
3781
This creates a new directory and populates it with files that allow it
3782
to be used as a buildmaster's base directory.
3783
3784
buildbot master BASEDIR
3785
3786
slave
3787
=====
3788
3789
This creates a new directory and populates it with files that let it
3790
be used as a buildslave's base directory. You must provide several
3791
arguments, which are used to create the initial `buildbot.tac' file.
3792
3793
buildbot slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD
3794
3795
start
3796
=====
3797
3798
This starts a buildmaster or buildslave which was already created in
3799
the given base directory. The daemon is launched in the background,
3800
with events logged to a file named `twistd.log'.
3801
3802
buildbot start BASEDIR
3803
3804
stop
3805
====
3806
3807
This terminates the daemon (either buildmaster or buildslave) running
3808
in the given directory.
3809
3810
buildbot stop BASEDIR
3811
3812
sighup
3813
======
3814
3815
This sends a SIGHUP to the buildmaster running in the given directory,
3816
which causes it to re-read its `master.cfg' file.
3817
3818
buildbot sighup BASEDIR
3819
3820
3821
File: buildbot.info, Node: Developer Tools, Next: Other Tools, Prev: Administrator Tools, Up: Command-line tool
3822
3823
8.2 Developer Tools
3824
===================
3825
3826
These tools are provided for use by the developers who are working on
3827
the code that the buildbot is monitoring.
3828
3829
* Menu:
3830
3831
* statuslog::
3832
* statusgui::
3833
* try::
3834
3835
3836
File: buildbot.info, Node: statuslog, Next: statusgui, Prev: Developer Tools, Up: Developer Tools
3837
3838
8.2.1 statuslog
3839
---------------
3840
3841
buildbot statuslog --master MASTERHOST:PORT
3842
3843
This command starts a simple text-based status client, one which
3844
just prints out a new line each time an event occurs on the
3845
buildmaster.
3846
3847
The `--master' option provides the location of the
3848
`client.PBListener' status port, used to deliver build information to
3849
realtime status clients. The option is always in the form of a
3850
string, with hostname and port number separated by a colon
3851
(`HOSTNAME:PORTNUM'). Note that this port is _not_ the same as the
3852
slaveport (although a future version may allow the same port number
3853
to be used for both purposes).
3854
3855
The `--master' option can also be provided by the `masterstatus'
3856
name in `.buildbot/options' (*note .buildbot config directory::).
3857
3858
3859
File: buildbot.info, Node: statusgui, Next: try, Prev: statuslog, Up: Developer Tools
3860
3861
8.2.2 statusgui
3862
---------------
3863
3864
buildbot statusgui --master MASTERHOST:PORT
3865
3866
This command starts a simple Gtk+-based status client, which
3867
contains a few boxes for each Builder that change color as events
3868
occur. It uses the same `--master' argument as the `buildbot
3869
statuslog' command (*note statuslog::).
3870
3871
3872
File: buildbot.info, Node: try, Prev: statusgui, Up: Developer Tools
3873
3874
8.2.3 try
3875
---------
3876
3877
This lets a developer to ask the question "What would happen if I
3878
committed this patch right now?". It runs the unit test suite (across
3879
multiple build platforms) on the developer's current code, allowing
3880
them to make sure they will not break the tree when they finally
3881
commit their changes.
3882
3883
The `buildbot try' command is meant to be run from within a
3884
developer's local tree, and starts by figuring out the base revision
3885
of that tree (what revision was current the last time the tree was
3886
updated), and a patch that can be applied to that revision of the tree
3887
to make it match the developer's copy. This (revision, patch) pair is
3888
then sent to the buildmaster, which runs a build with that
3889
SourceStamp. If you want, the tool will emit status messages as the
3890
builds run, and will not terminate until the first failure has been
3891
detected (or the last success).
3892
3893
For this command to work, several pieces must be in place:
3894
3895
TryScheduler
3896
============
3897
3898
The buildmaster must have a `scheduler.Try' instance in the config
3899
file's `c['schedulers']' list. This lets the administrator control
3900
who may initiate these "trial" builds, which branches are eligible
3901
for trial builds, and which Builders should be used for them.
3902
3903
The `TryScheduler' has various means to accept build requests: all
3904
of them enforce more security than the usual buildmaster ports do.
3905
Any source code being built can be used to compromise the buildslave
3906
accounts, but in general that code must be checked out from the VC
3907
repository first, so only people with commit privileges can get
3908
control of the buildslaves. The usual force-build control channels can
3909
waste buildslave time but do not allow arbitrary commands to be
3910
executed by people who don't have those commit privileges. However,
3911
the source code patch that is provided with the trial build does not
3912
have to go through the VC system first, so it is important to make
3913
sure these builds cannot be abused by a non-committer to acquire as
3914
much control over the buildslaves as a committer has. Ideally, only
3915
developers who have commit access to the VC repository would be able
3916
to start trial builds, but unfortunately the buildmaster does not, in
3917
general, have access to VC system's user list.
3918
3919
As a result, the `TryScheduler' requires a bit more configuration.
3920
There are currently two ways to set this up:
3921
3922
*jobdir (ssh)*
3923
This approach creates a command queue directory, called the
3924
"jobdir", in the buildmaster's working directory. The buildmaster
3925
admin sets the ownership and permissions of this directory to
3926
only grant write access to the desired set of developers, all of
3927
whom must have accounts on the machine. The `buildbot try'
3928
command creates a special file containing the source stamp
3929
information and drops it in the jobdir, just like a standard
3930
maildir. When the buildmaster notices the new file, it unpacks
3931
the information inside and starts the builds.
3932
3933
The config file entries used by 'buildbot try' either specify a
3934
local queuedir (for which write and mv are used) or a remote one
3935
(using scp and ssh).
3936
3937
The advantage of this scheme is that it is quite secure, the
3938
disadvantage is that it requires fiddling outside the buildmaster
3939
config (to set the permissions on the jobdir correctly). If the
3940
buildmaster machine happens to also house the VC repository,
3941
then it can be fairly easy to keep the VC userlist in sync with
3942
the trial-build userlist. If they are on different machines,
3943
this will be much more of a hassle. It may also involve granting
3944
developer accounts on a machine that would not otherwise require
3945
them.
3946
3947
To implement this, the buildslave invokes 'ssh -l username host
3948
buildbot tryserver ARGS', passing the patch contents over stdin.
3949
The arguments must include the inlet directory and the revision
3950
information.
3951
3952
*user+password (PB)*
3953
In this approach, each developer gets a username/password pair,
3954
which are all listed in the buildmaster's configuration file.
3955
When the developer runs `buildbot try', their machine connects
3956
to the buildmaster via PB and authenticates themselves using
3957
that username and password, then sends a PB command to start the
3958
trial build.
3959
3960
The advantage of this scheme is that the entire configuration is
3961
performed inside the buildmaster's config file. The
3962
disadvantages are that it is less secure (while the "cred"
3963
authentication system does not expose the password in plaintext
3964
over the wire, it does not offer most of the other security
3965
properties that SSH does). In addition, the buildmaster admin is
3966
responsible for maintaining the username/password list, adding
3967
and deleting entries as developers come and go.
3968
3969
3970
For example, to set up the "jobdir" style of trial build, using a
3971
command queue directory of `MASTERDIR/jobdir' (and assuming that all
3972
your project developers were members of the `developers' unix group),
3973
you would first create that directory (with `mkdir MASTERDIR/jobdir
3974
MASTERDIR/jobdir/new MASTERDIR/jobdir/cur MASTERDIR/jobdir/tmp; chgrp
3975
developers MASTERDIR/jobdir MASTERDIR/jobdir/*; chmod g+rwx,o-rwx
3976
MASTERDIR/jobdir MASTERDIR/jobdir/*'), and then use the following
3977
scheduler in the buildmaster's config file:
3978
3979
from buildbot.scheduler import Try_Jobdir
3980
s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"],
3981
jobdir="jobdir")
3982
c['schedulers'] = [s]
3983
3984
Note that you must create the jobdir before telling the
3985
buildmaster to use this configuration, otherwise you will get an
3986
error. Also remember that the buildmaster must be able to read and
3987
write to the jobdir as well. Be sure to watch the `twistd.log' file
3988
(*note Logfiles::) as you start using the jobdir, to make sure the
3989
buildmaster is happy with it.
3990
3991
To use the username/password form of authentication, create a
3992
`Try_Userpass' instance instead. It takes the same `builderNames'
3993
argument as the `Try_Jobdir' form, but accepts an addtional `port'
3994
argument (to specify the TCP port to listen on) and a `userpass' list
3995
of username/password pairs to accept. Remember to use good passwords
3996
for this: the security of the buildslave accounts depends upon it:
3997
3998
from buildbot.scheduler import Try_Userpass
3999
s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"],
4000
port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] )
4001
c['schedulers'] = [s]
4002
4003
Like most places in the buildbot, the `port' argument takes a
4004
strports specification. See `twisted.application.strports' for
4005
details.
4006
4007
locating the master
4008
===================
4009
4010
The `try' command needs to be told how to connect to the
4011
`TryScheduler', and must know which of the authentication approaches
4012
described above is in use by the buildmaster. You specify the
4013
approach by using `--connect=ssh' or `--connect=pb' (or `try_connect
4014
= 'ssh'' or `try_connect = 'pb'' in `.buildbot/options').
4015
4016
For the PB approach, the command must be given a `--master'
4017
argument (in the form HOST:PORT) that points to TCP port that you
4018
picked in the `Try_Userpass' scheduler. It also takes a `--username'
4019
and `--passwd' pair of arguments that match one of the entries in the
4020
buildmaster's `userpass' list. These arguments can also be provided
4021
as `try_master', `try_username', and `try_password' entries in the
4022
`.buildbot/options' file.
4023
4024
For the SSH approach, the command must be given `--tryhost',
4025
`--username', and optionally `--password' (TODO: really?) to get to
4026
the buildmaster host. It must also be given `--trydir', which points
4027
to the inlet directory configured above. The trydir can be relative
4028
to the user's home directory, but most of the time you will use an
4029
explicit path like `~buildbot/project/trydir'. These arguments can be
4030
provided in `.buildbot/options' as `try_host', `try_username',
4031
`try_password', and `try_dir'.
4032
4033
In addition, the SSH approach needs to connect to a PBListener
4034
status port, so it can retrieve and report the results of the build
4035
(the PB approach uses the existing connection to retrieve status
4036
information, so this step is not necessary). This requires a
4037
`--master' argument, or a `masterstatus' entry in `.buildbot/options',
4038
in the form of a HOSTNAME:PORT string.
4039
4040
choosing the Builders
4041
=====================
4042
4043
A trial build is performed on multiple Builders at the same time, and
4044
the developer gets to choose which Builders are used (limited to a set
4045
selected by the buildmaster admin with the TryScheduler's
4046
`builderNames=' argument). The set you choose will depend upon what
4047
your goals are: if you are concerned about cross-platform
4048
compatibility, you should use multiple Builders, one from each
4049
platform of interest. You might use just one builder if that platform
4050
has libraries or other facilities that allow better test coverage than
4051
what you can accomplish on your own machine, or faster test runs.
4052
4053
The set of Builders to use can be specified with multiple
4054
`--builder' arguments on the command line. It can also be specified
4055
with a single `try_builders' option in `.buildbot/options' that uses
4056
a list of strings to specify all the Builder names:
4057
4058
try_builders = ["full-OSX", "full-win32", "full-linux"]
4059
4060
specifying the VC system
4061
========================
4062
4063
The `try' command also needs to know how to take the developer's
4064
current tree and extract the (revision, patch) source-stamp pair.
4065
Each VC system uses a different process, so you start by telling the
4066
`try' command which VC system you are using, with an argument like
4067
`--vc=cvs' or `--vc=tla'. This can also be provided as `try_vc' in
4068
`.buildbot/options'.
4069
4070
finding the top of the tree
4071
===========================
4072
4073
Some VC systems (notably CVS and SVN) track each directory
4074
more-or-less independently, which means the `try' command needs to
4075
move up to the top of the project tree before it will be able to
4076
construct a proper full-tree patch. To accomplish this, the `try'
4077
command will crawl up through the parent directories until it finds a
4078
marker file. The default name for this marker file is
4079
`.buildbot-top', so when you are using CVS or SVN you should `touch
4080
.buildbot-top' from the top of your tree before running `buildbot
4081
try'. Alternatively, you can use a filename like `ChangeLog' or
4082
`README', since many projects put one of these files in their
4083
top-most directory (and nowhere else). To set this filename, use
4084
`--try-topfile=ChangeLog', or set it in the options file with
4085
`try_topfile = 'ChangeLog''.
4086
4087
You can also manually set the top of the tree with
4088
`--try-topdir=~/trees/mytree', or `try_topdir = '~/trees/mytree''. If
4089
you use `try_topdir', in a `.buildbot/options' file, you will need a
4090
separate options file for each tree you use, so it may be more
4091
convenient to use the `try_topfile' approach instead.
4092
4093
If the `try' command cannot find the top directory, it will abort
4094
with an error message. Other VC systems which work on full projects
4095
instead of individual directories (tla, baz, darcs, monotone) do not
4096
require `try' to know the top directory, so the `--try-topfile' and
4097
`--try-topdir' arguments will be ignored.
4098
4099
determining the branch name
4100
===========================
4101
4102
Some VC systems record the branch information in a way that "try" can
4103
locate it, in particular Arch (both `tla' and `baz'). For the others,
4104
if you are using something other than the default branch, you will
4105
have to tell the buildbot which branch your tree is using. You can do
4106
this with either the `--branch' argument, or a `try_branch' entry in
4107
the `.buildbot/options' file.
4108
4109
determining the revision and patch
4110
==================================
4111
4112
Each VC system has a separate approach for determining the tree's base
4113
revision and computing a patch.
4114
4115
`CVS'
4116
Wow, good question. We have to assume that you've done an `cvs
4117
update' on the whole tree... [TODO]
4118
4119
`SVN'
4120
`try' does a `svn status -u' to find the latest repository
4121
revision number (emitted on the last line in the "Status against
4122
revision: NN" message). It then performs an `svn diff -rNN' to
4123
find out how your tree differs from the repository version, and
4124
sends the resulting patch to the buildmaster. If your tree is not
4125
up to date, this will result in the "try" tree being created with
4126
the latest revision, then _backwards_ patches applied to bring it
4127
"back" to the version you actually checked out (plus your actual
4128
code changes), but this will still result in the correct tree
4129
being used for the build.
4130
4131
`baz'
4132
`try' does a `baz tree-id' to determine the fully-qualified
4133
version and patch identifier for the tree
4134
(ARCHIVE/VERSION-patch-NN), and uses the VERSION-patch-NN
4135
component as the base revision. It then does a `baz diff' to
4136
obtain the patch.
4137
4138
`tla'
4139
`try' does a `tla tree-version' to get the fully-qualified
4140
version identifier (ARCHIVE/VERSION), then takes the first line
4141
of `tla logs --reverse' to figure out the base revision. Then it
4142
does `tla changes --diffs' to obtain the patch.
4143
4144
`darcs'
4145
`darcs changes --context' emits a text file that contains a list
4146
of all patches back to and including the last tag was made. This
4147
text file (plus the location of a repository that contains all
4148
these patches) is sufficient to re-create the tree. Therefore
4149
the contents of this "context" file _are_ the revision stamp for
4150
a Darcs-controlled source tree.
4151
4152
So `try' does a `darcs changes --context' to determine what your
4153
tree's base revision is, and then does a `darcs diff -u' to
4154
compute the patch relative to that revision.
4155
4156
4157
waiting for results
4158
===================
4159
4160
If you provide the `--wait' option (or `try_wait = True' in
4161
`.buildbot/options'), the `buildbot try' command will wait until your
4162
changes have either been proven good or bad before exiting. Unless
4163
you use the `--quiet' option (or `try_quiet=True'), it will emit a
4164
progress message every 60 seconds until the builds have completed.
4165
4166
4167
File: buildbot.info, Node: Other Tools, Next: .buildbot config directory, Prev: Developer Tools, Up: Command-line tool
4168
4169
8.3 Other Tools
4170
===============
4171
4172
These tools are generally used by buildmaster administrators.
4173
4174
* Menu:
4175
4176
* sendchange::
4177
* debugclient::
4178
4179
4180
File: buildbot.info, Node: sendchange, Next: debugclient, Prev: Other Tools, Up: Other Tools
4181
4182
8.3.1 sendchange
4183
----------------
4184
4185
This command is used to tell the buildmaster about source changes. It
4186
is intended to be used from within a commit script, installed on the
4187
VC server.
4188
4189
buildbot sendchange --master MASTERHOST:PORT --username USER FILENAMES..
4190
4191
There are other (optional) arguments which can influence the
4192
`Change' that gets submitted:
4193
4194
`--branch'
4195
This provides the (string) branch specifier. If omitted, it
4196
defaults to None, indicating the "default branch". All files
4197
included in this Change must be on the same branch.
4198
4199
`--revision_number'
4200
This provides a (numeric) revision number for the change, used
4201
for VC systems that use numeric transaction numbers (like
4202
Subversion).
4203
4204
`--revision'
4205
This provides a (string) revision specifier, for VC systems that
4206
use strings (Arch would use something like patch-42 etc).
4207
4208
`--revision_file'
4209
This provides a filename which will be opened and the contents
4210
used as the revision specifier. This is specifically for Darcs,
4211
which uses the output of `darcs changes --context' as a revision
4212
specifier. This context file can be a couple of kilobytes long,
4213
spanning a couple lines per patch, and would be a hassle to pass
4214
as a command-line argument.
4215
4216
`--comments'
4217
This provides the change comments as a single argument. You may
4218
want to use `--logfile' instead.
4219
4220
`--logfile'
4221
This instructs the tool to read the change comments from the
4222
given file. If you use `-' as the filename, the tool will read
4223
the change comments from stdin.
4224
4225
4226
File: buildbot.info, Node: debugclient, Prev: sendchange, Up: Other Tools
4227
4228
8.3.2 debugclient
4229
-----------------
4230
4231
buildbot debugclient --master MASTERHOST:PORT --passwd DEBUGPW
4232
4233
This launches a small Gtk+/Glade-based debug tool, connecting to
4234
the buildmaster's "debug port". This debug port shares the same port
4235
number as the slaveport (*note Setting the slaveport::), but the
4236
`debugPort' is only enabled if you set a debug password in the
4237
buildmaster's config file (*note Debug options::). The `--passwd'
4238
option must match the `c['debugPassword']' value.
4239
4240
`--master' can also be provided in `.debug/options' by the
4241
`master' key. `--passwd' can be provided by the `debugPassword' key.
4242
4243
The `Connect' button must be pressed before any of the other
4244
buttons will be active. This establishes the connection to the
4245
buildmaster. The other sections of the tool are as follows:
4246
4247
`Reload .cfg'
4248
Forces the buildmaster to reload its `master.cfg' file. This is
4249
equivalent to sending a SIGHUP to the buildmaster, but can be
4250
done remotely through the debug port. Note that it is a good
4251
idea to be watching the buildmaster's `twistd.log' as you reload
4252
the config file, as any errors which are detected in the config
4253
file will be announced there.
4254
4255
`Rebuild .py'
4256
(not yet implemented). The idea here is to use Twisted's
4257
"rebuild" facilities to replace the buildmaster's running code
4258
with a new version. Even if this worked, it would only be used
4259
by buildbot developers.
4260
4261
`poke IRC'
4262
This locates a `words.IRC' status target and causes it to emit a
4263
message on all the channels to which it is currently connected.
4264
This was used to debug a problem in which the buildmaster lost
4265
the connection to the IRC server and did not attempt to
4266
reconnect.
4267
4268
`Commit'
4269
This allows you to inject a Change, just as if a real one had
4270
been delivered by whatever VC hook you are using. You can set
4271
the name of the committed file and the name of the user who is
4272
doing the commit. Optionally, you can also set a revision for
4273
the change. If the revision you provide looks like a number, it
4274
will be sent as an integer, otherwise it will be sent as a
4275
string.
4276
4277
`Force Build'
4278
This lets you force a Builder (selected by name) to start a
4279
build of the current source tree.
4280
4281
`Currently'
4282
(obsolete). This was used to manually set the status of the given
4283
Builder, but the status-assignment code was changed in an
4284
incompatible way and these buttons are no longer meaningful.
4285
4286
4287
4288
File: buildbot.info, Node: .buildbot config directory, Prev: Other Tools, Up: Command-line tool
4289
4290
8.4 .buildbot config directory
4291
==============================
4292
4293
Many of the `buildbot' tools must be told how to contact the
4294
buildmaster that they interact with. This specification can be
4295
provided as a command-line argument, but most of the time it will be
4296
easier to set them in an "options" file. The `buildbot' command will
4297
look for a special directory named `.buildbot', starting from the
4298
current directory (where the command was run) and crawling upwards,
4299
eventually looking in the user's home directory. It will look for a
4300
file named `options' in this directory, and will evaluate it as a
4301
python script, looking for certain names to be set. You can just put
4302
simple `name = 'value'' pairs in this file to set the options.
4303
4304
For a description of the names used in this file, please see the
4305
documentation for the individual `buildbot' sub-commands. The
4306
following is a brief sample of what this file's contents could be.
4307
4308
# for status-reading tools
4309
masterstatus = 'buildbot.example.org:12345'
4310
# for 'sendchange' or the debug port
4311
master = 'buildbot.example.org:18990'
4312
debugPassword = 'eiv7Po'
4313
4314
`masterstatus'
4315
Location of the `client.PBListener' status port, used by
4316
`statuslog' and `statusgui'.
4317
4318
`master'
4319
Location of the `debugPort' (for `debugclient'). Also the
4320
location of the `pb.PBChangeSource' (for `sendchange'). Usually
4321
shares the slaveport, but a future version may make it possible
4322
to have these listen on a separate port number.
4323
4324
`debugPassword'
4325
Must match the value of `c['debugPassword']', used to protect the
4326
debug port, for the `debugclient' command.
4327
4328
`username'
4329
Provides a default username for the `sendchange' command.
4330
4331
4332
4333
File: buildbot.info, Node: Resources, Next: Developer's Appendix, Prev: Command-line tool, Up: Top
4334
4335
9 Resources
4336
***********
4337
4338
The Buildbot's home page is at `http://buildbot.sourceforge.net/'
4339
4340
For configuration questions and general discussion, please use the
4341
`buildbot-devel' mailing list. The subscription instructions and
4342
archives are available at
4343
`http://lists.sourceforge.net/lists/listinfo/buildbot-devel'
4344
4345
4346
File: buildbot.info, Node: Developer's Appendix, Next: Index, Prev: Resources, Up: Top
4347
4348
Developer's Appendix
4349
********************
4350
4351
This appendix contains random notes about the implementation of the
4352
Buildbot, and is likely to only be of use to people intending to
4353
extend the Buildbot's internals.
4354
4355
The buildmaster consists of a tree of Service objects, which is
4356
shaped as follows:
4357
4358
BuildMaster
4359
ChangeMaster (in .change_svc)
4360
[IChangeSource instances]
4361
[IScheduler instances] (in .schedulers)
4362
BotMaster (in .botmaster)
4363
[IStatusTarget instances] (in .statusTargets)
4364
4365
The BotMaster has a collection of Builder objects as values of its
4366
`.builders' dictionary.
4367
4368
4369
File: buildbot.info, Node: Index, Prev: Developer's Appendix, Up: Top
4370
4371
Index
4372
*****
4373
4374
[index ]
4375
* Menu:
4376
4377
* Arch Checkout: Arch. (line 6)
4378
* Bazaar Checkout: Bazaar. (line 6)
4379
* Builder: Builder. (line 6)
4380
* BuildRequest: BuildRequest. (line 6)
4381
* BuildSet: BuildSet. (line 6)
4382
* c['bots']: Buildslave Specifiers.
4383
(line 6)
4384
* c['buildbotURL']: Defining the Project.
4385
(line 24)
4386
* c['builders']: Defining Builders. (line 6)
4387
* c['debugPassword']: Debug options. (line 6)
4388
* c['manhole']: Debug options. (line 17)
4389
* c['projectName']: Defining the Project.
4390
(line 15)
4391
* c['projectURL']: Defining the Project.
4392
(line 19)
4393
* c['schedulers']: Listing Change Sources and Schedulers.
4394
(line 14)
4395
* c['slavePortnum']: Setting the slaveport.
4396
(line 6)
4397
* c['sources']: Listing Change Sources and Schedulers.
4398
(line 6)
4399
* c['status']: Defining Status Targets.
4400
(line 11)
4401
* Configuration: Configuration. (line 6)
4402
* CVS Checkout: CVS. (line 6)
4403
* Darcs Checkout: Darcs. (line 6)
4404
* Dependencies: Build Dependencies. (line 6)
4405
* Dependent: Build Dependencies. (line 6)
4406
* installation: Installing the code.
4407
(line 6)
4408
* introduction: Introduction. (line 6)
4409
* IRC: IRC Bot. (line 6)
4410
* locks: Interlocks. (line 6)
4411
* logfiles: Logfiles. (line 6)
4412
* PBListener: PBListener. (line 6)
4413
* Perforce Update: P4Sync. (line 6)
4414
* Philosophy of operation: History and Philosophy.
4415
(line 6)
4416
* Scheduler: Schedulers. (line 6)
4417
* SVN Checkout: SVN. (line 6)
4418
* treeStableTimer: BuildFactory Attributes.
4419
(line 8)
4420
* Users: Users. (line 6)
4421
* Version Control: Version Control Systems.
4422
(line 6)
4423
* Waterfall: HTML Waterfall. (line 6)
4424
4425
4426
4427
Tag Table:
4428
Node: Top332
4429
Node: Introduction3581
4430
Node: History and Philosophy5458
4431
Node: System Architecture8183
4432
Node: Control Flow9614
4433
Node: Installation12452
4434
Node: Requirements12767
4435
Node: Installing the code15001
4436
Node: Creating a buildmaster16954
4437
Node: Creating a buildslave19377
4438
Node: Buildslave Options24721
4439
Node: Launching the daemons27641
4440
Node: Logfiles30153
4441
Node: Shutdown30692
4442
Node: Maintenance31276
4443
Node: Troubleshooting32668
4444
Node: Starting the buildslave32939
4445
Node: Connecting to the buildmaster34070
4446
Node: Forcing Builds35111
4447
Node: Concepts35861
4448
Node: Version Control Systems36239
4449
Ref: Version Control Systems-Footnote-137081
4450
Node: Generalizing VC Systems37227
4451
Ref: Generalizing VC Systems-Footnote-140675
4452
Node: Source Tree Specifications40896
4453
Ref: Source Tree Specifications-Footnote-143769
4454
Ref: Source Tree Specifications-Footnote-243963
4455
Node: How Different VC Systems Specify Sources44093
4456
Node: Attributes of Changes47897
4457
Node: Schedulers51370
4458
Node: BuildSet53760
4459
Node: BuildRequest56419
4460
Node: Builder57407
4461
Node: Users58680
4462
Node: Doing Things With Users59804
4463
Node: Email Addresses62169
4464
Node: IRC Nicknames64225
4465
Node: Live Status Clients65460
4466
Node: Configuration66082
4467
Node: Config File Format67307
4468
Node: Loading the Config File69682
4469
Node: Defining the Project70503
4470
Node: Listing Change Sources and Schedulers72111
4471
Ref: Listing Change Sources and Schedulers-Footnote-175464
4472
Node: Scheduler Types75581
4473
Node: Build Dependencies77721
4474
Node: Setting the slaveport79951
4475
Node: Buildslave Specifiers81369
4476
Node: Defining Builders82336
4477
Node: Defining Status Targets85895
4478
Node: Debug options86975
4479
Node: Getting Source Code Changes88695
4480
Node: Change Sources89829
4481
Node: Choosing ChangeSources90440
4482
Node: CVSToys - PBService91557
4483
Node: CVSToys - mail notification94317
4484
Node: Other mail notification ChangeSources95685
4485
Node: PBChangeSource96206
4486
Node: Build Process98541
4487
Node: Build Steps99741
4488
Node: Common Parameters101100
4489
Node: Source Checkout103118
4490
Node: CVS108331
4491
Node: SVN109473
4492
Node: Darcs115382
4493
Node: Arch117083
4494
Node: Bazaar117875
4495
Node: P4Sync118401
4496
Node: ShellCommand118958
4497
Node: Simple ShellCommand Subclasses120963
4498
Node: Configure121424
4499
Node: Compile121842
4500
Node: Test122275
4501
Node: Interlocks122501
4502
Ref: Interlocks-Footnote-1129585
4503
Node: Build Factories129894
4504
Node: BuildStep Objects130871
4505
Node: BuildFactory131888
4506
Node: BuildFactory Attributes135401
4507
Node: Quick builds136063
4508
Node: Process-Specific build factories136799
4509
Node: GNUAutoconf137343
4510
Node: CPAN139922
4511
Node: Python distutils140683
4512
Node: Python/Twisted/trial projects141957
4513
Node: Status Delivery148832
4514
Node: HTML Waterfall149871
4515
Ref: HTML Waterfall-Footnote-1151918
4516
Node: IRC Bot152087
4517
Node: PBListener154322
4518
Node: Command-line tool154764
4519
Node: Administrator Tools155290
4520
Node: Developer Tools156524
4521
Node: statuslog156843
4522
Node: statusgui157721
4523
Node: try158129
4524
Node: Other Tools172213
4525
Node: sendchange172476
4526
Node: debugclient174157
4527
Node: .buildbot config directory176733
4528
Node: Resources178546
4529
Node: Developer's Appendix178967
4530
Node: Index179674
4531
4532
End Tag Table
Loggerhead is a web-based interface for Breezy
Version: 2.0.1