1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
|
Operating cdimage
=================
cdimage is a collection of Python scripts wrapped around the debian-cd
package. Originally, it was just the few things you had to type to make
debian-cd build CDs; since then it has evolved to handle generating package
lists using germinate, live CD building, and CD publication for multiple
projects. The following is a quick operator's guide.
cdimage requires Python >= 2.7. It should also work with Python >= 3.1 (not
3.0), but at the moment this has only been confirmed by unit-testing and not
by field experience.
Running the cdimage test suite also requires python-mock.
Day-to-day operation
--------------------
etc/crontab lists the jobs that are run regularly from cron. These include
the following scripts, which may be called by hand:
* cron.daily
Build install CDs for all relevant Ubuntu architectures.
* cron.daily-live
Build live CDs for all relevant Ubuntu architectures.
* cron.dvd
Build DVDs for all relevant Ubuntu architectures. These are normally
combined install/live images.
* for-project
Execute a command in the context of a certain project: run for-project
without arguments to see the available list. At the moment, this just
sets the PROJECT and CAPPROJECT environment variables and then runs the
command (many parts of cdimage and of the Ubuntu branch of debian-cd pay
attention to these variables), but you should always use the for-project
command to do this in case this changes.
The project context affects a number of parameters of image building and
publication, such as the set of seeds passed to germinate, the location
of live filesystem images, the preseed files installed on the CD, the
place where images are published in the WWW output tree, and the
frequency with which old published images are purged.
Each of these scripts sets a few variables and then calls the main
underlying script, which is the core of cdimage:
* build-image-set
This calls lots of other scripts to build a set of images. For day-to-day
operation it is not necessary to be intimately familiar with all of these
scripts, but fixing bugs may require some level of knowledge of them.
Here's a quick run-down of the steps:
* anonftpsync
Sync cdimage's mirror of the Ubuntu archive.
* update-local-indices
Update index files for the local override archive; packages in this
archive will be used in preference to packages in the Ubuntu archive
proper. Nowadays, of course, this local archive is rarely used, but it
is still occasionally useful for specialised custom builds.
* extract-debootstrap
Extract debootstrap scripts from debootstrap-udeb packages in the
archive. debian-cd uses these to make sure that it has a complete base
system on the CD. (This is largely unnecessary as of Ubuntu 5.10, since
debootstrap now calculates the base system based on the Packages file.)
* run-germinate
Run germinate for each architecture to determine the list of packages in
each seed output (desktop, installer, etc.). See
https://wiki.ubuntu.com/SeedManagement and the germinate documentation
for more details.
* germinate-to-tasks
Translate germinate output into debian-cd task lists.
* update-tasks
Install new debian-cd task lists. This can optionally mail the
differences versus the last set of task lists to an operator, which can
be useful for spotting problems.
* download-live-filesystems
For live CD builds, fetch the live filesystem image from a build daemon.
At this stage, debian-cd's build_all.sh script is invoked to build the
images.
* check-installable
For install CD builds, generate a report on uninstallable packages on
the image.
* publish-daily
Publish the image to the WWW output tree.
* purge-old-images
Delete images older than a certain number of days (configured in
etc/purge-days) from the WWW output tree.
* sync-mirrors
Trigger mirrors to update from the WWW output tree, thereby making the
images available to the world.
Releasing images
----------------
From time to time, an operator will need to tag a set of images as a release
of some kind. The script responsible for this (which is always run by hand,
never automatically) is:
* publish-release
This is heavily geared to the requirements of Ubuntu and its
derivatives, although it's possible that other users could adapt it to
their needs. It publishes images from a daily build (as published by
publish-daily) to a special release tree. An OFFICIAL argument governs
the layout of the result. Run publish-release without arguments to see
a usage message.
There are in fact three important subtrees of the release tree. The
'simple' tree is intended for smaller mirrors and for ease of use by naïve
end users. It contains a pool of images and a tree per release of symlinks
into that pool with filenames that include the status of the image (e.g.
preview, sounder-9, release).
The 'full' tree contains everything except the releases that are in the
simple tree (so in practice it contains alpha/beta releases), and has a more
complicated structure that ordinary users ultimately shouldn't have to pay
too much attention to.
The 'torrent' tree is mirrored by the BitTorrent tracker so that it knows
what it's supposed to be tracking. It is not otherwise visible to the
world.
It is best to give a few examples here. The Breezy Colony 5 release was
published using the following commands:
for-project ubuntu publish-release daily 20050923.3 install no colony-5
for-project ubuntu publish-release daily-live 20050923.1 live no colony-5
sync-mirrors
As you can see, it is possible to publish two sets of images into the same
release directory, provided that they have different types (the "install"
and "live" arguments).
Note that it is always necessary to run sync-mirrors after publish-release;
publish-release does not do so automatically, in order to give the operator
a chance to make sure that the release tree is correct before publishing it
to the world with sync-mirrors.
The Edubuntu 5.10 Preview release (which consisted of only an install CD)
was initially published using the following commands:
for-project edubuntu publish-release daily 20050909.4 install poolonly preview
sync-mirrors
The "poolonly" OFFICIAL argument instructs publish-release to place the
images only in the .pool directory, which is generally not visible in
directory listings on mirrors. This is polite to mirrors by giving them a
little time to catch up before the images are made visible to everyone (and
generally announced shortly afterwards) using the following commands:
for-project edubuntu publish-release daily 20050909.4 install yes preview
sync-mirrors
When the Edubuntu 5.10 release is later published, publish-release will
automatically remove the preview images.
HTML indices
------------
A requirement common to both daily and release publication is the generation
of HTML pages describing the images that are available. This is mostly
handled automatically, but the script is full of special cases and sadly the
output is not always perfect, so it is sometimes necessary to correct it and
run it by hand.
* make-web-indices
Generate HEADER.html, FOOTER.html, and .htaccess files instructing
Apache to emit directory listings with added descriptions of what images
are available.
The HEADER.html, FOOTER.html, and .htaccess files at the top of the simple
tree and in each project's subdirectory are currently generated by hand, as
are the version number symlinks; because they only change every six months,
there has not been much impetus to automate this task. When publishing a
release to the simple tree for the first time, the operator will need to
edit these files manually.
|