~xubuntu-dev/ubuntu-cdimage/xubuntu-base

Operating cdimage

=================

cdimage is a collection of Python scripts wrapped around the debian-cd

package.  Originally, it was just the few things you had to type to make

debian-cd build CDs; since then it has evolved to handle generating package

lists using germinate, live CD building, and CD publication for multiple

projects.  The following is a quick operator's guide.

cdimage requires Python >= 2.7.  It should also work with Python >= 3.1 (not

3.0), but at the moment this has only been confirmed by unit-testing and not

by field experience.

Running the cdimage test suite also requires python-mock.

Day-to-day operation

--------------------

etc/crontab lists the jobs that are run regularly from cron.  These include

the following scripts, which may be called by hand:

  * cron.daily

    Build install CDs for all relevant Ubuntu architectures.

  * cron.daily-live

    Build live CDs for all relevant Ubuntu architectures.

  * cron.dvd

    Build DVDs for all relevant Ubuntu architectures.  These are normally

    combined install/live images.

  * for-project

    Execute a command in the context of a certain project: run for-project

    without arguments to see the available list.  At the moment, this just

    sets the PROJECT and CAPPROJECT environment variables and then runs the

    command (many parts of cdimage and of the Ubuntu branch of debian-cd pay

    attention to these variables), but you should always use the for-project

    command to do this in case this changes.

    The project context affects a number of parameters of image building and

    publication, such as the set of seeds passed to germinate, the location

    of live filesystem images, the preseed files installed on the CD, the

    place where images are published in the WWW output tree, and the

    frequency with which old published images are purged.

Each of these scripts sets a few variables and then calls the main

underlying script, which is the core of cdimage:

  * build-image-set

This calls lots of other scripts to build a set of images.  For day-to-day

operation it is not necessary to be intimately familiar with all of these

scripts, but fixing bugs may require some level of knowledge of them.

Here's a quick run-down of the steps:

  * anonftpsync

    Sync cdimage's mirror of the Ubuntu archive.

  * update-local-indices

    Update index files for the local override archive; packages in this

    archive will be used in preference to packages in the Ubuntu archive

    proper.  Nowadays, of course, this local archive is rarely used, but it

    is still occasionally useful for specialised custom builds.

  * extract-debootstrap

    Extract debootstrap scripts from debootstrap-udeb packages in the

    archive.  debian-cd uses these to make sure that it has a complete base

    system on the CD.  (This is largely unnecessary as of Ubuntu 5.10, since

    debootstrap now calculates the base system based on the Packages file.)

  * run-germinate

    Run germinate for each architecture to determine the list of packages in

    each seed output (desktop, installer, etc.).  See

    https://wiki.ubuntu.com/SeedManagement and the germinate documentation

    for more details.

  * germinate-to-tasks

    Translate germinate output into debian-cd task lists.

  * update-tasks

    Install new debian-cd task lists.  This can optionally mail the

    differences versus the last set of task lists to an operator, which can

    be useful for spotting problems.

  * download-live-filesystems

    For live CD builds, fetch the live filesystem image from a build daemon.

At this stage, debian-cd's build_all.sh script is invoked to build the

images.

  * check-installable

    For install CD builds, generate a report on uninstallable packages on

    the image.

  * publish-daily

    Publish the image to the WWW output tree.

  * purge-old-images

    Delete images older than a certain number of days (configured in

    etc/purge-days) from the WWW output tree.

  * sync-mirrors

    Trigger mirrors to update from the WWW output tree, thereby making the

    images available to the world.

Releasing images

----------------

From time to time, an operator will need to tag a set of images as a release

of some kind.  The script responsible for this (which is always run by hand,

never automatically) is:

  * publish-release

    This is heavily geared to the requirements of Ubuntu and its

    derivatives, although it's possible that other users could adapt it to

    their needs.  It publishes images from a daily build (as published by

    publish-daily) to a special release tree.  An OFFICIAL argument governs

    the layout of the result.  Run publish-release without arguments to see

    a usage message.

There are in fact three important subtrees of the release tree.  The

'simple' tree is intended for smaller mirrors and for ease of use by naïve

end users.  It contains a pool of images and a tree per release of symlinks

into that pool with filenames that include the status of the image (e.g.

preview, sounder-9, release).

The 'full' tree contains everything except the releases that are in the

simple tree (so in practice it contains alpha/beta releases), and has a more

complicated structure that ordinary users ultimately shouldn't have to pay

too much attention to.

The 'torrent' tree is mirrored by the BitTorrent tracker so that it knows

what it's supposed to be tracking.  It is not otherwise visible to the

world.

It is best to give a few examples here.  The Breezy Colony 5 release was

published using the following commands:

  for-project ubuntu publish-release daily 20050923.3 install no colony-5

  for-project ubuntu publish-release daily-live 20050923.1 live no colony-5

  sync-mirrors

As you can see, it is possible to publish two sets of images into the same

release directory, provided that they have different types (the "install"

and "live" arguments).

Note that it is always necessary to run sync-mirrors after publish-release;

publish-release does not do so automatically, in order to give the operator

a chance to make sure that the release tree is correct before publishing it

to the world with sync-mirrors.

The Edubuntu 5.10 Preview release (which consisted of only an install CD)

was initially published using the following commands:

  for-project edubuntu publish-release daily 20050909.4 install poolonly preview

  sync-mirrors

The "poolonly" OFFICIAL argument instructs publish-release to place the

images only in the .pool directory, which is generally not visible in

directory listings on mirrors.  This is polite to mirrors by giving them a

little time to catch up before the images are made visible to everyone (and

generally announced shortly afterwards) using the following commands:

  for-project edubuntu publish-release daily 20050909.4 install yes preview

  sync-mirrors

When the Edubuntu 5.10 release is later published, publish-release will

automatically remove the preview images.

HTML indices

------------

A requirement common to both daily and release publication is the generation

of HTML pages describing the images that are available.  This is mostly

handled automatically, but the script is full of special cases and sadly the

output is not always perfect, so it is sometimes necessary to correct it and

run it by hand.

  * make-web-indices

    Generate HEADER.html, FOOTER.html, and .htaccess files instructing

    Apache to emit directory listings with added descriptions of what images

    are available.

The HEADER.html, FOOTER.html, and .htaccess files at the top of the simple

tree and in each project's subdirectory are currently generated by hand, as

are the version number symlinks; because they only change every six months,

there has not been much impetus to automate this task.  When publishing a

release to the simple tree for the first time, the operator will need to

edit these files manually.