~fta/+junk/ppa-scripts

This package contains a bot that could be used to automatically create

new revisions of one or more packages and submit them to one or more

repositories, most likely PPAs. It could do several distributions in a row,

assuming the packaging is ready for that.

It could run in cron to produce daily/nightly builds, or manually whenever

needed to help with the package maintenance.

Prerequisites: bzr bzr-builddeb

The packaging (debian/ directory) has to be in a bazaar branch in order to

be mergeable with bzr-builddeb. Native packages are also allowed.

The bot will create a few bzr branches per package and it will never touch

the packaging branch itself.

The packaging *must* provide a get-orig-source target (in debian/rules) in order

for the bot to create the proper tarballs. Special attention must be given to

the versionning scheme. Versions *must* always increase. If the package is also

in the official Ubuntu repositories, it is also advised to make sure that

packages created by this bot integrate well with the official packages.

Please read the official recommendations regarding Versionning in PPAs:

https://help.launchpad.net/Packaging/PPA#Versioning

The bot is composed of some scripts and some configuration files. Those scripts

could be run independently or chained (by daily.sh).

- ppabot.conf:

=====

{

  # 'dist'          => `lsb_release -c -s`,

  # 'alt_dists'     => [ 'intrepid', 'hardy' ], # other than $dist, will be taggued as ~dist

  # 'who'           => `bzr whoami`,

  # 'watchers'      => `bzr whoami`, # list of contacts who will receive the logs

  # 'dput_target'   => "ppa", # from ~/.dput.cf

  # 'iteration_tag' => "ppa", # "xxx" => -0ubuntu1~xxx1

  # 'ppa_branchtag' => "ppa", # "xxx" => package.head.xxx

  # 'gpg_key'       => undef, # "ABCDEF"

  # 'test_suffix'   => "test",

  # 'build_dir'     => "$ENV{'HOME'}/ppa",

  # 'tarballs_dir'  => "../tarballs", # absolute dir or relative to the branch dir

}

=====

Those are the default values for all packages of a given pocket (group).

If some values need to be changed, the corresponding lines must be uncommented.

If you have more than one version of the same source package (like a stable and a

dev version), you need to put then in different pockets and assign them different

build_dir and tarballs_dir values (one pair could stay the default).

note: it's a perl script so the syntax could be manually checked with perl -c

- ppabot-pkgs.conf

This file describes a given pocket.

=====

{

  'pkgs' => {

    # 'nspr'               => { 'branch' => 'nspr.head' },

    # 'nss'                => { 'branch' => 'nss.head' },

    # 'xulrunner-1.9.1'    => { 'branch' => 'xulrunner-1.9.1.head' },

    # 'firefox-3.1'        => { 'branch' => 'firefox-3.1.head' },

    # 'fennec'             => { 'branch' => 'fennec.head'},

   },

  'order' => [

    # 'nspr', 'nss',

    # 'xulrunner-1.9.1', 'firefox-3.1',

    # 'fennec',

  ],

  'deps' => {       # bin => src

    # 'libnspr4-dev'        => 'nspr',

    # 'libnss3-dev'         => 'nss',

    # 'xulrunner-1.9.1-dev' => 'xulrunner-1.9.1',

  },

  # Don't update dependencies for those packages

  'nodeps' => [

    # 'fennec',

  ],

};

=====

note: it's a perl script so the syntax could be manually checked with perl -c

More fields are recognized, especially in the 'pkgs' subsection.

The comprehensive list is:

  Mandatory: 

  * 'branch': basename of the packaging branch (as stored locally)

  * 'lp': full URL of the remote bzr branch (usually on Launchpad)

  Optional:

  * 'no_auto': a boolean allowing a package to be skipped by the bot while

     remaining usable manually. Default: 0 / "false"

  * 'no_update': similar to 'no_auto' but just skip the tarball update. The

     sync is still performed, allowing packaging changes to be merged in.

     Default: 0 / "false"

  * 'upstream': directory where to keep a local copy of the upstream branch.

     It is passed to get-orig-source through LOCAL_BRANCH by update-pkg.sh

     (see below). No default (no cache)

  * 'local_init': a shell command which could be used to populate the 'upstream'

     directory if it's empty or absent. No default (no cache)

  * 'upgrade_flags': some flags passed to get-orig-sources. No default.

  * 'ppa': basename of the PPA branch containing the result of the merges. This

     where work happens. Default: 'branch'.'ppa_branchtag' (from the main

     configuration file)

  * 'vpattern': a regular expression applied on the version. It must match

     in order to continue with that new tarball. This is useful if you want

     to make sure that a versionned package stays bound. No default (no

     matching requested). For example, firefox-3.6 with a vpattern set to

     '^3\.6(\~|\.)' will match 3.6.x but not 3.7.

  * 'build-deps': a structure used to dynamically add build-dependencies. It

     could be global, per package and per distribution. This is useful when

     touching the packaging branch is not desirable, or not possible (like

     when some build-deps are only available on some distributions).

     Example:

     'pkgs' => {

       'chromium-browser' => {

         'branch'     => 'chromium-browser.head',

         'lp'         => 'lp:~chromium-team/chromium-browser/chromium-browser.head',

         'upstream'   => 'upstream/chromium-browser.svn',

         'build-deps' => {

           'all_dists' => [ 'foo | bar', 'baz (>= 1.2.3)' ],

           'karmic'    => [ 'binutils-gold' ],

         }

       }

     }

     ...

     'build-deps' => {

       'all_pkgs' => [ 'lintian' ],

       'ia32-libs-chromium-browser' => [ 'dep1', 'dep2 [!amd64]' ],

     }

     In this example, all packages will have lintian for all arches,

     ia32-libs-chromium-browser will have 'dep1, dep2 [!amd64]' for all

     arches, 'chromium-browser' will have 'foo | bar, baz (>= 1.2.3)' for

     all distributions and binutils-gold only for karmic.

- update-pkg.sh

=====

Usage: update-pkg.sh [-n|--new] [-c conf] [-p pkg-conf] [-d|--date YYYYMMDDtHHMM] [-L .../branch] branch [branch2 [...]]

       update-pkg.sh [-n|--new] [-D|--daily] [--test] [-t|--tag SOMETAG=someversion] branch

       update-pkg.sh [-N|--no-upgrade] [...] branch

 By default, email, date and version are always updated.

 To create a new version, use -n

 To prevent a bzr pull/push, use --test

 To prevent a bzr pull, and push in overwrite mode, use --fix

 To work from a local branch, use -L

 To update the package branch from the last daily snapshot, use --daily

 To prevent the upgrade, but still do all the checking, use --no-upgrade

=====

update-pkg.sh performs the following actions:

 1/ it tries to 'pull' the packaging branch (in case there are new revisions on

    the server)

   *Note* it implies the location has already been registered with --remember

    for pull, unless it is specified in the package configuration file.

 2/ it calls the get-orig-source target from that branch, possibly with some

    attributes:

    - DEBIAN_DATE  = value of --date

    - DEBIAN_TAG   = value of --tag

    - LOCAL_BRANCH = value of -L

    get-orig-source could make use of these attributes to get a particular

    upstream revision based on a tag or a date. The local branch could be

    useful to keep a local copy of the upstream branch and then do an update

    instead of a full checkout.

    It is important to understand the magic has to be in get-orig-source as

    there's no generic way for the script to know the specifics of upstream and

    its packaging.

 3/ it updates debian/changelog, either by updating the version & date of the

    current revision (should be UNRELEASED), or by creating a new entry if --new

    was received.

 4/ it pushes the packaging branch back

   *Note* it implies the location has already been registered with --remember

   for push, unless it is specified in the package configuration file.

 Examples:

   $ update-pkg.sh -t FIREFOX_3_5b99_RELEASE=3.5~b99 firefox-3.5.head

   $ update-pkg.sh -L ../upstream/chromium-browser.svn chromium-browser.head

   $ update-pkg.sh -c ../confs/pkg-ucd.conf -p ../confs/pkg-pkgd-ucd.conf chromium-browser.head

- sync-ppa.pl

=====

Usage: sync-ppa.pl [-d archive] [-c conf] [-p pkgconf] [-f] [-i] [-P] [-t] [-h] [-A] [pkg1] [pkg2] [...]

  -h           this help screen

  -c conf      use 'conf' as alternative conf file

  -p pkgconf   use 'pkgconf' as alternative package conf file

  -d archive   push to an alternative archive (default: ppa)

  -t           use test branches, if necessary branched off the configured ones

  -A           don't do alternative dists

  -i           initialize the PPA branches and quit

  -f           force the merge, even if it's not needed

  -P           do the dPut at the end

  -r           revert the last sync

=====

sync-ppa.pl creates and maintains one branch per package and per distribution.

Those branches are merged down, starting from the packaging branch.

It is able to detect when a new upload is needed, either if there's a new

tarball, or some changes to the packaging branch. It takes care of the

version iterators.

  Examples:

   $ sync-ppa.pl -c ../confs/pkg-ucd.conf -p ../confs/pkg-pkgd-ucd.conf -P chromium-browser.head

  Note: when -c (resp. -p) is not specified, sync-ppa.pl will try to use

   ~/.ppabot.conf (resp. ~/.ppabot-pkgs.conf), or /etc/ppabot.conf (resp.

   /etc/ppabot-pkgs.conf) if they exist (the first found wins).

- daily.sh

Usage: daily.sh [-n|--no-tarball-update] [-f|--force] pocket1 [pocket2] [...]

   -e or --email                Email logs to the pocket watchers

   -n or --no-tarball-update    Don't update tarballs

   -s or --no-sync              Don't sync the branches (implies --no-push)

   -p no --no-push              Prevent the final dput to the builders

   -f or --force                Force a sync

   -h or --help                 This help screen

daily.sh is the core the bot, calling the other scripts with the right

arguments.

if --email is specified, all the 'watchers' for a given pocket will receive

the logs for that pocket. It defaults to 'who'.

Note: this requires MIME::Entity (libmime-tools-perl) and a valid sendmail

===

FIXME: for now, this script must be edited to update the following variables:

WORKDIR=/data/bot      => where all the branches live

BINDIR=/data/bot/src   => where all the scripts are located

CONFDIR=$WORKDIR/confs => where all the conf files are located

===

It expects a specific format for the conf filenames.

For a given pocket $POCKET, $CONFDIR/ppabot-pkgs-$POCKET.conf and

$CONFDIR/ppabot-$POCKET.conf must exist.

This allows the bot to work with several pockets.

This script could be run in cron.

===

FIXME: the output is really ugly, mostly useful to troubleshoot at this point

===

Here is a full example with gwibber:

get-orig-source:

See http://bazaar.launchpad.net/~gwibber-team/gwibber/packaging/annotate/head:/debian/rules

It takes advantage of DEBIAN_TAG.

configuration files:

http://bazaar.launchpad.net/~fta/+junk/ppa-confs/annotate/head:/ppabot-gwibber.conf

http://bazaar.launchpad.net/~fta/+junk/ppa-confs/annotate/head:/ppabot-pkgs-gwibber.conf

Once get-orig-source and the conf files are in place, it's just a matter of

running "daily.sh gwibber" in cron

The resulting debs are available here:

https://edge.launchpad.net/~gwibber-daily/+archive/ppa?field.name_filter=&field.status_filter=any&field.series_filter=any

The bot also publishes the daily branch:

https://code.edge.launchpad.net/~gwibber-daily/gwibber/gwibber.head.daily