~linaro-automation/linaro-license-protection/configs

Linaro License Protection configuration
=======================================

This branch contains current production configuration for

 * releases.linaro.org (hosted on mombin.canonical.com)
 * snapshots.linaro.org (mombin)
 * staging.releases.linaro.org (kahaku.canonical.com)
 * staging.snapshots.linaro.org (kahaku.canonical.com)


Dependencies
------------

libapache2-mod-xsendfile >= 0.10
libapache2-mod-python
python-django >= 1.3.1
python-django-openid-auth
python-apache-openid

python-apache-openid is needed until we migrate existing apache
openid-protected directories to the new setup.


Pushing framework on snapshots.linaro.org
-----------------------------------------

Pushing/uploading side is currently implemented as a multi-system setup:

 * configuration on jenkins
 * configuration of users on server

Configuration of jenkins
........................

 * We are using Publish-over-SSH plugin for jenkins

 * We perform two build (not publish, to allow for LAVA to have known URLs)
   steps:

   * publish artifacts over SSH (ends up on a private location on the server)

   * call out a trigger script over SSH (to move files securely to a public
     location)

 * We use strictly "push from master" in advanced settings of publish-over-ssh
   plugin, and keep private keys for both actions above on the master:

    /home/ubuntu/snapshots-sync2/linaro-android-build-publish —
      linaro-android-build-publish
    /home/ubuntu/snapshots-sync2/linaro-android-build-publish-trigger —
      linaro-android-build-publish-trigger

Configuration of users on the receiving server (snapshots.l.o)
..............................................................

For the two actions above, we have two separate users for all hosts
accessing the system (two for android-build.linaro.org and another two
for ci.linaro.org at this time).

One of the users is a sftp/push user, and another is the trigger user that
shuffles the files around.

Current users and their configurations:

 * linaro-android-build-publish
   /etc/ssh/user-authorized-keys/linaro-android-build-publish

     command="/usr/lib/sftp-server",no-pty,no-port-forwarding,\
     no-X11-forwarding,no-agent-forwarding,from="50.17.250.69" \
     <PUBLIC-SSH-KEY-DATA>

   (chrooted to /srv/snapshots.linaro.org/uploads/)

 * linaro-android-build-publish-trigger:

   /etc/ssh/user-authorized-keys/linaro-android-build-publish-trigger

     command="/home/linaro-android-build-publish-trigger/scripts\
              /trigger-linaro-android-build-publish.sh \
              ${SSH_ORIGINAL_COMMAND#* }",\
     no-pty,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,
     from="50.17.250.69" <PUBLIC-SSH-KEY-DATA>

   The script being executed passes parameters to publish_to_snapshots
   which moves files from /uploads/ into appropriate public directory:

     #!/bin/sh
     if [ "$(id -un)" != "linaro-android-build-publish-trigger" ]; then
       echo "This script is designed to be run as linaro-android-build-publish-trigger user"
       exit 1
     fi
     sudo -u linaro-android-build-publish /home/linaro-android-build-publish/linaro-license-protection/scripts/publish_to_snapshots.py "$@"

 * linaro-ci-publish

   Exactly the same as linaro-android-build-publish, except for the IP
   (allowing ci.linaro.org from 50.17.200.206) and SSH key data.

 * linaro-ci-publish-trigger

   Like linaro-android-build-publish-trigger, with different IP
   and SSH key data.  Script lives in

     /home/linaro-ci-publish-trigger/scripts/trigger-linaro-ci-publish.sh

   and is of similar structure to the one for the user
   linaro-android-build-publish-trigger.

android-build.linaro.org
........................

Runs Jenkins and uses SFTP plugin to access the above two users.  Private
keys live in

  /home/ubuntu/snapshots-sync2/linaro-android-build-publish —
    linaro-android-build-publish
  /home/ubuntu/snapshots-sync2/linaro-android-build-publish-trigger —
    linaro-android-build-publish-trigger

To ensure serialization of steps, and allow LAVA submission, these happen as
build steps, and not as publishing steps.

Deployment steps
----------------

This documents our current deployment while at the same time representing
an example production deployment.

1. Install the dependencies
   (see the "Dependencies" section)

2. Get the code

     mkdir -p /srv/shared-branches
     cd /srv/shared-branches
     bzr branch lp:linaro-license-protection

   (we are actually using http URLs since lp: defaults to bzr+ssh which
   doesn't work on system accounts)

3. Get the configuration

   Configuration files for deploying to snapshots.linaro.org,
   releases.linaro.org, staging.snapshots.linaro.org
   and staging.releases.linaro.org all live in

     lp:linaro-license-protection/configs

   We need to branch that into /srv/shared-branches as well:

     cd /srv/shared-branches
     bzr branch lp:linaro-license-protection/configs linaro-license-protection-configs

4. Checkout branches for the services you want to use:

     (cd /srv/staging.snapshots.linaro.org &&
      bzr checkout /srv/shared-branches/linaro-license-protection-configs configs &&
      bzr checkout /srv/shared-branches/linaro-license-protection)

   Replace "/srv/staging.snapshots.linaro.org" with one of
     /srv/staging.releases.linaro.org
     /srv/snapshots.linaro.org
     /srv/releases.linaro.org

   depending on the service you are deploying.

   If you use these paths, none of the config files will need updating.

5. Configure apache2

   Make sure mod-xsendfile and mod-python are enabled.
   Copy appropriate
     /srv/staging.snapshots.linaro.org/configs/apache/staging.snapshots.linaro.org
   files to /etc/apache2/sites-available

   Copy apache/security/ directory to /etc/apache2/security (it contains OpenID group
   mappings, etc).

   Add appropriate NameVirtualHost directive to

     /etc/apache2/sites-enabled/000-default

   Run "a2ensite staging.snapshots.linaro.org".

   You also need to set up SSL certificate for *snapshots.linaro.org.

6. Create databases and set up static files

   Make sure appropriate configs/django directory is in the PYTHONPATH
   and set DJANGO_SETTINGS_MODULE and then run django-admin:

     export PYTHONPATH=/srv/staging.snapshots.linaro.org:/srv/staging.snapshots.linaro.org/linaro-license-protection:/srv/staging.snapshots.linaro.org/configs/django
     export DJANGO_SETTINGS_MODULE=settings_staging_snapshots
     mkdir -p /srv/staging.snapshots.linaro.org/db
     django-admin syncdb --noinput
     django-admin collectstatic --noinput

7. Set up a cron job to keep code updating automatically

   You may want to automatically keep code up to date (mostly for staging)
   instances.

   There is

     scripts/update-deployment.py

   from lp:linaro-license-protection which can be run from a cronjob to keep
   the above deployment always up to date.

8. Ensure license_protected_downloads/config.py contains relevant white-listed
   IP addresses:

     For releases.linaro.org:
      * android-build.linaro.org (50.17.250.69)
      * validation.linaro.org (213.123.120.124)