~allenap/maas/consolidate-namespace-metadataserver

.. -*- mode: rst -*-

************
Hacking MAAS
************


Coding style
============

MAAS follows the `Launchpad Python Style Guide`_, except where it gets
Launchpad specific, and where it talks about `method naming`_. MAAS
instead adopts `PEP-8`_ naming in all cases, so method names should
usually use the ``lowercase_with_underscores`` form.

.. _Launchpad Python Style Guide:
  https://dev.launchpad.net/PythonStyleGuide

.. _method naming:
  https://dev.launchpad.net/PythonStyleGuide#Naming

.. _PEP-8:
  http://www.python.org/dev/peps/pep-0008/


Prerequisites
=============

You will need to manually install Postgres 9.1, RabbitMQ, Apache 2,
python-dev, avahi-daemon, daemontools, pyinotify, and make::

    $ sudo apt-get install postgresql-9.1 rabbitmq-server python-dev \
        avahi-daemon make apache2-mpm-prefork daemontools python-pyinotify

Also, you might want to install Bazaar (``bzr``) to grab the source
code directly from Launchpad::

    $ sudo apt-get install bzr

This is the list of runtime dependencies that you'll need to install::

    $ sudo apt-get install python-django python-django-piston \
        python-django-south python-twisted python-txamqp python-amqplib \
        python-formencode python-oauth python-oops python-oops-datedir-repo \
        python-twisted python-oops-wsgi python-oops-twisted \
        python-psycopg2 python-yaml python-convoy python-django-south \
        python-avahi python-dbus python-celery

Additionally, you need to install the following python libraries
for development convenience::

    $ sudo apt-get install python-sphinx python-lxml python-pocket-lint

If you intend to run the test suite, you also need xvfb and firefox::

    $ sudo apt-get install xvfb firefox avahi-utils

All other development dependencies are pulled automatically from `PyPI`_
when buildout runs.

.. _PyPI:
  http://pypi.python.org/


First time using buildout?
==========================

Buildout is used to develop MAAS. Buildout, if configured so, can
cache downloaded files and built eggs. If you've not already done
something similar, the following snippet will massively improve build
times::

    [buildout]
    download-cache = /home/<your-user-name>/.buildout/cache
    eggs-directory = /home/<your-user-name>/.buildout/eggs

Put this in ``~/.buildout/default.cfg`` and create the ``cache``
directory::

    $ mkdir /home/<your-user-name>/.buildout/cache

The ``eggs`` directory will be created automatically.


Getting the latest version of the code
======================================

You can grab the code manually from Launchpad but Bazaar makes it easy to fetch
the last version of the code. Go into the directory where you want the code
to reside and run::

    $ bzr branch lp:maas maas && cd maas


Running tests
=============

To run the whole suite::

    $ make test

To run tests at a lower level of granularity::

    $ bin/test.maas maasserver.tests.test_api -v
    $ bin/test.pserv maas.tests.tests.test_api -v

The test runner is `nose`_, so you can pass in options like
``--with-coverage`` and ``--nocapture`` (short option: ``-s``). The
latter is essential when using ``pdb`` so that stdout is not
adulterated.

.. _nose: http://readthedocs.org/docs/nose/en/latest/


Running tests with a real Cobbler
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Once you have a Cobbler instance running somewhere there are some
tests that can be run against it::

    $ PSERV_TEST_COBBLER_URL=http://user:pass@localhost/cobbler_api \
        bin/test.pserv maas.pserv.tests.test_api

Adjust ``PSERV_TEST_COBBLER_URL`` as necessary. **Warning:** these
tests will modify your Cobbler database.


Running JavaScript tests
^^^^^^^^^^^^^^^^^^^^^^^^

The JavaScript tests are run using Selenium_.  Firefox is the default
browser but any browser supported by Selenium can be used to run the
tests. Note that you might need to download the appropriate driver and
make it available in the path.  You can then choose which browsers to use by
setting the environment variable ``MAAS_TEST_BROWSERS`` to a comma-separated
list of the names of the browsers to use.  For instance, to run the tests
with Firefox and Chrome::

    $ export MAAS_TEST_BROWSERS="Firefox, Chrome"

.. _Selenium: http://seleniumhq.org/


Running JavaScript tests with browsers on other platforms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The JavaScript tests can be run using the `SauceLabs' OnDemand`_
service. There is a free version of this that provides 45 minutes a
month of testing. To get started, `sign up`_ and go to your `account
page`_, select the *Account* tab, and click *View my API Key*. Now
save your credentials::

  $ mkdir -p ~/.saucelabs/connect
  $ chmod go-rwx ~/.saucelabs/connect
  $ echo "${username} ${api_key}" > ~/.saucelabs/connect/credentials

(You need to substitute your SauceLabs username and API key above.)

Next, like when running Selenium tests locally, you need to specify
the browsers to run on. At the time of writing there are four to
choose from, all running on a Windows host::

  $ export MAAS_REMOTE_TEST_BROWSERS="IE7, IE8, IE9, Chrome"

By default, when ``MAAS_REMOTE_TEST_BROWSERS`` is not specified,
testing via OnDemand is *not* attempted.

.. _SauceLabs' OnDemand: http://saucelabs.com/

.. _sign up: http://saucelabs.com/pricing

.. _account page: https://saucelabs.com/account


Development MAAS server setup
=============================

Access to the database is configured in ``src/maas/development.py``.

The ``Makefile`` or the test suite sets up a development database
cluster inside your branch. It lives in the ``db`` directory, which
gets created on demand. You'll want to shut it down before deleting a
branch; see below.

First, set up the project. This fetches all the required dependencies
and sets up some useful commands in ``bin/``::

    $ make

Create the database cluster and initialize the development database::

    $ make syncdb

Optionally, populate your database with the sample data::

    $ make sampledata

Install Cobbler, and install the preseed files and snippets::

    $ sudo apt-get install apache2 cobbler
    $ sudo cp contrib/preseeds/* /var/lib/cobbler/kickstarts/
    $ sudo cp contrib/snippets/* /var/lib/cobbler/snippets/

Now run ``maas-import-isos`` to download Ubuntu precise and create the
necessary MAAS profiles::

    $ sudo http_proxy=$http_proxy ./scripts/maas-import-isos

The http_proxy variable is only needed if you're downloading through a
proxy; "sudo" wouldn't pass it on to the script without the assignment.
Or if you don't have it set but do want to download through a proxy, pass
your proxy's URL: "http_proxy=http://proxy.example.com/"

Now add yourself as a user in Cobbler. For convenience, give yourself the
password ``test``::

    $ sudo htdigest /etc/cobbler/users.digest Cobbler $USER

Alternatively, if you're not interested in the Provisioning Server or
Cobbler, set ``USE_REAL_PSERV`` to ``False`` in one of ``maas``'s
settings files (typically ``src/maas/demo.py``), and a fake Provisioning
Server will be used instead.

Run the development webserver and watch all the logs go by::

    $ make run

Point your browser to http://localhost:5240/

If you've populated your instance with the sample data, you can login as a
simple user using the test account (username: 'test', password: 'test') or the
admin account (username: 'admin', password: 'test').

To shut down the database cluster and clean up all other generated files in
your branch::

    $ make distclean


Development services
====================

The development environment uses *daemontools* to manage the various
services that are required. These are all defined in subdirectories in
``services/``.

There are familiar service-like commands::

  $ make start
  $ make status
  $ make restart
  $ make stop

The latter is a dependency of ``distclean`` so just running ``make
distclean`` when you've finished with your branch is enough to stop
everything.

Individual services can be manipulated too::

  $ make services/pserv/@start

The ``@<action>`` pattern works for any of the services.

There's an additional special action, ``run``::

  $ make run

This starts all services up and tails their log files. When you're
done, kill ``tail`` (e.g. Ctrl-c), and all the services will be
stopped.

However, when used with individual services::

  $ make services/webapp/@run

it does something even cooler. First it shuts down the service, then
it restarts it in the foreground so you can see the logs in the
console. More importantly, it allows you to use ``pdb``, for example.

A note of caution: some of the services have slightly different
behaviour when run in the foreground:

* Django (the *webapp* service) will be run with its auto-reloading
  enabled.

* Apache (the *web* service) will run with ``-X``, which puts it in
  debug mode: only one worker will be started and the server will not
  detach from the console.

There's a convenience target for hacking Django that starts everything
up, but with Django in the foreground::

  $ make run+webapp

Apparently Django needs a lot of debugging ;)


Adding new dependencies
=======================

Since MAAS is distributed mainly as Ubuntu package, all runtime dependencies
should be packaged and we should develop with the packaged version if
possible. You'll need to add the dependency to the
``allowed-eggs-from-site-packages`` option in the ``buildout.cfg`` file. You
also need to add it to setup.py (And don't forget to add the version to
``versions.cfg`` as we run with ``allowed-picked-version`` set to false.)

If it is a development-only dependency (i.e. only needed for the test suite, or
for developers' convenience), simply running ``buildout`` like this will make
the necessary updates to ``versions.cfg``::

    $ ./bin/buildout -v buildout:allow-picked-versions=true


Adding new source files
=======================

When creating a new source file, a Python module or test for example,
always start with the appropriate template from the ``templates``
directory.


Database schema changes
=======================

MAAS uses South_ to manage changes to the database schema.

.. _South: http://south.aeracode.org

Be sure to have a look at `South's documentation`_ before you make any change.

.. _South's documentation: http://south.aeracode.org/docs/


Changing the schema
^^^^^^^^^^^^^^^^^^^

Once you've made a change to ``src/<application>/models.py`` you have to run
South's `schemamigration`_ command to create a migration file that will be
stored in ``src/<application>/migrations/``.

.. _schemamigration: http://south.aeracode.org/docs/commands.html#schemamigration

For instance if you're making changes to ``src/maasserver/models.py``, you
will need to run::

    $ ./bin/maas schemamigration maasserver --auto description_of_the_change

This will generate a migration module named
``src/maasserver/migrations/<auto_number>_description_of_the_change.py``.  Don't
forget to add that file to the project with::

    $ bzr add \
        src/maasserver/migrations/<auto_number>_description_of_the_change.py

To apply that migration, run::

    $ make syncdb


Performing data migration
^^^^^^^^^^^^^^^^^^^^^^^^^

If you need to perform data migration, very much in the same way, you will need
to run South's `datamigration`_ command.  For instance, if you want to perform
changes to the ``maasserver`` application, run::

    $ ./bin/maas datamigration maasserver --auto description_of_the_change

.. _datamigration: http://south.aeracode.org/docs/commands.html#datamigration

This will generate a migration module named
``src/maasserver/migrations/<auto_number>_description_of_the_change.py``.
You will need to edit that file and fill the ``forwards`` and ``backwards``
methods where data should be actually migrated. Again, don't forget to
add that file to the project::

    $ bzr add \
        src/maasserver/migrations/<auto_number>_description_of_the_change.py

Once the methods have been written, apply that migration with::

    $ make syncdb


Documentation
=============

Use `reST`_ with the `convention for headings as used in the Python
documentation`_.

.. _reST: http://sphinx.pocoo.org/rest.html

.. _convention for headings as used in the Python documentation:
  http://sphinx.pocoo.org/rest.html#sections


Acceptance tests
================

MAAS uses `Checkbox`_ to verify features are implemented according to
the spec. They can be found in the `official checkbox repository`_, in
the ``jobs/`` directory.

.. _Checkbox: https://wiki.ubuntu.com/Testing/Automation/Checkbox

.. _official checkbox repository:
  https://code.launchpad.net/~nskaggs/checkbox/checkbox-app-testing-qt

You need to install additional QT dependencies::

    $ sudo apt-get install libqt4-dev qt4-qmake

Get a copy of the checkbox repository::

    $ bzr branch lp:~nskaggs/checkbox/checkbox-app-testing-qt

To run them, move into the branch directory and run::

    $ bin/checbox-app-testing --whitelist-file=