~andrewjbeach/juju-ci-tools/make-local-patcher

# Continuous Integration and Deployment scripts

The Juju QA team uses a few scripts to build release and testing tools
that are placed in private and public clouds. Juju provides several ways
to find tools. The scripts automate and ensure that tools are published
in a consistent manner. These scripts can be adapted to place specific
tools into private clouds.

Get a copy of all the scripts
    lp:/~juju-qa/juju-core/ci-cd-scripts2

The general process involves making a release tarball, making a package,
making a tree of tools and metadata, and lastly publishing the tools.
you can skip the tarball and package steps if you just want to publish
the juju tools (AKA jujud, servers, agents). If you want to test a fix
that is in the juju-core trunk branch, you can make your own release
tarball and package.


## Making a release tarball

The Juju QA team tests new revisions of the devel and stable juju-core
branches. The script will create a release tarball without a local copy
of juju-core or any of its dependencies. Anyone with Ubuntu, or possibly
any posix system, can create a release tarball with just this script.

Select a juju-core branch and a revision to create a tarball with
juju-core and all its dependent libraries. This command for example
selects revision 1985 of lp:juju-core/1.18
	make-release-tarball.bash 2262 lp:juju-core/1.18

The example will create juju-core_1.18.0.tar.gz.

You can pass ‘-1’ as the revision to select tip. The release version is
derived from the source code. If you are making your own release, with
your own changes to the source code, you will need to update the
major.minor.patch versions in the code:
	versions/version.go
	scripts/win-installer/setup.iss


## Making a package

You can make a package from a release tarball. The script uses
bzr-builddeb and a source package branch to create a either a binary or
source package. This command will create a source package based on the
Ubuntu juju-core source package branch.
	make-package-with-tarball.bash stable ./juju-core_1.18.0.tar.gz 'Full Name <user@example.com>'

The first script argument is the intent, <testing|devel|stable>. It
determines the base source package branch and whether to create a binary
package.
	testing: use the devel package  branch and create a binary package
	devel: use the devel package  branch and create a source package
	stable: use the stable package branch and create a source package

The devel and stable packaging branches are functionally the same at the
moment. If juju-core deps or rules need to change, we will update the
devel packaging branch first.

You must provide a proper name and email address to use for the package
changelog. The email address will also be used to select the identity to
sign the source package.


### Creating a binary package for testing

You can create a binary package using the testing mode of the packaging
script
    make-package-with-tarball.bash testing ./juju-core_1.18.0.tar.gz 'Full Name <user@example.com>'


## Assemble the juju tools

Every juju-core package contains a client and a server. The server is
the “juju tool”  that provides the state-server on the bootstrap node
and the agent on the unit nodes. You can create the directory tree of
tools and metadata for a new released by running:
	assemble-public-tools.bash 1.18.0 new-tools/

The script takes two arguments, a release version, and a destination
directory. The new-tools/ directory has 3 or 4 subdirectories for the
phases, debs/, tools/, juju-dist/, and juju-dist-testing/. The script
processes packages, tools, and metadata in several phases:
	1. Collect all the existing public tools from AWS (in tools/)
	2. Download all packages that match the first script argument
       from several archives (to debs/)
	3. Extract the new tools from the debs and add them to existing tools (tools/)
	4. Assemble the tools into the simple streams directories (juju-dist/)

If you pass a testing package instead of a release version, it will be
used as the found debian package to extract the tool from. The simple
stream data will be placed in juju-dist-testing/ instead of juju-dist/
to make a clear separation of streams that should not be used in
production. You can pass “ignore” instead of a release version to skip
the the package and extraction steps.


### Assembling tools for a private cloud

If you want to publish a subset of tools, you can manipulate content of
the new-tools/tools/ directory after the first run and rerun the script
with the PRIVATE argument to create a new-tools/juju-dist/ directory
with just the tools you will support in your cloud.
	assemble-public-tools.bash ignore new-tools/ PRIVATE

Thus to assemble the simple streams metadata and tools for 1.18.0, you
would run the script once to get all the tools. Then delete all the
older versions in new-tools/tools. You might even delete the unsupported
architectures such as i386 or arm. Then run the script with the PRIVATE
flag to get just the 1.18.0 tools for amd64.

You can copy your own deb packages to new-tools/debs after the first run
and then run the script again to extract the tools. This would be a
two-phased run of the  script. Run the script once  get all the released
tools.  Then add your own packages and tools and remove the unwanted
tools. Run the script again with the PRIVATE argument to generate a
new-tools/juju-dist/ dir of just what the private cloud wants.

One warning about versions and directories. The script knows that
juju-core 1.18.x is the first juju to use simple streams, and the last
juju to use tools in the public container. If your cloud is 1.18+ then
there are no concerns -- everything will be in simple-streams format. If
you are supporting Juju 1.10 to 1.14.1, you must include a version of
1.18 to ensure the old deployments can find the newer versions of juju.


## Publishing tools to a cloud

The publish-public-tools.bash script is used to upload the juju-dist
directory tree to each certified public cloud. It cannot be used by
anyone other than the juju release staff. Anyone can use this script to
develop their own script. Consider the script to be an example of how to
publish to openstack swift, aws s3, and azure storage, for both release
and testing purposes.

Setup a user in your private cloud that will own the public container
that you will publish the tools to. The user’s public container is the
base location used by all juju clients. The container might also be
called the public bucket in some clouds. Create these paths if your
cloud has subcontainers or directories. The paths must be readable by
everyone, but only the owner can make changes:
	juju-dist/
	juju-dist/tools/
	juju-dist/tools/releases/
	juju-dist/tools/streams/
	juju-dist/tools/streams/v1/

The publish_to_canonistack() example shows how upload tools to the
public swift container. The swift client is using these variables
defined in the env: OS_USERNAME, OS_TENANT_NAME, OS_PASSWORD,
OS_AUTH_URL, OS_REGION_NAME, which selects the public container. It
uploads tools to the path that matches to local path (releases/). It
uploads metadata to a path that matches the local path (v1/), then it
uploads any 1.18 tools I can find to the historic path (tools/).


## Configure clients to use the published tools

The juju client is designed to “just work” with public clouds; it knows
how to find public tools from many sources. You can configure the juju
client to look for tools at the location you uploaded them to in  your
private cloud.  From the previous step, the location is the
juju-dist/tools/ path in the public container
	tools-url: https://<swift.example.com>/v1/<public-container>/juju-dist/tools

*Note:* In old configs, the public-bucket-url key was used to find tools.
Juju 1.18 will assume that tools are located in /juju-dist/tools path of
the public-bucket-url when tools-url is not defined. Update the configs
with tools-url to explicitly set where tools will be found.

*Note:* in future configs, starting with Juju 1.18, use the
tools-metadata-url key to specify the location you uploaded the tools to
in your private cloud.