94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
1 |
# Continuous Integration and Deployment scripts
|
2 |
||
3 |
The Juju QA team uses a few scripts to build release and testing tools |
|
4 |
that are placed in private and public clouds. Juju provides several ways |
|
5 |
to find tools. The scripts automate and ensure that tools are published |
|
6 |
in a consistent manner. These scripts can be adapted to place specific |
|
7 |
tools into private clouds. |
|
8 |
||
9 |
Get a copy of all the scripts |
|
10 |
lp:/~juju-qa/juju-core/ci-cd-scripts2 |
|
11 |
||
12 |
The general process involves making a release tarball, making a package, |
|
13 |
making a tree of tools and metadata, and lastly publishing the tools. |
|
14 |
you can skip the tarball and package steps if you just want to publish |
|
15 |
the juju tools (AKA jujud, servers, agents). If you want to test a fix |
|
16 |
that is in the juju-core trunk branch, you can make your own release |
|
17 |
tarball and package. |
|
18 |
||
19 |
||
20 |
## Making a release tarball
|
|
21 |
||
22 |
The Juju QA team tests new revisions of the devel and stable juju-core |
|
23 |
branches. The script will create a release tarball without a local copy |
|
24 |
of juju-core or any of its dependencies. Anyone with Ubuntu, or possibly |
|
25 |
any posix system, can create a release tarball with just this script. |
|
26 |
||
27 |
Select a juju-core branch and a revision to create a tarball with |
|
28 |
juju-core and all its dependent libraries. This command for example |
|
350
by Curtis Hovey
Remove publication of 1.16. |
29 |
selects revision 1985 of lp:juju-core/1.18 |
30 |
make-release-tarball.bash 2262 lp:juju-core/1.18 |
|
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
31 |
|
350
by Curtis Hovey
Remove publication of 1.16. |
32 |
The example will create juju-core_1.18.0.tar.gz. |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
33 |
|
34 |
You can pass ‘-1’ as the revision to select tip. The release version is |
|
35 |
derived from the source code. If you are making your own release, with |
|
36 |
your own changes to the source code, you will need to update the |
|
37 |
major.minor.patch versions in the code: |
|
38 |
versions/version.go |
|
39 |
scripts/win-installer/setup.iss |
|
40 |
||
41 |
||
42 |
## Making a package
|
|
43 |
||
44 |
You can make a package from a release tarball. The script uses |
|
45 |
bzr-builddeb and a source package branch to create a either a binary or |
|
46 |
source package. This command will create a source package based on the |
|
47 |
Ubuntu juju-core source package branch. |
|
350
by Curtis Hovey
Remove publication of 1.16. |
48 |
make-package-with-tarball.bash stable ./juju-core_1.18.0.tar.gz 'Full Name <user@example.com>' |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
49 |
|
50 |
The first script argument is the intent, <testing|devel|stable>. It |
|
51 |
determines the base source package branch and whether to create a binary |
|
52 |
package. |
|
53 |
testing: use the devel package branch and create a binary package |
|
54 |
devel: use the devel package branch and create a source package |
|
55 |
stable: use the stable package branch and create a source package |
|
56 |
||
57 |
The devel and stable packaging branches are functionally the same at the |
|
58 |
moment. If juju-core deps or rules need to change, we will update the |
|
59 |
devel packaging branch first. |
|
60 |
||
61 |
You must provide a proper name and email address to use for the package |
|
62 |
changelog. The email address will also be used to select the identity to |
|
63 |
sign the source package. |
|
64 |
||
65 |
||
66 |
### Creating a binary package for testing
|
|
67 |
||
68 |
You can create a binary package using the testing mode of the packaging |
|
69 |
script |
|
350
by Curtis Hovey
Remove publication of 1.16. |
70 |
make-package-with-tarball.bash testing ./juju-core_1.18.0.tar.gz 'Full Name <user@example.com>' |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
71 |
|
72 |
||
73 |
## Assemble the juju tools
|
|
74 |
||
75 |
Every juju-core package contains a client and a server. The server is |
|
76 |
the “juju tool” that provides the state-server on the bootstrap node |
|
77 |
and the agent on the unit nodes. You can create the directory tree of |
|
78 |
tools and metadata for a new released by running: |
|
350
by Curtis Hovey
Remove publication of 1.16. |
79 |
assemble-public-tools.bash 1.18.0 new-tools/ |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
80 |
|
81 |
The script takes two arguments, a release version, and a destination |
|
82 |
directory. The new-tools/ directory has 3 or 4 subdirectories for the |
|
83 |
phases, debs/, tools/, juju-dist/, and juju-dist-testing/. The script |
|
84 |
processes packages, tools, and metadata in several phases: |
|
85 |
1. Collect all the existing public tools from AWS (in tools/)
|
|
86 |
2. Download all packages that match the first script argument
|
|
87 |
from several archives (to debs/) |
|
88 |
3. Extract the new tools from the debs and add them to existing tools (tools/)
|
|
89 |
4. Assemble the tools into the simple streams directories (juju-dist/)
|
|
90 |
||
91 |
If you pass a testing package instead of a release version, it will be |
|
92 |
used as the found debian package to extract the tool from. The simple |
|
93 |
stream data will be placed in juju-dist-testing/ instead of juju-dist/ |
|
94 |
to make a clear separation of streams that should not be used in |
|
95 |
production. You can pass “ignore” instead of a release version to skip |
|
96 |
the the package and extraction steps. |
|
97 |
||
98 |
||
99 |
### Assembling tools for a private cloud
|
|
100 |
||
101 |
If you want to publish a subset of tools, you can manipulate content of |
|
102 |
the new-tools/tools/ directory after the first run and rerun the script |
|
103 |
with the PRIVATE argument to create a new-tools/juju-dist/ directory |
|
104 |
with just the tools you will support in your cloud. |
|
105 |
assemble-public-tools.bash ignore new-tools/ PRIVATE |
|
106 |
||
350
by Curtis Hovey
Remove publication of 1.16. |
107 |
Thus to assemble the simple streams metadata and tools for 1.18.0, you |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
108 |
would run the script once to get all the tools. Then delete all the |
109 |
older versions in new-tools/tools. You might even delete the unsupported |
|
110 |
architectures such as i386 or arm. Then run the script with the PRIVATE |
|
350
by Curtis Hovey
Remove publication of 1.16. |
111 |
flag to get just the 1.18.0 tools for amd64. |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
112 |
|
113 |
You can copy your own deb packages to new-tools/debs after the first run |
|
114 |
and then run the script again to extract the tools. This would be a |
|
115 |
two-phased run of the script. Run the script once get all the released |
|
116 |
tools. Then add your own packages and tools and remove the unwanted |
|
117 |
tools. Run the script again with the PRIVATE argument to generate a |
|
118 |
new-tools/juju-dist/ dir of just what the private cloud wants. |
|
119 |
||
120 |
One warning about versions and directories. The script knows that |
|
350
by Curtis Hovey
Remove publication of 1.16. |
121 |
juju-core 1.18.x is the first juju to use simple streams, and the last |
122 |
juju to use tools in the public container. If your cloud is 1.18+ then |
|
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
123 |
there are no concerns -- everything will be in simple-streams format. If |
124 |
you are supporting Juju 1.10 to 1.14.1, you must include a version of |
|
350
by Curtis Hovey
Remove publication of 1.16. |
125 |
1.18 to ensure the old deployments can find the newer versions of juju. |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
126 |
|
127 |
||
128 |
## Publishing tools to a cloud
|
|
129 |
||
130 |
The publish-public-tools.bash script is used to upload the juju-dist |
|
131 |
directory tree to each certified public cloud. It cannot be used by |
|
132 |
anyone other than the juju release staff. Anyone can use this script to |
|
133 |
develop their own script. Consider the script to be an example of how to |
|
134 |
publish to openstack swift, aws s3, and azure storage, for both release |
|
135 |
and testing purposes. |
|
136 |
||
137 |
Setup a user in your private cloud that will own the public container |
|
138 |
that you will publish the tools to. The user’s public container is the |
|
139 |
base location used by all juju clients. The container might also be |
|
140 |
called the public bucket in some clouds. Create these paths if your |
|
141 |
cloud has subcontainers or directories. The paths must be readable by |
|
142 |
everyone, but only the owner can make changes: |
|
143 |
juju-dist/ |
|
144 |
juju-dist/tools/ |
|
145 |
juju-dist/tools/releases/ |
|
146 |
juju-dist/tools/streams/ |
|
147 |
juju-dist/tools/streams/v1/ |
|
148 |
||
149 |
The publish_to_canonistack() example shows how upload tools to the |
|
150 |
public swift container. The swift client is using these variables |
|
151 |
defined in the env: OS_USERNAME, OS_TENANT_NAME, OS_PASSWORD, |
|
152 |
OS_AUTH_URL, OS_REGION_NAME, which selects the public container. It |
|
153 |
uploads tools to the path that matches to local path (releases/). It |
|
154 |
uploads metadata to a path that matches the local path (v1/), then it |
|
350
by Curtis Hovey
Remove publication of 1.16. |
155 |
uploads any 1.18 tools I can find to the historic path (tools/). |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
156 |
|
157 |
||
158 |
## Configure clients to use the published tools
|
|
159 |
||
160 |
The juju client is designed to “just work” with public clouds; it knows |
|
161 |
how to find public tools from many sources. You can configure the juju |
|
162 |
client to look for tools at the location you uploaded them to in your |
|
163 |
private cloud. From the previous step, the location is the |
|
164 |
juju-dist/tools/ path in the public container |
|
165 |
tools-url: https://<swift.example.com>/v1/<public-container>/juju-dist/tools |
|
166 |
||
167 |
*Note:* In old configs, the public-bucket-url key was used to find tools.
|
|
350
by Curtis Hovey
Remove publication of 1.16. |
168 |
Juju 1.18 will assume that tools are located in /juju-dist/tools path of |
94
by Curtis Hovey
Added a readme to explain the scripts that non QA staff can use. |
169 |
the public-bucket-url when tools-url is not defined. Update the configs |
170 |
with tools-url to explicitly set where tools will be found. |
|
171 |
||
172 |
*Note:* in future configs, starting with Juju 1.18, use the
|
|
173 |
tools-metadata-url key to specify the location you uploaded the tools to |
|
174 |
in your private cloud. |