1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
This builds maas ephemeral images from daily builds of cloud images.
Ie, it creates stuff that goes on
http://maas.ubuntu.com/images/ephemeral-v2/daily
from stuff on
http://cloud-images.ubuntu.com/daily
== setup ==
Run './system-setup' to set up the system.
This will install necessary dependencies.
the tools expect that the 'bin' dir is in $PATH.
export PATH=$PWD/bin:$PATH
== building images ==
You can build images and kernel in one of 2 ways
=== build a single release/arch combination ===
This will build output for a single release and architecture.
You must provide a url or file to use as the source image.
meph2-build arch release version source-image output.d
example:
$ burl=http://cloud-images.ubuntu.com/daily/server/
$ meph2-build amd64 xenial 20160101 \
$burl/xenial/current/xenial-server-cloudimg-amd64-root.tar.xz \
xenial-output.d/
This will build from source-image into output.d for 'arch' and 'release'.
The 'version' provided is a serial (YYYYMMDD such as 20160101).
source-image can be either a url or a local file of type
'.tar.gz', '-root.tar.gz', or '-root.tar.xz' or '.img'
In order to collect debian installer data you will need '--enable-di'.
=== syncing against cloud-images.ubuntu.com ===
This will pull the latest images down from cloud-images.ubuntu.com
and build maas images and kernel data all in one go. It only builds
if it notices that maas images are out of date with cloud-images.
To force a build, use '--target=force'
To run:
meph2-cloudimg-sync out.d
You can then add filters to the end, like 'release=trusty' or
'arch=amd64'. The resultant out.d/streams/v1/ will have information
copied from maas.ubuntu.com for all other products, but will only
create new content for products that match the provided filters.
If the maas ephemeral data is up to date with the latest cloud image daily
then meph2-cloudimg-sync will not do anything.
In order to 'rebuild' something. To create a new '20160101.1' even when
there is an existing '20160101' maas image and the latest cloud image
is 20160101, then pass '--rebuild=20160101:20160101.1'.
For faster debugging, you can disable the crawling of d-i kernel/initramfs
by passing '--disable-di'
=== debugging ===
To debug and use the fake 'maas-cloudimg2eph2', run like:
env MAAS_CLOUDIMG2EPH2=fake-maas-cloudimg2eph2 meph2-cloudimg-sync out.d
This will still download di-kernels and di-initramfs.
It will produce output that looks much like the final output, but
runs in seconds rather than minutes.
== streams management ==
Building as above will create out.d that has *metadata* describing
everything that is in the source stream and also the things that were
just built. data (images) will only be present for the newly built things.
So after building new data and metadata, you'll then insert the new data into
an existing stream. This option (meph2-util insert) is additive only.
=== insert data ===
Given output of a meph2-cloudimg-sync command above in build-output.d
and local daily stream directory daily-stream.d you can add the new data
with:
meph2-util insert build-output.d daily-stream.d
If you do not want to sign (create .gpg and .sjson files), then you will
have to use '--no-sign'.
You can use '--dry-run' to only report what would be done.
=== Promoting Builds ===
Promoting is done on a specific version of a product.
You promote a daily build to a released build with a given label.
Likely that label is 'release'.
# list the product-ids and version are available for promotion
$ sstream-query --output-format="%(product_name)s %(version_name)s" --max=1 \
daily.d/streams/v1/index.json arch=i386 | sort -u
com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-p 20140414
com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q 20140414
com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-r 20140414
com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-s 20140414
com.ubuntu.maas.daily:v2:boot:13.10:i386:hwe-s 20140416
com.ubuntu.maas.daily:v2:boot:14.04:i386:hwe-t 20140416.1
# promote the listed version
$ meph2-util promote --label=release \
daily.d/streams/v1/index.json releases.d/ 20140414 \
product_name=com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q
== cleaning old images ==
Cleaning is done in 2 parts:
* cleaning meta-data (removing entries from the products files)
* finding orphans: identify which files in the tree are no longer referenced.
* purging orphans: removing files that are known to have been orpaned for
a given amount of time (3 days).
=== cleaning meta-data ===
clean-md is the meph-util subcommand that allows you to keep only a given
number of builds. For example, to keep the latest 3 versions of every
product in the meta-data, you would run:
meph2-util clean-md --max=3 daily-sream.d/
This *only* modifies the meta-data, it does not delete files, use find-orphans
and reap-orphans for that.
There is a --dry-run and --no-sign arguments also.
=== finding orphans ===
finding orphans creates a json formated file that contains a entry for each
file that is no longer referenced, and a timestamp of "now" (when it was first
found to be orphaned).
Example usage:
The following will populate my-orphans.json. Subsequent runs will update
my-orphans.json with any new items, but will not update first-orphaned
timestamp.
$ meph2-util find-orphans my-orphans.json ./daily/ ./daily/streams/v1/*.json
More complex usage:
Its possible that 2 streams dirs could share a data directory, similar
to the 'pool' concept of apt for debian packages. For example, the following
directory tree layout (note, 'path' elements would have 'data' in them in
this example):
daily/
data -> ../data/
streams/v1/index.json
streams/v1/daily.json
released/
data -> ../data
streams/v1/index.json
streams/v1/released.json
data/
trusty/...
utopic/...
Then, we could run meph-util find-orphans with:
$ meph2-util find-orphans my-orphans.json ./data/ \
./daily/streams/v1/index.json released/streams/v1/index.json
That will look through files in data/ that are not referenced by path
elements in either the daily/streams/v1/ or released/streams/v1/ files.
There is no --dry-run for finding orphans as it is non-destructive.
=== cleaning / reaping orphans ===
Reaping orphans is what actually removes files. There is a '--dry-run' to
indicate what would be done.
its usage is fairly simple.
$ meph2-util reap-orphans --older 7d my-orphans.json ./data/
'--older' takes input like:
Ni
Where N is a number and i can be one of:
d: days
h: hours
m: minutes
s: seconds
The default is '3d'.
|