1
Arriving in 0.7.2 is a new way to handle dictionary merging in cloud-init.
7
This was done because it has been a common feature request that there be a
8
way to specify how cloud-config yaml "dictionaries" are merged together when
9
there are multiple yamls to merge together (say when performing an #include).
11
Since previously the merging algorithm was very simple and would only overwrite
12
and not append lists, or strings, and so on it was decided to create a new and
13
improved way to merge dictionaries (and there contained objects) together in a
14
way that is customizable, thus allowing for users who provide cloud-config data
15
to determine exactly how there objects will be merged.
29
The previous way of merging the following 2 objects would result in a final
30
cloud-config object that contains the following.
32
#cloud-config (merged)
37
Typically this is not what users want, instead they would likely prefer:
39
#cloud-config (merged)
46
This way makes it easier to combine the various cloud-config objects you have
47
into a more useful list, thus reducing duplication that would have had to
48
occur in the previous method to accomplish the same result.
53
Since the above merging algorithm may not always be the desired merging
54
algorithm (like how the merging algorithm in < 0.7.2 was not always the preferred
55
one) the concept of customizing how merging can be done was introduced through
56
a new concept call 'merge classes'.
58
A merge class is a class defintion which provides functions that can be used
59
to merge a given type with another given type.
61
An example of one of these merging classes is the following:
64
def __init__(self, merger, opts):
66
self._overwrite = 'overwrite' in opts
68
# This merging algorithm will attempt to merge with
69
# another dictionary, on encountering any other type of object
70
# it will not merge with said object, but will instead return
73
# On encountering a dictionary, it will create a new dictionary
74
# composed of the original and the one to merge with, if 'overwrite'
75
# is enabled then keys that exist in the original will be overwritten
76
# by keys in the one to merge with (and associated values). Otherwise
77
# if not in overwrite mode the 2 conflicting keys themselves will
79
def _on_dict(self, value, merge_with):
80
if not isinstance(merge_with, (dict)):
83
for (k, v) in merge_with.items():
85
if not self._overwrite:
86
merged[k] = self._merger.merge(merged[k], v)
93
As you can see there is a '_on_dict' method here that will be given a source value
94
and a value to merge with. The result will be the merged object. This code itself
95
is called by another merging class which 'directs' the merging to happen by
96
analyzing the types of the objects to merge and attempting to find a know object
97
that will merge that type. I will avoid pasting that here, but it can be found
98
in the mergers/__init__.py file (see LookupMerger and UnknownMerger).
100
So following the typical cloud-init way of allowing source code to be downloaded
101
and used dynamically, it is possible for users to inject there own merging files
102
to handle specific types of merging as they choose (the basic ones included will
103
handle lists, dicts, and strings). Note how each merge can have options associated
104
with it which affect how the merging is performed, for example a dictionary merger
105
can be told to overwrite instead of attempt to merge, or a string merger can be
106
told to append strings instead of discarding other strings to merge with.
111
There are a few ways to activate the merging algorithms, and to customize them
114
1. The first way involves the usage of MIME messages in cloud-init to specify
115
multipart documents (this is one way in which multiple cloud-config is joined
116
together into a single cloud-config). Two new headers are looked for, both
117
of which can define the way merging is done (the first header to exist wins).
118
These new headers (in lookup order) are 'Merge-Type' and 'X-Merge-Type'. The value
119
should be a string which will satisfy the new merging format defintion (see
120
below for this format).
121
2. The second way is actually specifying the merge-type in the body of the
122
cloud-config dictionary. There are 2 ways to specify this, either as a string
123
or as a dictionary (see format below). The keys that are looked up for this
124
definition are the following (in order), 'merge_how', 'merge_type'.
128
The string format that is expected is the following.
130
"classname(option1,option2)+classname2(option3,option4)" (and so on)
132
The class name there will be connected to class names used when looking for the
133
class that can be used to merge and options provided will be given to the class
134
on construction of that class.
136
For example, the default string that is used when none is provided is the following:
138
"list(extend)+dict()+str(append)"
142
In cases where a dictionary can be used to specify the same information as the
143
string format (ie option #2 of above) it can be used, for example.
153
This would be the equivalent format for default string format but in dictionary
154
form instead of string form.
156
Specifying multiple types and its effect
157
----------------------------------------
159
Now you may be asking yourself, if I specify a merge-type header or dictionary
160
for every cloud-config that I provide, what exactly happens?
162
The answer is that when merging, a stack of 'merging classes' is kept, the
163
first one on that stack is the default merging classes, this set of mergers
164
will be used when the first cloud-config is merged with the initial empty
165
cloud-config dictionary. If the cloud-config that was just merged provided a
166
set of merging classes (via the above formats) then those merging classes will
167
be pushed onto the stack. Now if there is a second cloud-config to be merged then
168
the merging classes from the cloud-config before the first will be used (not the
169
default) and so on. This way a cloud-config can decide how it will merge with a
170
cloud-config dictionary coming after it.
175
The default merging algorithm for merging conf.d yaml files (which form a initial
176
yaml config for cloud-init) was also changed to use this mechanism so its full
177
benefits (and customization) can also be used there as well. Other places that
178
used the previous merging are also similar now extensible (metadata merging for
added
removed
doc/merging.txt
