5
The SQL database schema will over time require upgrading to support new
6
features. This is supported via schema migration.
8
Migrations are embodied in individual Python classes, which themselves may
9
load SQL into the database. The naming scheme for migration files is:
11
mm_YYYYMMDDHHMMSS_comment.py
13
where `YYYYMMDDHHMMSS` is a required numeric year, month, day, hour, minute,
14
and second specifier providing unique ordering for processing. Only this
15
component of the file name is used to determine the ordering. The prefix is
16
required due to Python module naming requirements, but it is actually
17
ignored. `mm_` is reserved for Mailman's own use.
19
The optional `comment` part of the file name can be used as a short
20
description for the migration, although comments and docstrings in the
21
migration files should be used for more detailed descriptions.
23
Migrations are applied automatically when Mailman starts up, but can also be
24
applied at any time by calling in the API directly. Once applied, a
25
migration's version string is registered so it will not be applied again.
27
We see that the base migration, as well as subsequent standard migrations, are
30
>>> from mailman.model.version import Version
31
>>> results = config.db.store.find(Version, component='schema')
34
>>> versions = sorted(result.version for result in results)
35
>>> for version in versions:
46
Migrations can be loaded at any time, and can be found in the migrations path
47
specified in the configuration file.
49
.. Create a temporary directory for the migrations::
51
>>> import os, sys, tempfile
52
>>> tempdir = tempfile.mkdtemp()
53
>>> path = os.path.join(tempdir, 'migrations')
55
>>> sys.path.append(tempdir)
56
>>> config.push('migrations', """
58
... migrations_path: migrations
61
.. Clean this up at the end of the doctest.
64
... from mailman.config import config
65
... config.pop('migrations')
66
... shutil.rmtree(tempdir)
67
>>> cleanups.append(cleanup)
69
Here is an example migrations module. The key part of this interface is the
70
``upgrade()`` method, which takes four arguments:
72
* `database` - The database class, as derived from `StormBaseDatabase`
73
* `store` - The Storm `Store` object.
74
* `version` - The version string as derived from the migrations module's file
75
name. This will include only the `YYYYMMDDHHMMSS` string.
76
* `module_path` - The dotted module path to the migrations module, suitable
77
for lookup in `sys.modules`.
79
This migration module just adds a marker to the `version` table.
81
>>> with open(os.path.join(path, '__init__.py'), 'w') as fp:
83
>>> with open(os.path.join(path, 'mm_20159999000000.py'), 'w') as fp:
85
... from __future__ import unicode_literals
86
... from mailman.model.version import Version
87
... def upgrade(database, store, version, module_path):
88
... v = Version(component='test', version=version)
90
... database.load_schema(store, version, None, module_path)
93
This will load the new migration, since it hasn't been loaded before.
95
>>> config.db.load_migrations()
96
>>> results = config.db.store.find(Version, component='schema')
97
>>> for result in sorted(result.version for result in results):
104
>>> test = config.db.store.find(Version, component='test').one()
105
>>> print(test.version)
108
Migrations will only be loaded once.
110
>>> with open(os.path.join(path, 'mm_20159999000001.py'), 'w') as fp:
112
... from __future__ import unicode_literals
113
... from mailman.model.version import Version
115
... def upgrade(database, store, version, module_path):
117
... # Pad enough zeros on the left to reach 14 characters wide.
118
... marker = '{0:=#014d}'.format(_marker)
120
... v = Version(component='test', version=marker)
122
... database.load_schema(store, version, None, module_path)
125
The first time we load this new migration, we'll get the 801 marker.
127
>>> config.db.load_migrations()
128
>>> results = config.db.store.find(Version, component='schema')
129
>>> for result in sorted(result.version for result in results):
137
>>> test = config.db.store.find(Version, component='test')
138
>>> for marker in sorted(marker.version for marker in test):
143
We do not get an 802 marker because the migration has already been loaded.
145
>>> config.db.load_migrations()
146
>>> results = config.db.store.find(Version, component='schema')
147
>>> for result in sorted(result.version for result in results):
155
>>> test = config.db.store.find(Version, component='test')
156
>>> for marker in sorted(marker.version for marker in test):
165
It's possible (mostly for testing purposes) to only do a partial upgrade, by
166
providing a timestamp to `load_migrations()`. To demonstrate this, we add two
167
additional migrations, intended to be applied in sequential order.
169
>>> from shutil import copyfile
170
>>> from mailman.testing.helpers import chdir
171
>>> with chdir(path):
172
... copyfile('mm_20159999000000.py', 'mm_20159999000002.py')
173
... copyfile('mm_20159999000000.py', 'mm_20159999000003.py')
174
... copyfile('mm_20159999000000.py', 'mm_20159999000004.py')
176
Now, only migrate to the ...03 timestamp.
178
>>> config.db.load_migrations('20159999000003')
180
You'll notice that the ...04 version is not present.
182
>>> results = config.db.store.find(Version, component='schema')
183
>>> for result in sorted(result.version for result in results):
196
Because the Version table holds schema migration data, it will not be
197
cleaned up by the standard test suite. This is generally not a problem
198
for SQLite since each test gets a new database file, but for PostgreSQL,
199
this will cause migration.rst to fail on subsequent runs. So let's just
200
clean up the database explicitly.
202
>>> if config.db.TAG != 'sqlite':
203
... results = config.db.store.execute("""
204
... DELETE FROM version WHERE version.version >= '201299990000'
205
... OR version.component = 'test';
207
... config.db.commit()
added
removed
src/mailman/database/docs/migration.rst-skip
