~justin-fathomdb/nova/justinsb-openstack-api-volumes

<?xml version="1.0" encoding="utf-8"?><!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">

<head>

<title>Twisted Documentation: Upgrading Applications</title>

</head>

<h1 class="title">Upgrading Applications</h1>

<div class="toc"><ol><li><a href="#auto0">Basic Persistence: Application and .tap files</a></li><li><a href="#auto1">Versioned: New Code Meets Old Data</a></li><li><a href="#auto2">Rebuild: Loading New Code Without Restarting</a></li></ol></div>

Applications must frequently deal with data that lives longer than the

programs that create it. Sometimes the structure of that data changes over

time, but new versions of a program must be able to accomodate data created

by an older version. These versions may change very quickly, especially

during development of new code. Sometimes different versions of the same

program are running at the same time, sharing data across a network

connection. These situations all result in a need for a way to upgrade data

structures.

<h2>Basic Persistence: Application and .tap files<a name="auto0"/></h2>

Simple object persistence (using <code>pickle</code> or

<code>jelly</code>) provides the fundamental <q>save the object to disk</q>

functionality at application shutdown. If you use the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.application.service.Application.html" title="twisted.application.service.Application">Application</a></code> object, every object

referenced by your Application will be saved into the

<code>-shutdown.tap</code> file when the program terminates. When you use

<code>twistd</code> to launch that new .tap file, the Application object

will be restored along with all of its referenced data.

This provides a simple way to have data outlive any particular invocation

of your program: simply store it as an attribute of the Application. Note

that all Services are referenced by the Application, so their attributes

will be stored as well. Ports that have been bound with listenTCP (and the

like) are also remembered, and the sockets are created at startup time (when

<code>Application.run</code> is called).

To influence the way that the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.application.service.Application.html" title="twisted.application.service.Application">Application</a></code> is persisted, you can adapt

it to <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.sob.IPersistable.html" title="twisted.persisted.sob.IPersistable">twisted.persisted.sob.IPersistable</a></code> and use

the <code class="python">setStyle(style)</code> method with

a string like <q>pickle</q> or <q>source</q>. These use different serializers (and different

extensions: <q>.tap</q> and <q>.tas</q> respectively) for the

saved Application.

You can manually cause the application to be saved by calling its

<code>.save</code> method (on the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.sob.IPersistable.html" title="twisted.persisted.sob.IPersistable">twisted.persisted.sob.IPersistable</a></code>

adapted object).

<h2>Versioned: New Code Meets Old Data<a name="auto1"/></h2>

So suppose you're running version 1 of some application, and you want to

upgrade to version 2. You shut down the program, giving you a .tap file that

you could restore with twistd to get back to the same state that you had

before. The upgrade process is to then install the new version of the

application, and then use twistd to launch the saved .tap file. The old data

will be loaded into classes created with the new code, and now you'll have a

program running with the new behavior but the old data.

But what about the data structures that have changed? Since these

structures are really just pickled class instances, the real question is

what about the class definitions that have changed? Changes to class methods

are easy: nothing about them is saved in the .tap file. The issue is when

the data attributes of a instance are added, removed, or their format is

changed.

Twisted provides a mechanism called <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.styles.Versioned.html" title="twisted.persisted.styles.Versioned">Versioned</a></code> to ease these upgrades.

Each version of the data structure (i.e. each version of the class) gets a

version number. This number must change every time you add or remove a data

attribute to the class. It must also change every time you modify one of

those data attributes: for example, if you use a string in one version and

an integer in another, those versions must have different version numbers.

The version number is defined in a class attribute named

<code>persistenceVersion</code>. This is an integer which will be stored in

the .tap file along with the rest of the instance state. When the object is

unserialized, the saved persistenceVersion is compared against the current

class's value, and if they differ, special upgrade methods are called. These

methods are named <code>upgradeToVersionNN</code>, and there must be one for

each intermediate version. These methods are expected to manipulate the

instance's state from the previous version's format into that of the new

version.

To use this, simply have your class inherit from <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.styles.Versioned.html" title="twisted.persisted.styles.Versioned">Versioned</a></code>. You don't have to do this

from the very beginning of time: all objects have an implicit version number

of <q>0</q> when they don't inherit from Versioned. So when you first make

an incompatible data-format change to your class, add Versioned to the

inheritance list, and add an <code>upgradeToVersion1</code> method.

For example, suppose the first version of our class saves an integer

which measures the size of a line. We release this as version 1.0 of our

neat application:

<pre class="python">1

class Thing:

100

def __init__(self, length):

101

self.length = length

102

</pre>

103

104

Then we fix some bugs elsewhere, and release versions 1.1 and 1.2 of the

105

application. Later, we decide that we should add some units to the length,

106

so that people can refer to it in inches or meters. Version 1.3 is shipped

107

with the following code:

108

109

<pre class="python">1

110

111

112

113

114

115

class Thing(Versioned):

116

persistenceVersion = 1

117

def __init__(self, length, units):

118

self.length = "%d %s" % (length, units)

119

def upgradeToVersion1(self):

120

self.length = "%d inches" % self.length

121

</pre>

122

123

Note that we must make an assumption about what the previous value meant:

124

in this case, we assume the number was in inches.

125

126

1.4 and 1.5 are shipped with other changes. Then in version 1.6 we decide

127

that saving the two values as a string was foolish and that it would be

128

better to save the number and the string separately, using a tuple. We ship

129

1.6 with the following:

130

131

<pre class="python">1

132

133

134

135

136

137

138

139

140

class Thing(Versioned):

141

persistenceVersion = 2

142

143

self.length = (length, units)

144

def upgradeToVersion1(self):

145

146

def upgradeToVersion2(self):

147

(length, units) = self.length.split()

148

self.length = (length, units)

149

</pre>

150

151

Note that we must provide both <code>upgradeToVersion1</code>

152

and <code>upgradeToVersion2</code>. We have to assume that the

153

saved .tap files which will be provided to this class come from a random

154

assortment of old versions: we must be prepared to accept anything ever

155

saved by a released version of our application.

156

157

Finally, version 2.0 adds multiple dimensions. Instead of merely

158

recording the length of a line, it records the size of an N-dimensional

159

rectangular solid. For backwards compatiblity, all 1.X version of the

160

program are assumed to be dealing with a 1-dimensional line. We change the

161

name of the attribute from <code>.length</code> to <code>.size</code> to

162

reflect the new meaning.

163

164

<pre class="python"> 1

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

class Thing(Versioned):

180

persistenceVersion = 3

181

def __init__(self, dimensions):

182

# dimensions is a list of tuples, each is (length, units)

183

self.size = dimensions

184

self.name = ["line", "square", "cube", "hypercube"][len(dimensions)]

185

def upgradeToVersion1(self):

186

187

def upgradeToVersion2(self):

188

189

self.length = (length, units)

190

def upgradeToVersion3(self):

191

self.size = [self.length]

192

del self.length

193

self.name = "line"

194

</pre>

195

196

If a .tap file from the earliest version of our program were to be loaded

197

by the latest code, the following sequence would occur for each Thing

198

instance contained inside:

199

200

<ol>

201

202

<li>An instance of Thing would be created, with a __dict__ that contained

203

a single attribute <code>.size</code>, which was an integer, like

204

205

206

<li><code class="python">self.upgradeToVersion1()</code> would be called,

207

changing <code>self.size</code> into a string, like <q>5 inches</q>.</li>

208

209

<li><code class="python">self.upgradeToVersion2()</code> would be called,

210

changing <code>self.size</code> into a tuple, like (5,

211

<q>inches</q>).</li>

212

213

<li>Finally, <code class="python">self.upgradeToVersion3()</code> would be

214

called, creating <code>self.size</code> as a list holding a single

215

dimension, like [(5, <q>inches</q>)]. The old <code>.length</code>

216

attribute is deleted, and a new <code>.name</code> is created with the

217

type of shape this instance represents (<q>line</q>).</li>

218

219

</ol>

220

221

Some hints for the <code>upgradeVersion</code> methods:

222

223

<ul>

224

225

<li>They must do everything the <code>__init__</code> method would have

226

done, as well as any methods that might have been called during the

227

lifetime of the object.</li>

228

229

<li>If the class has (or used to have) methods which can add attributes

230

that weren't created in <code>__init__</code>, then the saved object may

231

have a haphazard subset of those attributes, depending upon which methods

232

were called. The upgradeVersion methods must be prepared to deal with

233

this. <code>hasattr</code> and <code>.get</code> may be useful.</li>

234

235

<li>Once you have released a class with a given

236

<code>upgradeVersion</code> method, you should never change that method.

237

(assuming you care about infinite backwards compatibility).</li>

238

239

<li>You must add a new <code>upgradeVersion</code> method (and bump the

240

persistenceVersion value) for each and every release that has a different

241

set of data attributes than the previous release.</li>

242

243

<li><code>Versioned</code> works by providing <code>__setstate__</code>

244

and <code>__getstate__</code> methods. You probably don't want to override

245

these methods without being very careful to call the Versioned versions at

246

exactly the right time. It also requires a <code>doUpgrade</code> function

247

to be called after all the objects are loaded. This is done automatically

248

by <code>Application.run</code>.</li>

249

250

<li>Depending upon how they are serialized, <code>Versioned</code> objects

251

can probably be sent across a network connection, and the upgrade process

252

can be made to occur upon receipt. (You'll want to look at the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.styles.requireUpgrade.html" title="twisted.persisted.styles.requireUpgrade">requireUpgrade</a></code>

253

function). This might be useful in providing compability with an older

254

peer. Note, however, that <code>Versioned</code> does not let you go

255

backwards in time; there is no <code>downgradeVersionNN</code> method.

256

This means it is probably only useful for compatibility in one direction:

257

the newer-to-older direction must still be explicitly handled by the

258

application.</li>

259

260

<li>In general, backwards compatibility is handled by pretending that the

261

old code was restricting itself to a narrow subset of the capabilities of

262

the new code. The job of the upgrade routines is then to translate the old

263

representation into a new one.</li>

264

265

</ul>

266

267

For more information, look at the doc strings for <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.styles.Versioned.html" title="twisted.persisted.styles.Versioned">styles.Versioned</a></code>, as well as the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.app.Application.html" title="twisted.internet.app.Application">app.Application</a></code> class and the <a href="application.html" shape="rect">Application HOWTO</a>.

268

269

270

<h2>Rebuild: Loading New Code Without Restarting<a name="auto2"/></h2>

271

272

<code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/Versioned.html" title="Versioned">Versioned</a></code> is good for handling changes between

273

released versions of your program, where the application state is saved on

274

disk during the upgrade. But while you are developing that code, you often

275

want to change the behavior of the running program, without the

276

slowdown of saving everything out to disk, shutting down, and restarting.

277

Sometimes it will be difficult or time-consuming to get back to the previous

278

state: the running program could include ephemeral objects (like open

279

sockets) which cannot be persisted.

280

281

<code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.python.rebuild.html" title="twisted.python.rebuild">twisted.python.rebuild</a></code> provides a function

282

called <code>rebuild</code> which helps smooth this cycle. It allows objects

283

in a running program to be upgraded to a new version of the code without

284

shutting down.

285

286

To use it, simply call <code class="python">rebuild</code> on the module

287

that holds the classes you want to be upgraded. Through deep <code class="python">gc</code> magic, all instances of classes in that module will

288

be located and upgraded.

289

290

Typically, this is done in response to a privileged command sent over a

291

network connection. The usual development cycle is to start the server, get

292

it into an interesting state, see a problem, edit the class definition, then

293

push the <q>rebuild yourself</q> button. That <q>button</q> could be a magic

294

web page which, when requested, runs <code class="python">rebuild(mymodule)</code>, or a special IRC command, or

295

perhaps just a socket that listens for connections and accepts a password to

296

trigger the rebuild. (You want this to be a privileged operation to prevent

297

someone from making your server do a rebuild while you're in the middle of

298

editing the code).

299

300

A few useful notes about the rebuild process:

301

302

<ul>

303

<li>If the module has a top-level attribute named

304

<code>ALLOW_TWISTED_REBUILD</code>, this attribute must evaluate to True.

305

Should it be false, the rebuild attempt will raise an exception.</li>

306

307

<li>Adapters (from <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.python.components.html" title="twisted.python.components">twisted.python.components</a></code>) use

308

top-level registration function calls. These are handled correctly during

309

rebuilds, and the usual duplicate registration errors are not raised.</li>

310

311

<li>Rebuilds may be slow: every single object known to the interpreter

312

must be examined to see if it is one of the classes being changed.</li>

313

</ul>

314

315

Finally, note that <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.python.rebuild.rebuild.html" title="twisted.python.rebuild.rebuild">rebuild</a></code> cannot currently be

316

mixed with <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.persisted.styles.Versioned.html" title="twisted.persisted.styles.Versioned">Versioned</a></code>. <code>rebuild</code> does

317

not run any of the classes' methods, whereas <code>Versioned</code> works by

318

running <code>__setstate__</code> during the load process and

319

<code>doUpgrade</code> afterwards. This means <code>rebuild</code> can only

320

be used to process upgrades that do not change the data attributes of any of

321

the involved classes. Any time attributes are added or removed, the program

322

must be shut down, persisted, and restarted, with upgradeToVersionNN methods

323

used to handle the attributes. (this may change in the future, but for now

324

the implementation is easier and more reliable with this restriction).

325

326

</div>

327

328

<a href="index.html">Index</a>

329

Version: 10.0.0

330

</body>

331

</html>

b'\\ No newline at end of file'

Older »