2
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
3
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
<html xmlns="http://www.w3.org/1999/xhtml">
7
<title>The Twisted Network Framework</title>
11
<p><em><strong>Note:</strong> This document is relevant for the
12
version of Twisted that were current previous to <a
13
href="http://www.python10.com">IPC10</a>. Even at the time of
14
its release, <a href="ipc10errata.html">there were errata
15
issued</a> to make it current. It is remaining unaltered for
16
historical purposes but it is no longer accurate.</em></p>
18
<h1>The Twisted Network Framework</h1>
21
href="mailto:m@moshez.org">m@moshez.org</a></h6>
23
<h6>Glyph Lefkowitz <a
24
href="mailto:glyph@twistedmatrix.com">glyph@twistedmatrix.com</a></h6>
28
<p>Twisted is a framework for writing asynchronous,
29
event-driven networked programs in Python -- both clients and
30
servers. In addition to abstractions for low-level system calls
31
like <code>select(2)</code> and <code>socket(2)</code>, it also
32
includes a large number of utility functions and classes, which
33
make writing new servers easy. Twisted includes support for
34
popular network protocols like HTTP and SMTP, support for GUI
35
frameworks like <code>GTK+</code>/<code>GNOME</code> and
36
<code>Tk</code> and many other classes designed to make network
37
programs easy. Whenever possible, Twisted uses Python's
38
introspection facilities to save the client programmer as much
39
work as possible. Even though Twisted is still work in
40
progress, it is already usable for production systems -- it can
41
be used to bring up a Web server, a mail server or an IRC
42
server in a matter of minutes, and require almost no
45
<p><strong>Keywords:</strong> internet, network, framework,
46
event-based, asynchronous</p>
50
<p>Python lends itself to writing frameworks. Python has a
51
simple class model, which facilitates inheritance. It has
52
dynamic typing, which means code needs to assume less. Python
53
also has built-in memory management, which means application
54
code does not need to track ownership. Thus, when writing a new
55
application, a programmer often finds himself writing a
56
framework to make writing this kind of application easier.
57
Twisted evolved from the need to write high-performance
58
interoperable servers in Python, and making them easy to use
59
(and difficult to use incorrectly).</p>
61
<p>There are three ways to write network programs:</p>
64
<li>Handle each connection in a separate process</li>
66
<li>Handle each connection in a separate thread</li>
68
<li>Use non-blocking system calls to handle all connections
72
<p>When dealing with many connections in one thread, the
73
scheduling is the responsibility of the application, not the
74
operating system, and is usually implemented by calling a
75
registered function when each connection is ready to for
76
reading or writing -- commonly known as event-driven, or
77
callback-based, programming.</p>
79
<p>Since multi-threaded programming is often tricky, even with
80
high level abstractions, and since forking Python processes has
81
many disadvantages, like Python's reference counting not
82
playing well with copy-on-write and problems with shared state,
83
it was felt the best option was an event-driven framework. A
84
benefit of such approach is that by letting other event-driven
85
frameworks take over the main loop, server and client code are
86
essentially the same - making peer-to-peer a reality. While
87
Twisted includes its own event loop, Twisted can already
88
interoperate with <code>GTK+</code>'s and <code>Tk</code>'s
89
mainloops, as well as provide an emulation of event-based I/O
90
for Jython (specific support for the Swing toolkit is planned).
91
Client code is never aware of the loop it is running under, as
92
long as it is using Twisted's interface for registering for
93
interesting events.</p>
95
<p>Some examples of programs which were written using the
96
Twisted framework are <code>twisted.web</code> (a web server),
97
<code>twisted.mail</code> (a mail server, supporting both SMTP
98
and POP3, as well as relaying), <code>twisted.words</code> (a
99
chat application supporting integration between a variety of IM
100
protocols, like IRC, AOL Instant Messenger's TOC and
101
Perspective Broker, a remote-object protocol native to
102
Twisted), <code>im</code> (an instant messenger which connects
103
to twisted.words) and <code>faucet</code> (a GUI client for the
104
<code>twisted.reality</code> interactive-fiction framework).
105
Twisted can be useful for any network or GUI application
106
written in Python.</p>
108
<p>However, event-driven programming still contains some tricky
109
aspects. As each callback must be finished as soon as possible,
110
it is not possible to keep persistent state in function-local
111
variables. In addition, some programming techniques, such as
112
recursion, are impossible to use. Event-driven programming has
113
a reputation of being hard to use due to the frequent need to
114
write state machines. Twisted was built with the assumption
115
that with the right library, event-driven programming is easier
116
then multi-threaded programming. Twisted aims to be that
119
<p>Twisted includes both high-level and low-level support for
120
protocols. Most protocol implementation by twisted are in a
121
package which tries to implement "mechanisms, not policy". On
122
top of those implementations, Twisted includes usable
123
implementations of those protocols: for example, connecting the
124
abstract HTTP protocol handler to a concrete resource-tree, or
125
connecting the abstract mail protocol handler to deliver mail
126
to maildirs according to domains. Twisted tries to come with as
127
much functionality as possible out of the box, while not
128
constraining a programmer to a choice between using a
129
possibly-inappropriate class and rewriting the non-interesting
132
<p>Twisted also includes Perspective Broker, a simple
133
remote-object framework, which allows Twisted servers to be
134
divided into separate processes as the end deployer (rather
135
then the original programmer) finds most convenient. This
136
allows, for example, Twisted web servers to pass requests for
137
specific URLs with co-operating servers so permissions are
138
granted according to the need of the specific application,
139
instead of being forced into giving all the applications all
140
permissions. The co-operation is truly symmetrical, although
141
typical deployments (such as the one which the Twisted web site
142
itself uses) use a master/slave relationship.</p>
144
<p>Twisted is not alone in the niche of a Python network
145
framework. One of the better known frameworks is Medusa. Medusa
146
is used, among other things, as Zope's native server serving
147
HTTP, FTP and other protocols. However, Medusa is no longer
148
under active development, and the Twisted development team had
149
a number of goals which would necessitate a rewrite of large
150
portions of Medusa. Twisted seperates protocols from the
151
underlying transport layer. This seperation has the advantages
152
of resuability (for example, using the same clients and servers
153
over SSL) and testability (because it is easy to test the
154
protocol with a much lighter test harness) among others.
155
Twisted also has a very flexible main-loop which can
156
interoperate with third-party main-loops, making it usable in
157
GUI programs too.</p>
159
<h3>Complementing Python</h3>
161
<p>Python comes out of the box with "batteries included".
162
However, it seems that many Python projects rewrite some basic
163
parts: logging to files, parsing options and high level
164
interfaces to reflection. When the Twisted project found itself
165
rewriting those, it moved them into a separate subpackage,
166
which does not depend on the rest of the twisted framework.
167
Hopefully, people will use <code>twisted.python</code> more and
168
solve interesting problems instead. Indeed, it is one of
169
Twisted's goals to serve as a repository for useful Python
172
<p>One useful module is <code>twisted.python.reflect</code>,
173
which has methods like <code>prefixedMethods</code>, which
174
returns all methods with a specific prefix. Even though some
175
modules in Python itself implement such functionality (notably,
176
<code>urllib2</code>), they do not expose it as a function
177
usable by outside code. Another useful module is
178
<code>twisted.python.hook</code>, which can add pre-hooks and
179
post-hooks to methods in classes.</p>
183
# Add all method names beginning with opt_ to the given
184
# dictionary. This cannot be done with dir(), since
185
# it does not search in superclasses
187
reflect.addMethodNamesToDict(self.__class__, dct, "opt_")
189
# Sum up all lists, in the given class and superclasses,
190
# which have a given name. This gives us "different class
191
# semantics": attributes do not override, but rather append
193
reflect.accumulateClassList(self.__class__, 'optFlags', flags)
195
# Add lock-acquire and lock-release to all methods which
196
# are not multi-thread safe
197
for methodName in klass.synchronized:
198
hook.addPre(klass, methodName, _synchPre)
199
hook.addPost(klass, methodName, _synchPost)
203
<h6>Listing 1: Using <code>twisted.python.reflect</code> and
204
<code>twisted.python.hook</code></h6>
207
<p>The <code>twisted.python</code> subpackage also contains a
208
high-level interface to getopt which supplies as much power as
209
plain getopt while avoiding long
210
<code>if</code>/<code>elif</code> chains and making many common
211
cases easier to use. It uses the reflection interfaces in
212
<code>twisted.python.reflect</code> to find which options the
213
class is interested in, and constructs the argument to
214
<code>getopt</code>. Since in the common case options' values
215
are just saved in instance attributes, it is very easy to
216
indicate interest in such options. However, for the cases
217
custom code needs to be run for an option (for example,
218
counting how many <code>-v</code> options were given to
219
indicate verbosity level), it will call a method which is named
224
class ServerOptions(usage.Options):
225
# Those are (short and long) options which
226
# have no argument. The corresponding attribute
227
# will be true iff this option was given
228
optFlags = [['nodaemon','n'],
233
# This are options which require an argument
234
# The default is used if no such option was given
235
# Note: since options can only have string arguments,
236
# putting a non-string here is a reliable way to detect
237
# whether the option was given
238
optStrings = [['logfile','l',None],
239
['file','f','twistd.tap'],
241
['pidfile','','twistd.pid'],
244
# For methods which can be called multiple times
245
# or have other unusual semantics, a method will be called
246
# Twisted assumes that the option needs an argument if and only if
247
# the method is defined to accept an argument.
248
def opt_plugin(self, pkgname):
249
pkg = __import__(pkgname)
250
self.python = os.path.join(os.path.dirname(
251
os.path.abspath(pkg.__file__)), 'config.tac')
253
# Most long options based on methods are aliased to short
254
# options. If there is only one letter, Twisted knows it is a short
255
# option, so it is "-g", not "--g"
259
config = ServerOptions()
260
config.parseOptions()
261
except usage.error, ue:
262
print "%s: %s" % (sys.argv[0], ue)
266
<h6>Listing 2: <code>twistd</code>'s Usage Code</h6>
269
<p>Unlike <code>getopt</code>, Twisted has a useful abstraction
270
for the non-option arguments: they are passed as arguments to
271
the <code>parsedArgs</code> method. This means too many
272
arguments, or too few, will cause a usage error, which will be
273
flagged. If an unknown number of arguments is desired,
274
explicitly using a tuple catch-all argument will work.</p>
276
<h3>Configuration</h3>
278
<p>The formats of configuration files have shown two visible
279
trends over the years. On the one hand, more and more
280
programmability has been added, until sometimes they become a
281
new language. The extreme end of this trend is using a regular
282
programming language, such as Python, as the configuration
283
language. On the other hand, some configuration files became
284
more and more machine editable, until they become a miniature
285
database formates. The extreme end of that trend is using a
286
generic database tool.</p>
288
<p>Both trends stem from the same rationale -- the need to use
289
a powerful general purpose tool instead of hacking domain
290
specific languages. Domain specific languages are usually
291
ad-hoc and not well designed, having neither the power of
292
general purpose languages nor the predictable machine editable
293
format of generic databases.</p>
295
<p>Twisted combines these two trends. It can read the
296
configuration either from a Python file, or from a pickled
297
file. To some degree, it integrates the approaches by
298
auto-pickling state on shutdown, so the configuration files can
299
migrate from Python into pickles. Currently, there is no way to
300
go back from pickles to equivalent Python source, although it
301
is planned for the future. As a proof of concept, the RPG
302
framework Twisted Reality already has facilities for creating
303
Python source which evaluates into a given Python object.</p>
307
from twisted.internet import main
308
from twisted.web import proxy, server
309
site = server.Site(proxy.ReverseProxyResource('www.yahoo.com', 80, '/'))
310
application = main.Application('web-proxy')
311
application.listenOn(8080, site)
314
<h6>Listing 3: The configuration file for a reverse web
318
<p>Twisted's main program, <code>twistd</code>, can receive
319
either a pickled <code>twisted.internet.main.Application</code>
320
or a Python file which defines a variable called
321
<code>application</code>. The application can be saved at any
322
time by calling its <code>save</code> method, which can take an
323
optional argument to save to a different file name. It would be
324
fairly easy, for example, to have a Twisted server which saves
325
the application every few seconds to a file whose name depends
326
on the time. Usually, however, one settles for the default
327
behavior which saves to a <code>shutdown</code> file. Then, if
328
the shutdown configuration proves suitable, the regular pickle
329
is replaced by the shutdown file. Hence, on the fly
330
configuration changes, regardless of complexity, can always
333
<p>There are several client/server protocols which let a
334
suitably privileged user to access to application variable and
335
change it on the fly. The first, and least common denominator,
336
is telnet. The administrator can telnet into twisted, and issue
337
Python statements to her heart's content. For example, one can
338
add ports to listen on to the application, reconfigure the web
339
servers and various other ways by simple accessing
340
<code>__main__.application</code>. Some proof of concepts for a
341
simple suite of command-line utilities to control a Twisted
342
application were written, including commands which allow an
343
administrator to shut down the server or save the current state
344
to a tap file. These are especially useful on Microsoft
345
Windows(tm) platforms, where the normal UNIX way of
346
communicating shutdown requests via signals are less
349
<p>If reconfiguration on the fly is not necessary, Python
350
itself can be used as the configuration editor. Loading the
351
application is as simple as unpickling it, and saving it is
352
done by calling its <code>save</code> method. It is quite easy
353
to add more services or change existing ones from the Python
354
interactive mode.</p>
356
<p>A more sophisticated way to reconfigure the application on
357
the fly is via the manhole service. Manhole is a client/server
358
protocol based on top of Perspective Broker, Twisted's
359
translucent remote-object protocol which will be covered later.
360
Manhole has a graphical client called <code>gtkmanhole</code>
361
which can access the server and change its state. Since Twisted
362
is modular, it is possible to write more services for user
363
friendly configuration. For example, through-the-web
364
configuration is planned for several services, notably
367
<p>For cases where a third party wants to distribute both the
368
code for a server and a ready to run configuration file, there
369
is the plugin configuration. Philosophically similar to the
370
<code>--python</code> option to <code>twistd</code>, it
371
simplifies the distribution process. A plugin is an archive
372
which is ready to be unpacked into the Python module path. In
373
order to keep a clean tree, <code>twistd</code> extends the
374
module path with some Twisted-specific paths, like the
375
directory <code>TwistedPlugins</code> in the user's home
376
directory. When a plugin is unpacked, it should be a Python
377
package which includes, alongside <code>__init__.py</code> a
378
file named <code>config.tac</code>. This file should define a
379
variable named <code>application</code>, in a similar way to
380
files loaded with <code>--python</code>. The plugin way of
381
distributing configurations is meant to reduce the temptation
382
to put large amount of codes inside the configuration file
385
<p>Putting class and function definition inside the
386
configuration files would make the persistent servers which are
387
auto-generated on shutdown useless, since they would not have
388
access to the classes and functions defined inside the
389
configuration file. Thus, the plugin method is intended so
390
classes and functions can still be in regular, importable,
391
Python modules, but still allow third parties distribute
392
powerful configurations. Plugins are used by some of the
393
Twisted Reality virtual worlds.</p>
395
<h3>Ports, Protocol and Protocol Factories</h3>
397
<p><code>Port</code> is the Twisted class which represents a
398
socket listening on a port. Currently, twisted supports both
399
internet and unix-domain sockets, and there are SSL classes
400
with identical interface. A <code>Port</code> is only
401
responsible for handling the transfer layer. It calls
402
<code>accept</code> on the socket, checks that it actually
403
wants to deal with the connection and asks its factory for a
404
protocol. The factory is usually a subclass of
405
<code>twisted.protocols.protocol.Factory</code>, and its most
406
important method is <code>buildProtocol</code>. This should
407
return something that adheres to the protocol interface, and is
408
usually a subclass of
409
<code>twisted.protocols.protocol.Protocol</code>.</p>
413
from twisted.protocols import protocol
414
from twisted.internet import main, tcp
416
class Echo(protocol.Protocol):
417
def dataReceived(self, data):
418
self.transport.write(data)
420
factory = protocol.Factory()
421
factory.protocol = Echo
422
port = tcp.Port(8000, factory)
423
app = main.Application("echo")
428
<h6>Listing 4: A Simple Twisted Application</h6>
431
<p>The factory is responsible for two tasks: creating new
432
protocols, and keeping global configuration and state. Since
433
the factory builds the new protocols, it usually makes sure the
434
protocols have a reference to it. This allows protocols to
435
access, and change, the configuration. Keeping state
436
information in the factory is the primary reason for keeping an
437
abstraction layer between ports and protocols. Examples of
438
configuration information is the root directory of a web server
439
or the user database of a telnet server. Note that it is
440
possible to use the same factory in two different Ports. This
441
can be used to run the same server bound to several different
442
addresses but not to all of them, or to run the same server on
443
a TCP socket and a UNIX domain sockets.</p>
445
<p>A protocol begins and ends its life with
446
<code>connectionMade</code> and <code>connectionLost</code>;
447
both are called with no arguments. <code>connectionMade</code>
448
is called when a connection is first established. By then, the
449
protocol has a <code>transport</code> attribute. The
450
<code>transport</code> attribute is a <code>Transport</code> -
451
it supports <code>write</code> and <code>loseConnection</code>.
452
Both these methods never block: <code>write</code> actually
453
buffers data which will be written only when the transport is
454
signalled ready to for writing, and <code>loseConnection</code>
455
marks the transport for closing as soon as there is no buffered
456
data. Note that transports do <em>not</em> have a
457
<code>read</code> method: data arrives when it arrives, and the
458
protocol must be ready for its <code>dataReceived</code>
459
method, or its <code>connectionLost</code> method, to be
460
called. The transport also supports a <code>getPeer</code>
461
method, which returns parameters about the other side of the
462
transport. For TCP sockets, this includes the remote IP and
467
# A tcp port-forwarder
468
# A StupidProtocol sends all data it gets to its peer.
469
# A StupidProtocolServer connects to the host/port,
470
# and initializes the client connection to be its peer
471
# and itself to be the client's peer
472
from twisted.protocols import protocol
474
class StupidProtocol(protocol.Protocol):
475
def connectionLost(self): self.peer.loseConnection();del self.peer
476
def dataReceived(self, data): self.peer.write(data)
478
class StupidProtocolServer(StupidProtocol):
479
def connectionMade(self):
480
clientProtocol = StupidProtocol()
481
clientProtocol.peer = self.transport
482
self.peer = tcp.Client(self.factory.host, self.factory.port,
485
# Create a factory which creates StupidProtocolServers, and
486
# has the configuration information they assume
487
def makeStupidFactory(host, port):
488
factory = protocol.Factory()
489
factory.host, factory.port = host, port
490
factory.protocol = StupidProtocolServer
494
<h6>Listing 5: TCP forwarder code</h6>
497
<h3>The Event Loop</h3>
499
<p>While Twisted has the ability to let other event loops take
500
over for integration with GUI toolkits, it usually uses its own
501
event loop. The event loop code uses global variables to
502
maintain interested readers and writers, and uses Python's
503
<code>select()</code> function, which can accept any object
504
which has a <code>fileno()</code> method, not only raw file
505
descriptors. Objects can use the event loop interface to
506
indicate interest in either reading to or writing from a given
507
file descriptor. In addition, for those cases where time-based
508
events are needed (for example, queue flushing or periodic POP3
509
downloads), Twisted has a mechanism for repeating events at
510
known delays. While far from being real-time, this is enough
511
for most programs' needs.</p>
513
<h3>Going Higher Level</h3>
515
<p>Unfortunately, handling arbitrary data chunks is a hard way
516
to code a server. This is why twisted has many classes sitting
517
in submodules of the twisted.protocols package which give
518
higher level interface to the data. For line oriented
519
protocols, <code>LineReceiver</code> translates the low-level
520
<code>dataReceived</code> events into <code>lineReceived</code>
521
events. However, the first naive implementation of
522
<code>LineReceiver</code> proved to be too simple. Protocols
523
like HTTP/1.1 or Freenet have packets which begin with header
524
lines that include length information, and then byte streams.
525
<code>LineReceiver</code> was rewritten to have a simple
526
interface for switching at the protocol layer between
527
line-oriented parts and byte-stream parts.</p>
529
<p>Another format which is gathering popularity is Dan J.
530
Bernstein's netstring format. This format keeps ASCII text as
531
ASCII, but allows arbitrary bytes (including nulls and
532
newlines) to be passed freely. However, netstrings were never
533
designed to be used in event-based protocols where over-reading
534
is unavoidable. Twisted makes sure no user will have to deal
535
with the subtle problems handling netstrings in event-driven
536
programs by providing <code>NetstringReceiver</code>.</p>
538
<p>For even higher levels, there are the protocol-specific
539
protocol classes. These translate low-level chunks into
540
high-level events such as "HTTP request received" (for web
541
servers), "approve destination address" (for mail servers) or
542
"get user information" (for finger servers). Many RFCs have
543
been thus implemented for Twisted (at latest count, more then
544
12 RFCs have been implemented). One of Twisted's goals is to be
545
a repository of event-driven implementations for various
546
protocols in Python.</p>
550
class DomainSMTP(SMTP):
552
def validateTo(self, helo, destination):
554
user, domain = string.split(destination, '@', 1)
557
if not self.factory.domains.has_key(domain):
559
if not self.factory.domains[domain].exists(user, domain, self):
563
def handleMessage(self, helo, origin, recipients, message):
564
# No need to check for existence -- only recipients which
565
# we approved at the validateTo stage are passed here
566
for recipient in recipients:
567
user, domain = string.split(recipient, '@', 1)
568
self.factory.domains[domain].saveMessage(origin, user, message,
572
<h6>Listing 6: Implementation of virtual domains using the
573
SMTP protocol class</h6>
576
<p>Copious documentation on writing new protocol abstraction
577
exists, since this is the largest amount of code written --
578
much like most operating system code is device drivers. Since
579
many different protocols have already been implemented, there
580
are also plenty of examples to draw on. Usually implementing
581
the client-side of a protocol is particularly challenging,
582
since protocol designers tend to assume much more state kept on
583
the client side of a connection then on the server side.</p>
585
<h3>The <code>twisted.tap</code> Package and
586
<code>mktap</code></h3>
588
<p>Since one of Twisted's configuration formats are pickles,
589
which are tricky to edit by hand, Twisted evolved a framework
590
for creating such pickles. This framework is contained in the
591
<code>twisted.tap</code> package and the <code>mktap</code>
592
script. New servers, or new ways to configure existing servers,
593
can easily participate in the twisted.tap framework by creating
594
a <code>twisted.tap</code> submodule.</p>
596
<p>All <code>twisted.tap</code> submodules must conform to a
597
rigid interface. The interface defines functions to accept the
598
command line parameters, and functions to take the processed
599
command line parameters and add servers to
600
<code>twisted.main.internet.Application</code>. Existing
601
<code>twisted.tap</code> submodules use
602
<code>twisted.python.usage</code>, so the command line format
603
is consistent between different modules.</p>
605
<p>The <code>mktap</code> utility gets some generic options,
606
and then the name of the server to build. It imports a
607
same-named <code>twisted.tap</code> submodule, and lets it
608
process the rest of the options and parameters. This makes sure
609
that the process configuring the <code>main.Application</code>
610
is agnostic for where it is used. This allowed
611
<code>mktap</code> to grow the <code>--append</code> option,
612
which appends to an existing pickle rather then creating a new
613
one. This option is frequently used to post-add a telnet server
614
to an application, for net-based on the fly configuration
617
<p>When running <code>mktap</code> under UNIX, it saves the
618
user id and group id inside the tap. Then, when feeding this
619
tap into <code>twistd</code>, it changes to this user/group id
620
after binding the ports. Such a feature is necessary in any
621
production-grade server, since ports below 1024 require root
622
privileges to use on UNIX -- but applications should not run as
623
root. In case changing to the specified user causes difficulty
624
in the build environment, it is also possible to give those
625
arguments to <code>mktap</code> explicitly.</p>
629
from twisted.internet import tcp, stupidproxy
630
from twisted.python import usage
633
usage: mktap stupid [OPTIONS]
635
Options are as follows:
636
--port <#>, -p: set the port number to <#>.
637
--host <host>, -h: set the host to <host>
638
--dest_port <#>, -d: set the destination port to <#>
641
class Options(usage.Options):
642
optStrings = [["port", "p", 6666],
643
["host", "h", "localhost"],
644
["dest_port", "d", 6665]]
646
def getPorts(app, config):
647
s = stupidproxy.makeStupidFactory(config.host, int(config.dest_port))
648
return [(int(config.port), s)]
651
<h6>Listing 7: <code>twisted.tap.stupid</code></h6>
654
<p>The <code>twisted.tap</code> framework is one of the reasons
655
servers can be set up with little knowledge and time. Simply
656
running <code>mktap</code> with arguments can bring up a web
657
server, a mail server or an integrated chat server -- with
658
hardly any need for maintainance. As a working
659
proof-on-concept, the <code>tap2deb</code> utility exists to
660
wrap up tap files in Debian packages, which include scripts for
661
running and stopping the server and interact with
662
<code>init(8)</code> to make sure servers are automatically run
663
on start-up. Such programs can also be written to interface
664
with the Red Hat Package Manager or the FreeBSD package
665
management systems.</p>
669
% mktap --uid 33 --gid 33 web --static /var/www --port 80
670
% tap2deb -t web.tap -m 'Moshe Zadka <moshez@debian.org>'
673
# dpkg -i .build/twisted-web_1.0_all.deb
676
<h6>Listing 8: Bringing up a web server on a Debian
680
<h3>Multi-thread Support</h3>
682
<p>Sometimes, threads are unavoidable or hard to avoid. Many
683
legacy programs which use threads want to use Twisted, and some
684
vendor APIs have no non-blocking version -- for example, most
685
database systems' API. Twisted can work with threads, although
686
it supports only one thread in which the main select loop is
687
running. It can use other threads to simulate non-blocking API
688
over a blocking API -- it spawns a thread to call the blocking
689
API, and when it returns, the thread calls a callback in the
690
main thread. Threads can call callbacks in the main thread
691
safely by adding those callbacks to a list of pending events.
692
When the main thread is between select calls, it searches
693
through the list of pending events, and executes them. This is
694
used in the <code>twisted.enterprise</code> package to supply
695
an event driven interfaces to databases, which uses Python's DB
698
<p>Twisted tries to optimize for the common case -- no threads.
699
If there is need for threads, a special call must be made to
700
inform the <code>twisted.python.threadable</code> module that
701
threads will be used. This module is implemented differently
702
depending on whether threads will be used or not. The decision
703
must be made before importing any modules which use threadable,
704
and so is usually done in the main application. For example,
705
<code>twistd</code> has a command line option to initialize
708
<p>Twisted also supplies a module which supports a threadpool,
709
so the common task of implementing non-blocking APIs above
710
blocking APIs will be both easy and efficient. Threads are kept
711
in a pool, and dispatch requests are done by threads which are
712
not working. The pool supports a maximum amount of threads, and
713
will throw exceptions when there are more requests than
714
allowable threads.</p>
716
<p>One of the difficulties about multi-threaded systems is
717
using locks to avoid race conditions. Twisted uses a mechanism
718
similar to Java's synchronized methods. A class can declare a
719
list of methods which cannot safely be called at the same time
720
from two different threads. A function in threadable then uses
721
<code>twisted.python.hook</code> to transparently add
722
lock/unlock around these methods. This allows Twisted classes
723
to be written without thought about threading, except for one
724
localized declaration which does not entail any performance
725
penalty for the single-threaded case.</p>
727
<h3>Twisted Mail Server</h3>
729
<p>Mail servers have a history of security flaws. Sendmail is
730
by now the poster boy of security holes, but no mail servers,
731
bar maybe qmail, are free of them. Like Dan Bernstein of qmail
732
fame said, mail cannot be simply turned off -- even the
733
simplest organization needs a mail server. Since Twisted is
734
written in a high-level language, many problems which plague
735
other mail servers, notably buffer overflows, simply do not
736
exist. Other holes are avoidable with correct design. Twisted
737
Mail is a project trying to see if it is possible to write a
738
high quality high performance mail server entirely in
741
<p>Twisted Mail is built on the SMTP server and client protocol
742
classes. While these present a level of abstraction from the
743
specific SMTP line semantics, they do not contain any message
744
storage code. The SMTP server class does know how to divide
745
responsibility between domains. When a message arrives, it
746
analyzes the recipient's address, tries matching it with one of
747
the registered domain, and then passes validation of the
748
address and saving the message to the correct domain, or
749
refuses to handle the message if it cannot handle the domain.
750
It is possible to specify a catch-all domain, which will
751
usually be responsible for relaying mails outwards.</p>
753
<p>While correct relaying is planned for the future, at the
754
moment we have only so-called "smarthost" relaying. All e-mail
755
not recognized by a local domain is relayed to a single outside
756
upstream server, which is supposed to relay the mail further.
757
This is the configuration for most home machines, which are
758
Twisted Mail's current target audience.</p>
760
<p>Since the people involved in Twisted's development were
761
reluctant to run code that runs as a super user, or with any
762
special privileges, it had to be considered how delivery of
763
mail to users is possible. The solution decided upon was to
764
have Twisted deliver to its own directory, which should have
765
very strict permissions, and have users pull the mail using
766
some remote mail access protocol like POP3. This means only a
767
user would write to his own mail box, so no security holes in
768
Twisted would be able to adversely affect a user.</p>
770
<p>Future plans are to use a Perspective Broker-based service
771
to hand mail to users to a personal server using a UNIX domain
772
socket, as well as to add some more conventional delivery
773
methods, as scary as they may be.</p>
775
<p>Because the default configuration of Twisted Mail is to be
776
an integrated POP3/SMTP servers, it is ideally suited for the
777
so-called POP toaster configuration, where there are a
778
multitude of virtual users and domains, all using the same IP
779
address and computer to send and receive mails. It is fairly
780
easy to configure Twisted as a POP toaster. There are a number
781
of deployment choices: one can append a telnet server to the
782
tap for remote configuration, or simple scripts can add and
783
remove users from the user database. The user database is saved
784
as a directory, where file names are keys and file contents are
785
values, so concurrency is not usually a problem.</p>
789
% mktap mail -d foobar.com=$HOME/Maildir/ -u postmaster=secret -b \
795
<h6>Bringing up a simple mail-server</h6>
798
<p>Twisted's native mail storage format is Maildir, a format
799
that requires no locking and is safe and atomic. Twisted
800
supports a number of standardized extensions to Maildir,
801
commonly known as Maildir++. Most importantly, it supports
802
deletion as simply moving to a subfolder named
803
<code>Trash</code>, so mail is recoverable if accessed through
804
a protocol which allows multiple folders, like IMAP. However,
805
Twisted itself currently does not support any such protocol
808
<h3>Introducing Perspective Broker</h3>
810
<h4>All the World's a Game</h4>
812
<p>Twisted was originally designed to support multi-player
813
games; a simulated "real world" environment. Experience with
814
game systems of that type is enlightening as to the nature of
815
computing on the whole. Almost all services on a computer are
816
modeled after some simulated real-world activity. For example,
817
e-"mail", or "document publishing" on the web. Even
818
"object-oriented" programming is based around the notion that
819
data structures in a computer simulate some analogous
820
real-world objects.</p>
822
<p>All such networked simulations have a few things in common.
823
They each represent a service provided by software, and there
824
is usually some object where "global" state is kept. Such a
825
service must provide an authentication mechanism. Often, there
826
is a representation of the authenticated user within the
827
context of the simulation, and there are also objects aside
828
from the user and the simulation itself that can be
831
<p>For most existing protocols, Twisted provides these
832
abstractions through <code>twisted.internet.passport</code>.
833
This is so named because the most important common
834
functionality it provides is authentication. A simulation
835
"world" as described above -- such as an e-mail system,
836
document publishing archive, or online video game -- is
837
represented by subclass of <code>Service</code>, the
838
authentication mechanism by an <code>Authorizer</code> (which
839
is a set of <code>Identities</code>), and the user of the
840
simulation by a <code>Perspective</code>. Other objects in the
841
simulation may be represented by arbitrary python objects,
842
depending upon the implementation of the given protocol.</p>
844
<p>New problem domains, however, often require new protocols,
845
and re-implementing these abstractions each time can be
846
tedious, especially when it's not necessary. Many efforts have
847
been made in recent years to create generic "remote object" or
848
"remote procedure call" protocols, but in developing Twisted,
849
these protocols were found to require too much overhead in
850
development, be too inefficient at runtime, or both.</p>
852
<p>Perspective Broker is a new remote-object protocol designed
853
to be lightweight and impose minimal constraints upon the
854
development process and use Python's dynamic nature to good
855
effect, but still relatively efficient in terms of bandwidth
856
and CPU utilization. <code>twisted.spread.pb</code> serves as a
857
reference implementation of the protocol, but implementation of
858
Perspective Broker in other languages is already underway.
859
<code>spread</code> is the <code>twisted</code> subpackage
860
dealing with remote calls and objects, and has nothing to do
861
with the <code>spread</code> toolkit.</p>
863
<p>Perspective Broker extends
864
<code>twisted.internet.passport</code>'s abstractions to be
865
concrete objects rather than design patterns. Rather than
866
having a <code>Protocol</code> implementation translate between
867
sequences of bytes and specifically named methods (as in the
868
other Twisted <code>Protocols</code>), Perspective Broker
869
defines a direct mapping between network messages and
870
quasi-arbitrary method calls.</p>
872
<h3>Translucent, not Transparent</h3>
874
<p>In a server application where a large number of clients may
875
be interacting at once, it is not feasible to have an
876
arbitrarily large number of OS threads blocking and waiting for
877
remote method calls to return. Additionally, the ability for
878
any client to call any method of an object would present a
879
significant security risk. Therefore, rather than attempting to
880
provide a transparent interface to remote objects,
881
<code>twisted.spread.pb</code> is "translucent", meaning that
882
while remote method calls have different semantics than local
883
ones, the similarities in semantics are mirrored by
884
similarities in the syntax. Remote method calls impose as
885
little overhead as possible in terms of volume of code, but "as
886
little as possible" is unfortunately not "nothing".</p>
888
<p><code>twisted.spread.pb</code> defines a method naming
889
standard for each type of remotely accessible object. For
890
example, if a client requests a method call with an expression
891
such as <code>myPerspective.doThisAction()</code>, the remote
892
version of <code>myPerspective</code> would be sent the message
893
<code>perspective_doThisAction</code>. Depending on the manner
894
in which an object is accessed, other method prefixes may be
895
<code>observe_</code>, <code>view_</code>, or
896
<code>remote_</code>. Any method present on a remotely
897
accessible object, and named appropriately, is considered to be
898
published -- since this is accomplished with
899
<code>getattr</code>, the definition of "present" is not just
900
limited to methods defined on the class, but instances may have
901
arbitrary callable objects associated with them as long as the
902
name is correct -- similarly to normal python objects.</p>
904
<p>Remote method calls are made on remote reference objects
905
(instances of <code>pb.RemoteReference</code>) by calling a
906
method with an appropriate name. However, that call will not
907
block -- if you need the result from a remote method call, you
908
pass in one of the two special keyword arguments to that method
909
-- <code>pbcallback</code> or <code>pberrback</code>.
910
<code>pbcallback</code> is a callable object which will be
911
called when the result is available, and <code>pberrback</code>
912
is a callable object which will be called if there was an
913
exception thrown either in transmission of the call or on the
916
<p>In the case that neither <code>pberrback</code> or
917
<code>pbcallback</code> is provided,
918
<code>twisted.spread.pb</code> will optimize network usage by
919
not sending confirmations of messages.</p>
924
class MyObject(pb.Referenceable):
925
def remote_doIt(self):
930
def myCallback(result):
931
print result # result will be 'did it'
932
def myErrback(stacktrace):
933
print 'oh no, mr. bill!'
935
myRemoteReference.doIt(pbcallback=myCallback,
939
<h6>Listing 9: A remotely accessible object and accompanying
943
<h3>Different Behavior for Different Perspectives</h3>
945
<p>Considering the problem of remote object access in terms of
946
a simulation demonstrates a requirement for the knowledge of an
947
actor with certain actions or requests. Often, when processing
948
message, it is useful to know who sent it, since different
949
results may be required depending on the permissions or state
952
<p>A simple example is a game where certain an object is
953
invisible, but players with the "Heightened Perception"
954
enchantment can see it. When answering the question "What
955
objects are here?" it is important for the room to know who is
956
asking, to determine which objects they can see. Parallels to
957
the differences between "administrators" and "users" on an
958
average multi-user system are obvious.</p>
960
<p>Perspective Broker is named for the fact that it does not
961
broker only objects, but views of objects. As a user of the
962
<code>twisted.spread.pb</code> module, it is quite easy to
963
determine the caller of a method. All you have to do is
964
subclass <code>Viewable</code>.</p>
969
class Greeter(pb.Viewable):
970
def view_greet(self, actor):
971
return "Hello %s!\n" % actor.perspectiveName
975
remoteGreeter.greet(pbcallback=sys.stdout.write)
979
<h6>Listing 10: An object responding to its calling
982
Before any arguments sent by the client, the actor
983
(specifically, the Perspective instance through which this
984
object was retrieved) will be passed as the first argument to
985
any <code>view_xxx</code> methods.
987
<h3>Mechanisms for Sharing State</h3>
989
<p>In a simulation of any decent complexity, client and server
990
will wish to share structured data. Perspective Broker provides
991
a mechanism for both transferring (copying) and sharing
992
(caching) that state.</p>
994
<p>Whenever an object is passed as an argument to or returned
995
from a remote method call, that object is serialized using
996
<code>twisted.spread.jelly</code>; a serializer similar in some
997
ways to Python's native <code>pickle</code>. Originally,
998
<code>pickle</code> itself was going to be used, but there were
999
several security issues with the <code>pickle</code> code as it
1000
stands. It is on these issues of security that
1001
<code>pickle</code> and <code>twisted.spread.jelly</code> part
1004
<p>While <code>twisted.spread.jelly</code> handles a few basic
1005
types such as strings, lists, dictionaries and numbers
1006
automatically, all user-defined types must be registered both
1007
for serialization and unserialization. This registration
1008
process is necessary on the sending side in order to determine
1009
if a particular object is shared, and whether it is shared as
1010
state or behavior. On the receiving end, it's necessary to
1011
prevent arbitrary code from being run when an object is
1012
unserialized -- a significant security hole in
1013
<code>pickle</code> for networked applications.</p>
1015
<p>On the sending side, the registration is accomplished by
1016
making the object you want to serialize a subclass of one of
1017
the "flavors" of object that are handled by Perspective Broker.
1018
A class may be <code>Referenceable</code>,
1019
<code>Viewable</code>, <code>Copyable</code> or
1020
<code>Cacheable</code>. These four classes correspond to
1021
different ways that the object will be seen remotely.
1022
Serialization flavors are mutually exclusive -- these 4 classes
1023
may not be mixed in with each other.</p>
1026
<li><code>Referenceable</code>: The remote side will refer to
1027
this object directly. Methods with the prefix
1028
<code>remote_</code> will be callable on it. No state will be
1031
<li><code>Viewable</code>: The remote side will refer to a
1032
proxy for this object, which indicates what perspective
1033
accessed this; as discussed above. Methods with the prefix
1034
<code>view_</code> will be callable on it, and have an
1035
additional first argument inserted (the perspective that
1036
called the method). No state will be transferred.</li>
1038
<li><code>Copyable</code>: Each time this object is
1039
serialized, its state will be copied and sent. No methods are
1040
remotely callable on it. By default, the state sent will be
1041
the instance's <code>__dict__</code>, but a method
1042
<code>getStateToCopyFor(perspective)</code> may be defined
1043
which returns an arbitrary serializable object for
1046
<li><code>Cacheable</code>: The first time this object is
1047
serialized, its state will be copied and sent. Each
1048
subsequent time, however, a reference to the original object
1049
will be sent to the receiver. No methods will be remotely
1050
callable on this object. By default, again, the state sent
1051
will be the instance's <code>__dict__</code>but a method
1052
<code>getStateToCacheAndObserveFor(perspective,
1053
observer)</code> may be defined to return alternative state.
1054
Since the state for this object is only sent once, the
1055
<code>observer</code> argument is an object representative of
1056
the receiver's representation of the <code>Cacheable</code>
1057
after unserialization -- method calls to this object will be
1058
resolved to methods prefixed with <code>observe_</code>,
1059
<em>on the receiver's <code>RemoteCache</code> of this
1060
object</em>. This may be used to keep the receiver's cache
1061
up-to-date as relevant portions of the <code>Cacheable</code>
1065
<h3>Publishing Objects with PB</h3>
1067
<p>The previous samples of code have shown how an individual
1068
object will interact over a previously-established PB
1069
connection. In order to get to that connection, you need to do
1070
some set-up work on both the client and server side; PB
1071
attempts to minimize this effort.</p>
1073
<p>There are two different approaches for setting up a PB
1074
server, depending on your application's needs. In the simplest
1075
case, where your application does not deal with the
1076
abstractions above -- services, identities, and perspectives --
1077
you can simply publish an object on a particular port.</p>
1080
<pre class="python">
1081
from twisted.spread import pb
1082
from twisted.internet import main
1083
class Echoer(pb.Root):
1084
def remote_echo(self, st):
1085
print 'echoing:', st
1087
if __name__ == '__main__':
1088
app = main.Application("pbsimple")
1089
app.listenOn(8789, pb.BrokerFactory(Echoer()))
1093
<h6>Listing 11: Creating a simple PB server</h6>
1096
<p>Listing 11 shows how to publish a simple object which
1097
responds to a single message, "echo", and returns whatever
1098
argument is sent to it. There is very little to explain: the
1099
"Echoer" class is a pb.Root, which is a small subclass of
1100
Referenceable designed to be used for objects published by a
1101
BrokerFactory, so Echoer follows the same rule for remote
1102
access that Referenceable does. Connecting to this service is
1103
almost equally simple.</p>
1106
<pre class="python">
1107
from twisted.spread import pb
1108
from twisted.internet import main
1109
def gotObject(object):
1110
print "got object:",object
1111
object.echo("hello network", pbcallback=gotEcho)
1113
print 'server echoed:',echo
1115
def gotNoObject(reason):
1116
print "no object:",reason
1118
pb.getObjectAt("localhost", 8789, gotObject, gotNoObject, 30)
1122
<h6>Listing 12: A client for Echoer objects.</h6>
1125
<p>The utility function <code>pb.getObjectAt</code> retrieves
1126
the root object from a hostname/port-number pair and makes a
1127
callback (in this case, <code>gotObject</code>) if it can
1128
connect and retrieve the object reference successfully, and an
1129
error callback (<code>gotNoObject</code>) if it cannot connect
1130
or the connection times out.</p>
1132
<p><code>gotObject</code> receives the remote reference, and
1133
sends the <code>echo</code> message to it. This call is
1134
visually noticeable as a remote method invocation by the
1135
distinctive <code>pbcallback</code> keyword argument. When the
1136
result from that call is received, <code>gotEcho</code> will be
1137
called, notifying us that in fact, the server echoed our input
1138
("hello network").</p>
1140
<p>While this setup might be useful for certain simple types of
1141
applications where there is no notion of a "user", the
1142
additional complexity necessary for authentication and service
1143
segregation is worth it. In particular, re-use of server code
1144
for things like chat (twisted.words) is a lot easier with a
1145
unified notion of users and authentication.</p>
1148
<pre class="python">
1149
from twisted.spread import pb
1150
from twisted.internet import main
1151
class SimplePerspective(pb.Perspective):
1152
def perspective_echo(self, text):
1153
print 'echoing',text
1155
class SimpleService(pb.Service):
1156
def getPerspectiveNamed(self, name):
1157
return SimplePerspective(name, self)
1158
if __name__ == '__main__':
1160
app = main.Application("pbecho")
1161
pbecho.SimpleService("pbecho",app).getPerspectiveNamed("guest")\
1162
.makeIdentity("guest")
1163
app.listenOn(pb.portno, pb.BrokerFactory(pb.AuthRoot(app)))
1167
<h6>Listing 13: A PB server using twisted's "passport"
1168
authentication.</h6>
1171
<p>In terms of the "functionality" it offers, this server is
1172
identical. It provides a method which will echo some simple
1173
object sent to it. However, this server provides it in a manner
1174
which will allow it to cooperate with multiple other
1175
authenticated services running on the same connection, because
1176
it uses the central Authorizer for the application.</p>
1178
<p>On the line that creates the <code>SimpleService</code>,
1179
several things happen.</p>
1182
<li>A SimpleService is created and persistently added to the
1183
<code>Application</code> instance.</li>
1185
<li>A SimplePerspective is created, via the overridden
1186
<code>getPerspectiveNamed</code> method.</li>
1188
<li>That <code>SimplePerspective</code> has an
1189
<code>Identity</code> generated for it, and persistently
1190
added to the <code>Application</code>'s
1191
<code>Authorizer</code>. The created identity will have the
1192
same name as the perspective ("guest"), and the password
1193
supplied (also, "guest"). It will also have a reference to
1194
the service "pbecho" and a perspective named "guest", by
1195
name. The <code>Perspective.makeIdentity</code> utility
1196
method prevents having to deal with the intricacies of the
1197
passport <code>Authorizer</code> system when one doesn't
1198
require strongly separate <code>Identity</code>s and
1199
<code>Perspective</code>s.</li>
1205
<p>Also, this server does not run itself, but instead persists
1206
to a file which can be run with twistd, offering all the usual
1207
amenities of daemonization, logging, etc. Once the server is
1208
run, connecting to it is similar to the previous example.</p>
1211
<pre class="python">
1212
from twisted.spread import pb
1213
from twisted.internet import main
1214
def success(message):
1215
print "Message received:",message
1218
print "Failure...",error
1220
def connected(perspective):
1221
perspective.echo("hello world",
1225
pb.connect(connected, failure, "localhost", pb.portno,
1226
"guest", "guest", "pbecho", "guest", 30)
1230
<h6>Listing 14: Connecting to an Authorized Service</h6>
1236
<p>This introduces a new utility -- <code>pb.connect</code>.
1237
This function takes a long list of arguments and manages the
1238
handshaking and challenge/response aspects of connecting to a
1239
PB service perspective, eventually calling back to indicate
1240
either success or failure. In this particular example, we are
1241
connecting to localhost on the default PB port (8787),
1242
authenticating to the identity "guest" with the password
1243
"guest", requesting the perspective "guest" from the service
1244
"pbecho". If this can't be done within 30 seconds, the
1245
connection will abort.</p>
1247
<p>In these examples, I've attempted to show how Twisted makes
1248
event-based scripting easier; this facilitates the ability to
1249
run short scripts as part of a long-running process. However,
1250
event-based programming is not natural to procedural scripts;
1251
it is more generally accepted that GUI programs will be
1252
event-driven whereas scripts will be blocking. An alternative
1253
client to our <code>SimpleService</code> using GTK illustrates
1254
the seamless meshing of Twisted and GTK.</p>
1257
<pre class="python">
1258
from twisted.internet import main, ingtkernet
1259
from twisted.spread.ui import gtkutil
1261
ingtkernet.install()
1263
def __init__(self, echoer):
1265
self.echoer = echoer
1266
w = gtk.GtkWindow(gtk.WINDOW_TOPLEVEL)
1267
vb = gtk.GtkVBox(); b = gtk.GtkButton("Echo:")
1268
self.entry = gtk.GtkEntry(); self.outry = gtk.GtkEntry()
1270
map(vb.add, [b, self.entry, self.outry])
1271
b.connect('clicked', self.clicked)
1272
w.connect('destroy', gtk.mainquit)
1274
def clicked(self, b):
1275
txt = self.entry.get_text()
1276
self.entry.set_text("")
1277
self.echoer.echo(txt, pbcallback=self.outry.set_text)
1278
l = gtkutil.Login(EchoClient, None, initialService="pbecho")
1283
<h6>Listing 15: A Twisted GUI application</h6>
1286
<h3>Event-Driven Web Object Publishing with Web.Widgets</h3>
1288
<p>Although PB will be interesting to those people who wish to
1289
write custom clients for their networked applications, many
1290
prefer or require a web-based front end. Twisted's built-in web
1291
server has been designed to accommodate this desire, and the
1292
presentation framework that one would use to write such an
1293
application is <code>twisted.web.widgets</code>. Web.Widgets
1294
has been designed to work in an event-based manner, without
1295
adding overhead to the designer or the developer's
1298
<p>Surprisingly, asynchronous web interfaces fit very well into
1299
the normal uses of purpose-built web toolkits such as PHP. Any
1300
experienced PHP, Zope, or WebWare developer will tell you that
1301
<em>separation of presentation, content, and logic</em> is very
1302
important. In practice, this results in a "header" block of
1303
code which sets up various functions which are called
1304
throughout the page, some of which load blocks of content to
1305
display. While PHP does not enforce this, it is certainly
1306
idiomatic. Zope enforces it to a limited degree, although it
1307
still allows control structures and other programmatic elements
1308
in the body of the content.</p>
1310
<p>In Web.Widgets, strict enforcement of this principle
1311
coincides very neatly with a "hands-free" event-based
1312
integration, where much of the work of declaring callbacks is
1313
implicit. A "Presentation" has a very simple structure for
1314
evaluating Python expressions and giving them a context to
1315
operate in. The "header" block which is common to many
1316
templating systems becomes a class, which represents an
1317
enumeration of events that the template may generate, each of
1318
which may be responded to either immediately or latently.</p>
1320
<p>For the sake of simplicity, as well as maintaining
1321
compatibility for potential document formats other than HTML,
1322
Presentation widgets do not attempt to parse their template as
1323
HTML tags. The structure of the template is <code>"HTML Text
1324
%%%%python_expression()%%%% more HTML Text"</code>. Every set
1325
of 4 percent signs (%%%%) switches back and forth between
1326
evaluation and printing.</p>
1328
<p>No control structures are allowed in the template. This was
1329
originally thought to be a potentially major inconvenience, but
1330
with use of the Web.Widgets code to develop a few small sites,
1331
it has seemed trivial to encapsulate any table-formatting code
1332
within a method; especially since those methods can take string
1333
arguments if there's a need to customize the table's
1336
<p>The namespace for evaluating the template expressions is
1337
obtained by scanning the class hierarchy for attributes, and
1338
getting each of those attributes from the current instance.
1339
This means that all methods will be bound methods, so
1340
indicating "self" explicitly is not required. While it is
1341
possible to override the method for creating namespaces, using
1342
this default has the effect of associating all presentation
1343
code for a particular widget in one class, along with its
1344
template. If one is working with a non-programmer designer, and
1345
the template is in an external file, it is always very clear to
1346
the designer what functionality is available to them in any
1347
given scope, because there is a list of available methods for
1348
any given class.</p>
1350
<p>A convenient event to register for would be a response from
1351
the PB service that we just implemented. We can use the
1352
<code>Deferred</code> class in order to indicate to the widgets
1353
framework that certain work has to be done later. This is a
1354
Twisted convention which one can currently use in PB as well as
1355
webwidgets; any framework which needs the ability to defer a
1356
return value until later should use this facility. Elements of
1357
the page will be rendered from top to bottom as data becomes
1358
available, so the page will not be blocked on rendering until
1359
all deferred elements have been completed.</p>
1362
<pre class="python">
1363
from twisted.spread import pb
1364
from twisted.python import defer
1365
from twisted.web import widgets
1366
class EchoDisplay(widgets.Presentation):
1367
template = """<H1>Welcome to my widget, displaying %%%%echotext%%%%.</h1>
1368
<p>Here it is: %%%%getEchoPerspective()%%%%</p>"""
1369
echotext = 'hello web!'
1370
def getEchoPerspective(self):
1371
d = defer.Deferred()
1372
pb.connect(d.callback, d.errback, "localhost", pb.portno,
1373
"guest", "guest", "pbecho", "guest", 1)
1374
d.addCallbacks(self.makeListOf, self.formatTraceback)
1375
return ['<b>',d,'</b>']
1376
def makeListOf(self, echoer):
1377
d = defer.Deferred()
1378
echoer.echo(self.echotext, pbcallback=d.callback, pberrback=d.errback)
1379
d.addCallbacks(widgets.listify, self.formatTraceback)
1381
if __name__ == "__main__":
1382
from twisted.web import server
1383
from twisted.internet import main
1384
a = main.Application("pbweb")
1385
gdgt = widgets.Gadget()
1386
gdgt.widgets['index'] = EchoDisplay()
1387
a.listenOn(8080, server.Site(gdgt))
1391
<h6>Listing 16: an event-based web widget.</h6>
1394
<p>Each time a Deferred is returned as part of the page, the
1395
page will pause rendering until the deferred's
1396
<code>callback</code> method is invoked. When that callback is
1397
made, it is inserted at the point in the page where rendering
1400
<p>If necessary, there are options within web.widgets to allow
1401
a widget to postpone or cease rendering of the entire page --
1402
for example, it is possible to write a FileDownload widget,
1403
which will override the rendering of the entire page and
1404
replace it with a file download.</p>
1406
<p>The final goal of web.widgets is to provide a framework
1407
which encourages the development of usable library code. Too
1408
much web-based code is thrown away due to its particular
1409
environment requirements or stylistic preconceptions it carries
1410
with it. The goal is to combine the fast-and-loose iterative
1411
development cycle of PHP with the ease of installation and use
1412
of Zope's "Product" plugins.</p>
1414
<h3>Things That Twisted Does Not Do</h3>
1416
<p>It is unfortunately well beyond the scope of this paper to
1417
cover all the functionality that Twisted provides, but it
1418
serves as a good overview. It may seem as though twisted does
1419
anything and everything, but there are certain features we
1420
never plan to implement because they are simply outside the
1421
scope of the project.</p>
1423
<p>Despite the multiple ways to publish and access objects,
1424
Twisted does not have or support an interface definition
1425
language. Some developers on the Twisted project have
1426
experience with remote object interfaces that require explicit
1427
specification of all datatypes during the design of an object's
1428
interface. We feel that such interfaces are in the spirit of
1429
statically-typed languages, and are therefore suited to the
1430
domain of problems where statically-typed languages excel.
1431
Twisted has no plans to implement a protocol schema or static
1432
type-checking mechanism, as the efficiency gained by such an
1433
approach would be quickly lost again by requiring the type
1434
conversion between Python's dynamic types and the protocol's
1435
static ones. Since one of the key advantages of Python is its
1436
extremely flexible dynamic type system, we felt that a
1437
dynamically typed approach to protocol design would share some
1438
of those advantages.</p>
1440
<p>Twisted does not assume that all data is stored in a
1441
relational database, or even an efficient object database.
1442
Currently, Twisted's configuration state is all stored in
1443
memory at run-time, and the persistent parts of it are pickled
1444
at one go. There are no plans to move the configuration objects
1445
into a "real" database, as we feel it is easier to keep a naive
1446
form of persistence for the default case and let
1447
application-specific persistence mechanisms handle persistence.
1448
Consequently, there is no object-relational mapping in Twisted;
1449
<code>twisted.enterprise</code> is an interface to the
1450
relational paradigm, not an object-oriented layer over it.</p>
1452
<p>There are other things that Twisted will not do as well, but
1453
these have been frequently discussed as possibilities for it.
1454
The general rule of thumb is that if something will increase
1455
the required installation overhead, then Twisted will probably
1456
not do it. Optional additions that enhance integration with
1457
external systems are always welcome: for example, database
1458
drivers for Twisted or a CORBA IDL for PB objects.</p>
1460
<h3>Future Directions</h3>
1462
<p>Twisted is still a work in progress. The number of protocols
1463
in the world is infinite for all practical purposes, and it
1464
would be nice to have a central repository of event-based
1465
protocol implementations. Better integration with frameworks
1466
and operating systems is also a goal. Examples for integration
1467
opportunities are automatic creation of installer for "tap"
1468
files (for Red Hat Packager-based distributions, FreeBSD's
1469
package management system or Microsoft Windows(tm) installers),
1470
and integration with other event-dispatch mechanisms, such as
1471
win32's native message dispatch.</p>
1473
<p>A still-nascent feature of Twisted, which this paper only
1474
touches briefly upon, is <code>twisted.enterprise</code>: it is
1475
planned that Twisted will have first-class database support
1476
some time in the near future. In particular, integration
1477
between twisted.web and twisted.enterprise to allow developers
1478
to have SQL conveniences that they are used to from other
1481
<p>Another direction that we hope Twisted will progress in is
1482
standardization and porting of PB as a messaging protocol. Some
1483
progress has already been made in that direction, with XEmacs
1484
integration nearly ready for release as of this writing.</p>
1486
<p>Tighter integration of protocols is also a future goal, such
1487
an FTP server that can serve the same resources as a web
1488
server, or a web server that allows users to change their POP3
1489
password. While Twisted is already a very tightly integrated
1490
framework, there is always room for more integration. Of
1491
course, all this should be done in a flexible way, so the
1492
end-user will choose which components to use -- and have those
1493
components work well together.</p>
1495
<h3>Conclusions</h3>
1497
<p>As shown, Twisted provides a lot of functionality to the
1498
Python network programmer, while trying to be in his way as
1499
little as possible. Twisted gives good tools for both someone
1500
trying to implement a new protocol, or someone trying to use an
1501
existing protocol. Twisted allows developers to prototype and
1502
develop object communication models with PB, without designing
1503
a byte-level protocol. Twisted tries to have an easy way to
1504
record useful deployment options, via the
1505
<code>twisted.tap</code> and plugin mechanisms, while making it
1506
easy to generate new forms of deployment. And last but not
1507
least, even Twisted is written in a high-level language and
1508
uses its dynamic facilities to give an easy API, it has
1509
performance which is good enough for most situations -- for
1510
example, the web server can easily saturate a T1 line serving
1511
dynamic requests on low-end machines.</p>
1513
<p>While still an active project, Twisted can already used for
1514
production programs. Twisted can be downloaded from the main
1515
Twisted site (http://www.twistedmatrix.com) where there is also
1516
documentation for using and programming Twisted.</p>
1518
<h3>Acknowledgements</h3>
1520
<p>We wish to thank Sean Riley, Allen Short, Chris Armstrong,
1521
Paul Swartz, Jürgen Hermann, Benjamin Bruheim, Travis B.
1522
Hartwell, and Itamar Shtull-Trauring for being a part of the
1523
Twisted development team with us.</p>
1525
<p>Thanks also to Jason Asbahr, Tommi Virtanen, Gavin Cooper,
1526
Erno Kuusela, Nick Moffit, Jeremy Fincher, Jerry Hebert, Keith
1527
Zaback, Matthew Walker, and Dan Moniz, for providing insight,
1528
commentary, bandwidth, crazy ideas, and bug-fixes (in no
1529
particular order) to the Twisted team.</p>
1534
<li>The Twisted site, http://www.twistedmatrix.com</li>
1536
<li>Douglas Schmidt, Michael Stal, Hans Rohnert and Frank
1537
Buschmann, Pattern-Oriented Software Architecture, Volume 2,
1538
Patterns for Concurrent and Networked Objects, John Wiley
1541
<li>Abhishek Chandra, David Mosberger, Scalability of Linux
1542
Event-Dispatch Mechanisms, USENIX 2001,
1543
http://lass.cs.umass.edu/~abhishek/papers/usenix01/paper.ps</li>
1545
<li>Protocol specifications, http://www.rfc-editor.com</li>
1547
<li>The Twisted Philosophical FAQ,
1548
http://www.twistedmatrix.com/page.epy/twistedphil.html</li>
1550
<li>Twisted Advocacy,
1551
http://www.twistedmatrix.com/page.epy/whytwisted.html</li>
1553
<li>Medusa, http://www.nightmare.com/medusa/index.html</li>
1555
<li>Using Spreadable Web Servers,
1556
http://www.twistedmatrix.com/users/jh.twistd/python/moin.cgi/TwistedWeb</li>
1558
<li>Twisted.Spread implementations for other languages,
1559
http://www.twistedmatrix.com/users/washort/</li>
1561
<li>PHP: Hypertext Preprocessor, http://www.php.net/</li>
1563
<li>The Z Object Publishing Environment,
1564
http://www.zope.org/, http://zope.com/</li>
added
removed
vendor/Twisted-10.0.0/doc/historic/ipc10paper.html
