Writing Servers in Twisted
Concepts
Twisted is an async framework, so remember that none of your code can
block. You'll get notified about new data coming in, you'll write small
functions which deal with it and end so something else can do its thing.
A Twisted server consists of several parts:
- twisted.main.Application
- This class is Twisted's connection to the world. Usually, most writers
do not deal with it (see below, on deployment considerations).
- twisted.protocols.protocol.Factory
- 90% of the time, this class will be good enough. This class seats
inside an application, listens to a port, and builds a protocol for
each new connection. The default buildProtocol calls the attribute
"protocol" (which is usually a class), and then initializes the
"factory" attribute. Overriding buildProtocol is easy: just return
something which follows the protocol interface.
- twisted.protocols.protocol.Protocol
- While there is no need to derive from this class to write a protocol
handler, it usually makes your work easier. The next section will deal
with writing a new protocol, since this is where most of your code will
be.
Writing protocols
The protocol interface has several methods:
- def makeConnection(self, transport, server = None)
- This method is already implemented inside
twisted.protocols.protocol.Protocol. It will initialize your protocol
with the "connected", "transport" and "server" attributes, and call
"connectionMade". If you want to do connection-specific initializing,
the right way to do it is to implement "connectionMade".
- def connectionMade(self)
- Called when a connection to the server has been established. Here you
should set up attributes, send greetings and similar things.
Note that for "clientish" connections, you will probably want to send
the initial request to the server here, unless the server sends a greeting,
in which case you would set up a callback to the greeting.
- def dataReceived(self, data)
- The protocol's raison d'etre: some data was sent by the other side.
You will probably want to handle it in some way.
There are a couple of Protocol derived classes which override this
method to provide a higher level view of the data - netstrings,
lines and length-prefixed messages.
- def connectionLost(self)
- The low-level connection is lost. Clean up after yourself.
- def connectionFailed(self)
- The connection cannot be made. This will only be called on client
protocols.
Transports
One of the most important instance variables of the Protocol class is
"transport", which must follow the transport interface. This is what
lets protocol classes talk to the world.
Transports have the following methods:
- .write(data)
- Write the data when next the connection is available for writing.
This method will never block, but it might not send the data right
away.
- .loseConnection()
- When there is no more pending data to write, close the connection
down.
- .getPeer()
- Returns a tuple. The first element of the tuple is a string describing
the kind of the transport, with 'INET' being the identifier of good
old TCP sockets.
Writing New Transport Types
I will concentrate here on server transports.
Writing a transport type is usually a two-step procedure. First, you
will need to write the port-like class. The class's constructor should
take, among other transport-specific administrative data (for example,
for TCP ports this could be the numeric port and the addresses to listen
on), a factory. The only thing that should be assumed by the class
is that the given factory has a .buildProtocol method which creates
protocol-interface class.
This class should be pickle-safe, which means any process specific
information (such as open files) should be thrown out by the __getstate__
method.
The class should have a .startListening() method, which should take whatever
action it needs to open a port for listening. It should also have a
.fileno method, which will return a valid file descriptor. When this
file descriptor is valid for reading, the .doRead method will be called.
Usually, this method will create a transport, call the factory's
.buildProtocol and call .makeConnection on the protocol with the transport
as an argument.
The transports written for every connection and passed to protocols'
.makeConnection must follow the transport interface:
- .write(data)
- Queue some data to write. Do not block.
- .loseConnection()
- When there will be no more data to write, close down
the connection.
- .getPeer()
- Return a tuple describing your peer. This should be a tuple -
(TYPE, ...), where the ... depend on your connection's semantics.
Use common sense. So far, for files, it is ('file', filename), for
TCP sockets it is ('INET', address, port) and for UNIX-domain
sockets it is ('UNIX', path). You can probably count on 'INET6'
being IPv6 TCP sockets.
In addition, the select loop will call the following methods:
- .doWrite()
- Flush the data you have, in a non-blocking manner.
- .doRead()
- Read data, and call the protocol's .dataReceived method with
this chunk.
- .connectionLost()
- If either .doRead() or .doWrite() return CONNECTION_LOST,
this method will be called by the select loop. It should clean
up any OS-level resources, and call the protocol's .connectionLost()
Deployment
Most protocols are deployed using the "tap" mechanism, which hides many
none interesting details. Tap-based deployment works by writing a module
in the twisted.tap package which is compatible to the tap interface:
- The Options class
- This should inherit from twised.python.usage.Options and handle valid
options.
It must be called Options. The next section will deal with
writing Option classes.
- The usage_message string
- This should be a helpful multiline message which would be displayed
when the user asks for help.
- The getPorts(app, config) function
- The function gets a twisted.main.Application and an instance of
the Options class defined, and should return an array of two-tuples
(port, Factory). See next section about how to get the command-line
options from an Option instance.
Writing Option Class
- optStrings
- This is a list of 3-lists, each should be [long_name, flag, default].
This will set the "long_name" attribute to the value of that option,
or to default, if it was not be given on the command line.
- optFlags
- This is a list of 2-lists, each should be [long_name, flag].
This will set the "long_name" attribute to 1 if it was given on the
command line, else to 0.
- opt_* methods
- If the method takes no arguments (except from self), it will be
called once for every time the part after the "opt_" is given.
If it takes one argument, then it will be the value for the option
given on the command line.
opt_* methods are called in the order the arguments are given on
the command line.
Finishing Touches
OK.
You have written a protocol, a factory, a twisted.tap submodule which
parses command line options and returns a valid value from getPorts.
What now?
- Run "mktap <your twisted.tap submodule> <valid options>". This will
create a file named "<your twisted.tap submodule>.tap", which is a
pickle containing a twisted.main.Application.
- Run "twistd -f <your twisted.tap.submodule>.tap". twistd knows to turn
pickled twisted.main.Application into a living, breathing application.
Some common arguments are "-l <logfile>" and "-n" (no daemonize).
That is it. You have written a twisted server.
Post-finishing Touches
So, you are happily running a twisted server via twistd, when suddenly,
you wish you could add another HTTP server running on port 833. Or maybe
you want your living HTTP server to become virtual-domains based. What
are you going to do?
Well, you have several options.
Warning: some of this stuff assumes a 0.9.6 or later version
of twisted. If you are running an older version, you should upgrade
To add another HTTP server to your tap, just use mktap's --append option.
Note how that does not require support in the twisted.tap.whatever
module, so it will just work even for your newly written protocol.
Changing the HTTP is a bit more complicated. What you will certainly
want to do is add a telnet server to your tap. Just use "mktap --append telnet".
It is recommended to give the username/password arguments and not use
the defaults, for obvious reasons.
Now that you have the telnet server, you can just telnet into your
application and change things there on the run.
When you telnet, you are given, after the usual name/password login box,
absolutely no prompt. Hopefully, this will change. The first thing you
will want to do is import __main__. In __main__, there is a global
named "application". This is your application. If you want, you can add
servers to it on the fly:
import __main__
from twisted.web import static, server
file = static.File("/var/www")
factory = server.Site(file)
__main__.application.listenTCP(8443, factory)
That's it! It really is this easy.
Now, let us say you want to configure the server to serve another
directory instead.
import __main__
__main__.application.ports
-> [<twisted.protocols.telnet.ShellFactory on 4040>, <twisted.web.server.Site on 8443>]
So it is the second port. Great!
__main__.application.ports[1].factory.resource.path = 'static'
In similar ways, you can configure your factory's resource to be
a vhost.NamedVirtualHost, configure the name servers, and leave it
running.
If you want to save known good configurations, just use
__main__.application.save("known-good")
Which will save the application to <original tap file>+"known-good.tap"
On shutdown, it will behave as if __main__.application.save("shutdown")
has been issued.