1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
4
<!-- $Id: 6-eventserver.html,v 2.17.2.6 2009/03/28 23:12:54 irmen Exp $ -->
5
<title>PYRO - Event Server</title>
6
<link rel="stylesheet" type="text/css" href="pyromanual_print.css" media="print">
7
<link rel="stylesheet" type="text/css" href="pyromanual.css" media="screen">
15
<td align="left"><a href="5-nameserver.html"><previous</a> | <a href="PyroManual.html">contents</a> |
16
<a href="7-features.html">next></a></td>
18
<td align="right">Pyro Manual</td>
24
<h2>6. Pyro Event Server</h2>
27
<li><a href="#intro">Introduction</a></li>
29
<li><a href="#starting">Starting the Event Server</a></li>
31
<li><a href="#publish">Using the Event Server (publish)</a></li>
33
<li><a href="#subscribe">Using the Event Server (subscribe)</a></li>
35
<li><a href="#thread">Threads, Subscribers and Queues</a></li>
37
<li><a href="#examples">Examples</a></li>
40
<h3><a name="event" id="event"></a><a name="intro" id="intro"></a>Introduction</h3>
42
<p>In various situations it is needed that the servers and the clients are decoupled. In abstract terms this means
43
that information producers do not know nor care about the parties that are interested in the information, and the
44
information consumers do not know nor care about the source or sources of the information. All they know is that they
45
produce or consume information on a certain subject.<br></p>
47
<p>Here does the Event Server fit in nicely. It is a third party that controls the flow of information
48
about certain subjects ("events"). A <em>publisher</em> uses the Event Server to publish
49
a message on a specific subject. A
50
<em>subscriber</em> uses the Event Server to subscribe itself to specific subjects, or to a pattern
51
that matches certain subjects. As soon as new information on a subject is produced (an "event" occurs)
52
all subscribers for this subject receive the information. Nobody knows (and cares) about anybody else.<br>
55
<p>It is important to rembember that all events processed by the ES are transient, which means they are not stored.
56
If there is no listener, all events disappear in the void. The store-and-forward programming model is part of a
57
messaging service, which is not what the ES is meant to do. It is also important to know that all subscription data
58
is transient. Once the ES is stopped, all subscriptions are lost. The clients that are subscribed are not notified of
59
this! If no care is taken, they keep on waiting forever for events to occur, because the ES doesn't know about them
62
<p>Usually your subscribers will receive the events in the order they are published. However, this is <em>not
63
guaranteed</em>. If you rely on the exact order of receiving events, you must add some logic to check this (possibly
64
by examining the event's timestamps). The chance of events not arriving in the order they were published is very,
65
very small in a high-performance LAN. Only on very high server load, high network traffic, or a high-latency (WAN?)
66
connection it is likely to occur.</p>
68
<p>Another thing to pay attention to is that the ES does not guarantee delivery of events. As mentioned above, the ES
69
does not have a store-and-forward mechanism, but even if everything is up and running, the ES does <em>not</em>
70
enhance Pyro's way of transporting messages. This means that it's still possible (perhaps due to a network error)
71
that an event gets lost. For reliable, guaranteed, asynchronous message delivery you'll have to look somewhere else,
74
<p>The ES is a multithreaded server and will not work if your Python installation doesn't have thread support.
75
Publications are dispatched to the subscribers in different threads, so they don't block eachother. Please note that
76
events may arrive at your listener in multithreaded fashion! Pyro itself starts another thread in your listener to
77
handle the new event, possibly while the previous one is still being handled. <em>The<code>event</code> method may be
78
called concurrently from several threads.</em> If you can't handle this, you have to use some form of thread locking
79
in your client! (see the <code>threading</code> module on <code>Semaphore</code>), or
80
<code>Pyro.util.getLockObject</code>. <br>
82
<span style="font-weight: bold;">To summarize:<br></span></p>
85
<li>decoupled event listeners and event producers; many-to-many<br></li>
87
<li>topic based communication</li>
89
<li>subscription to unique topics or topic patterns</li>
91
<li>events are transient and will disappear if nobody is listening</li>
93
<li>not a means of guaranteed or asynchronous messaging</li>
95
<li>multithreading mode required<br></li>
98
<h3><a name="starting" id="starting"></a>Starting the Event Server</h3>Start the ES using the <code>pyro-es</code> command
99
from the <code>bin</code> directory (use <code>pyro-es.cmd</code> on windows). You can specify the following
101
pyro-es [-h] [-n hostname] [-p port] [-N] [-i identification]
110
<dd>Change the hostname/ip address the server binds on. Useful with multiple network adapters.</dd>
114
<dd>Change the port number the server uses. (Omit to use Pyro defaults, 0 to let the operating system choose a random port).</dd>
116
<dd>Do not use the Name server</dd>
118
<dt>-i identification</dt>
120
<dd>Specify the authentication passphrase that will be required to connect to this server. If it contains spaces,
121
use quotes around the string. The same identification is also used to connect to other Pyro servers such as the
122
Name Server. (this is required ofcourse when the Name Server has been started with the -i option).</dd>
125
There is also: <code>pyro-essvc</code> (Windows-only Event Server 'NT-service' control scripts)</dt>
127
<dd>- Arguments: [options] install|update|remove|start [...]|stop|restart [...]<br>
128
- On windows NT (2000/XP) systems, it's possible to register and start the Event server as a NT-service. You'll
129
have to use the <code>essvc.cmd</code> script to register it as a service. Make sure you have Pyro properly
130
installed in your Python's site-packages. Or make sure to register the service using an account with the correct
131
PYTHONPATH setting, so that Pyro can be located. The ES service logs to <code>C:\Pyro_ES_svc.log</code> where C: is
132
your system drive.<br>
133
You can configure command line arguments for this service in the Registry. The key is:
134
<code>HKLM\System\CurrentControlSet\Services\PyroES</code>, and the value under that key is:
135
<code>PyroServiceArguments</code> (REG_SZ, it will be asked and created for you when doing a <code>essvc.cmd
136
install</code> from a command prompt).<br>
137
<em>Running the ES as a windows NT service it not well supported.</em></dd>
139
<dt>You can also use <code>python -m</code> to start it:</dt>
140
<dd><code>python -m Pyro.EventService.Server</code></dd>
143
<p><strong>Like the Name Server, if you want to start the Event Server from within your own program</strong>,
144
you can ofcourse start it by executing the start script mentioned above. You could also use the
145
<code>EventServiceStarter</code> class from the <code>Pyro.EventService.Server</code> module to start
146
it directly (this is what the script also does). Be sure to start it in a separate process or thread
147
because it will run in its own endless loop. Have a look at the "AllInOne" example to see how
148
you can start the Event Server using the
149
<code>EventServiceStarter</code> class.<br>
150
You probably have to wait until the ES has been fully started, call the <code>waitUntilStarted()</code> method on the
151
starter object. It returns true if the ES has been started, false if it is not yet ready. You can provide a timeout
152
argument (in seconds).</p>
154
<p>To start the ES you will first have to start the Name Server because the ES needs that to register itself. After
155
starting the ES you will then see something like this:<br></p>
156
<pre style="margin-left: 40px;">
157
*** Pyro Event Server ***<
158
Pyro Server Initialized. Using Pyro V3.2
159
URI= PYRO://192.168.1.40:7766/c0a8012804bc0c96774244d7d79d5db3
160
Event Server started.
163
<h4>Configuration options<br></h4>There are two config options specifically for the ES:
164
<code>PYRO_ES_QUEUESIZE</code> and <code>PYRO_ES_BLOCKQUEUE</code>. Read about them in the <a href=
165
"3-install.html">Installation and Configuration</a> chapter. By default, the ES will allocate moderately sized queues
166
for subscribers, and publishers will block if such a queue becomes full (so no events get lost). You might want to
167
change this behavior. Every subscriber has its own queue. So if the queue of a slow subscriber fills up, other
168
subscribers are still serviced nicely. By setting <code>PYRO_ES_BLOCKQUEUE</code> to <code>0</code>, new messages for
169
full queues are lost. This may be a way to allow slow subscribers to catch up, because new messages are put in the
170
queue when there is room again. Note that only messages to the slow or frozen subscribers are lost, normal running subscribers
171
still receive these messages.</p>
173
<h3><a name="publish" id="publish"></a>Using the Event Server (publish)<br></h3>
174
The ES is just a regular Pyro object,
175
with a few helper classes. Its name (to look it up in the Name Server) is available in
176
<code>Pyro.constants.EVENTSERVER_NAME</code>. All subjects are case insensitive, so if you publish
177
something on the "stockquotes" channel it is the same as if you published it on the "STOCKQuotes" channel.<br>
179
<p>To publish an event on a certain topic, you need to have a Pyro proxy object for the ES, and then call the
180
<code>publish</code> method:<code>publish(subjects, message)</code> where <code>subjects</code> is a subject name or
181
a sequence of one or more subject names (strings), and <code>message</code> is the actual message. The message can be
182
any Python object (as long as it can be pickled):<br></p>
183
<pre style="margin-left: 40px;">
185
import Pyro.constants
186
Pyro.core.initClient()
187
es = Pyro.core.getProxyForURI("PYRONAME://"+Pyro.constants.EVENTSERVER_NAME)
188
es.publish("StockQuotes",( "SUN", 22.44 ) )
191
<p>If you think this is too much work, or if you want to abstract from the Pyro details, you can use the
192
<code>Publisher</code> base class that is provided in <code>Pyro.EventService.Clients.</code> Subclass your event
193
publishers from this class. The init takes care of locating the ES, and you can just call the <code>publish(subjects,
194
message)</code> method of the base class. No ES proxy code needed:</p>
195
<pre style="margin-left: 40px;">
196
import Pyro.EventService.Clients
198
class StockPublisher(Pyro.EventService.Clients.Publisher):
200
Pyro.EventService.Clients.Publisher.__init__(self)
201
def publishQuote(self, symbol, quote):
202
self.publish("StockQuotes", ( symbol, quote) )
204
sp = StockPublisher()
205
sp.publishQuote("SUN", 22.44)
208
<h4>Authentication passphrase</h4>The <code>__init__</code> of both the Publisher and the Subscriber takes an
209
optional <code>ident</code> argument. Use this to specify the authentication passphrase that will be used to connect
210
to the ES (and also to connect to the Name Server).
212
<h4>Not using the name server</h4>
213
The <code>__init__</code> of both the Publisher and the Subscriber takes an
214
optional <code>esURI</code> argument. Set it to the URI of the Event Server (string format)
215
if you don't have a name server running. Look at the 'stockquotes' example to see how this can be done.
216
Note that the Event service usually prints its URI when started.
218
<h3><a name="subscribe" id="subscribe"></a>Using the Event Server (subscribe)</h3>
219
As pointed out above, the ES is
220
just a regular Pyro object, with a few helper classes. Its name (to look it up in the Name Server)
222
<code>Pyro.constants.EVENTSERVER_NAME</code>. All subjects are case insensitive, so if you publish
223
something on the "stockquotes" channel it is the same as if you published it on the "STOCKQuotes" channel.<br>
225
<p>Event subscribers are a little more involved that event publishers. This is becaue they are full-blown
226
Pyro server objects that receive calls from the ES when an event is published on one of the topics
227
you've subscribed to! Therefore, your clients (subscribers) need to call the Pyro daemon's <code>handleRequests</code> or
228
<code>requestLoop</code> (just like a Pyro server). They also have to call <code>Pyro.core.initServer()</code>because
229
they also act as a Pyro server. Furthermore, they usually have to run as a multithreaded server, because
230
the ES may call it as soon as a new event arrives and you are not done processing the previous event.
231
Single-threaded servers will build up a backlog of undelivered events if this happens. You still get
232
all events (with the original timestamp - so you could skip events that "have expired" to catch
233
up). You can change this behavior by changing the before mentioned config items.</p>
235
<h4>Subscribing to receive information</h4>The Event Server has a few important methods that you'll be using to
241
<td><code>subscribe(subjects, subscriber)</code></td>
243
<td>Subscribe to events. <code>subjects</code> is a subject name or a sequence of one or more subject names
244
(strings), and <code>subscriber</code> is <em>a proxy</em> for your subscriber object</td>
248
<td><code>subscribeMatch(subjectPatterns, subscriber)</code></td>
250
<td>Subscribe to events based on patterns. <code>subjectPatterns</code> is a subject <span style=
251
"font-style: italic;">pattern</span> or a sequence of one or more subject patterns (strings), and
252
<code>subscriber</code> is <em>a proxy</em> for your subscriber object</td>
256
<td><code>unsubscribe(subjects, subscriber)</code></td>
258
<td>Unsubscribe from subjects. <code>subjects</code> is a subject or subject <span style=
259
"font-style: italic;">pattern</span> or a sequence thereof, and <code>subscriber</code> is <em>a proxy</em> for
260
your subscriber object</td>
265
<p>But first, create a subscriber object, which must be a Pyro object (or use delegation). The subscriber object
266
should have an <code>event(self, event)</code> method. This method is called by the ES if a new event arrives on a
267
channel you subscribed to. <code>event</code> is a <code>Pyro.EventService.Event</code> object, which has the
268
following attributes:</p>
273
<td><code>msg</code></td>
275
<td>the actual message that was published. Can be any Python object.</td>
279
<td><code>subject</code></td>
281
<td>the subject (string) on which the message was published. (topic name)<br></td>
285
<td><code>time</code></td>
287
<td>the event's timestamp (from the server - synchronised for all subscribers). A float, taken from
288
<code>time.time()</code><br></td>
293
<p>To subscribe, call the <code>subscribe</code> method of the ES with the desired subject(s) and a proxy for your
294
subscriber object. If you want to subscribe to multiple subjects based on <strong>pattern matching,</strong> call the
295
<code>subscribeMatch</code> method instead with the desired subject pattern(s) and a proxy for your subscriber
296
object. The patterns are standard <code>re</code>-style regex expressions. See the standard <code>re</code> module
297
for more information. The pattern <code>'^STOCKQUOTE\\.S.*$'</code> matches STOCKQUOTE.SUN, STOCKQUOTE.SAP but not
298
STOCKQUOTE.IBM, NYSE.STOCKQUOTE.SUN etcetera. Once more: the subjects are case insensitive. The patterns are matched
299
case insensitive too.</p>
301
<p>To unsubscribe, call the <code>unsubscribe</code> method with the subject(s) or pattern(s) you want to unsubscribe
302
from, and a proxy for the subscriber object that has been previously subscribed. This will remove the subscriber from
303
the subscription list and also from the pattern match list if the subject occurs as a pattern there. The ES
304
(actually, Pyro) is smart enough to see if multiple (different) proxy objects point to the same subscriber object and
305
will act correctly.<br></p>
307
<h4>Using the Subscriber base class from the Event Server</h4>As you can see it can be a bit complex to get your
308
subcribers up and running. An easier way to do this is to use the <code>Subscriber</code> base class provided in
309
<code>Pyro.EventService.Clients. </code> Subclass your event listeners (subscribers) from this class. The init takes
310
care of locating the ES, and you can just call the
311
<code>subscribe(subjects)</code>,<code>subscribeMatch(subjectPatterns)</code> and <code>unsubscribe(subjects)</code>
312
methods on the object itself. No ES proxy code needed. This base class also starts a Pyro daemon and by calling
313
<code>listen()</code>, your code starts listening on incoming events. When you want to abort the event loop, you have
314
to call <code>self.abort()</code> from within the event handler method.
316
<p>The multithreading of the <code>event</code> method can be controlled using the
317
<code>setThreading(threading)</code> method. If you <code>threading=</code>0, the threading will be switched off (it
318
is on by default unless otherwise configured). Your events will then arrive purely sequentially, after processing
319
each event. Call this method before entering the <code>requestLoop</code> or <code>handleRequests</code> or
320
<code>listen.</code></p>
322
<p>A minimalistic event listener that prints the stockquote events published by the example code above:<br></p>
323
<pre style="margin-left: 40px;">
324
from Pyro.EventService.Clients import Subscriber<br>
325
class StockSubscriber(Subscriber):
327
Subscriber.__init__(self)
328
self.subscribe("StockQuotes")
329
def event(self, event):
330
print "Got a stockquote: %s=%f" % (event.msg)
332
sub = StockSubscriber()
335
<h4>Authentication passphrase</h4>
337
<p>The <code>__init__</code> of both the Publisher and the Subscriber takes an optional <code>ident</code> argument.
338
Use this to specify the authentication passphrase that will be used to connect to the ES (and also to connect to the
339
Name Server).<br></p>
341
<h3><a name="thread" id="thread"></a>Threads, Subscribers and Queues</h3>As pointed out above the events are
342
delivered to your subscribers in a multithreaded way. Your subscriber may still be processing an event when the next
343
one arrives. Use the <code>setThreading(threading)</code> method of the <code>Subscriber</code> base class to control
344
the threading. If you set threading=0, the threading will be switched off (it is on by default). But a better way to
345
process events sequentially is to use Python's <code>Queue</code> module: you create a Queue in your
346
subscriber process that is filled with arriving events, and you have a single event consumer process that takes
347
events out of the queue one-by-one:
349
<table align="center" class="noborder">
351
<td style="padding: 4pt; background: maroon; color: yellow; text-align: center;">Pyro Event Server</td>
353
<td><em>multithreaded</em></td>
357
<td style="text-align: center; font-size:150%;">↓</td>
363
<td style="padding: 4pt; background: navy; color: white; text-align: center;">Subscriber(s)</td>
365
<td><em>multithreaded</em></td>
369
<td style="text-align: center; font-size:150%;">↓</td>
375
<td style="padding: 4pt; background: teal; color: white; text-align: center;"><code>Queue.Queue</code></td>
381
<td style="text-align: center; font-size:150%;">↓</td>
387
<td style="padding: 4pt; background: navy; color: white; text-align: center;">Consumer/Worker</td>
389
<td><em>singlethreaded</em></td>
393
<h3><a name="examples" id="examples"></a>Examples</h3>
394
To see how you use the ES, have a look at the "stockquotes" and
395
"countingcars" examples. Also have a look at the client skeleton code that comes with the ES.
396
To exercise the ES to the max, have a look at the fully threaded "stresstest" example. To see
397
how to start and use the ES from within your own program, have a look at the "AllInOne" example.<br>
403
<td align="left"><a href="5-nameserver.html"><previous</a> | <a href="PyroManual.html">contents</a> |
404
<a href="7-features.html">next></a></td>
406
<td align="right">Pyro Manual</td>
added
removed
docs/6-eventserver.html
