~free.ekanayaka/landscape-client/lucid-1.5.4-0ubuntu0.10.04.0
» Revision
29
: landscape/lib/twisted_amp.py
« back to all changes in this revision
Viewing changes to landscape/lib/twisted_amp.py
- browse files at revision 29
- compare with another revision
- compare with revision 28
- download diff
- download tarball
- view history from revision 29
- Committer: Bazaar Package Importer
- Author(s): Free Ekanayaka
- Date: 2010-06-28 18:07:18 UTC
- mfrom: (1.2.5 upstream)
- Revision ID: james.westby@ubuntu.com-20100628180718-vytyqgbtkiirv5sb
Tags: 1.5.2.1-0ubuntu0.10.04.0
Filter duplicate network interfaces in get_active_interfaces (LP: #597000)
- files added:
- landscape/amp.py
- files removed:
- landscape/broker/broker.py
- files modified:
- Makefile
added
removed
landscape/lib/twisted_amp.py
1
# -*- test-case-name: twisted.test.test_amp -*-
2
# Copyright (c) 2005 Divmod, Inc.
3
# Copyright (c) 2007 Twisted Matrix Laboratories.
4
# See LICENSE for details.
5
6
"""
7
This module implements AMP, the Asynchronous Messaging Protocol.
8
9
AMP is a protocol for sending multiple asynchronous request/response pairs over
10
the same connection. Requests and responses are both collections of key/value
11
pairs.
12
13
AMP is a very simple protocol which is not an application. This module is a
14
"protocol construction kit" of sorts; it attempts to be the simplest wire-level
15
implementation of Deferreds. AMP provides the following base-level features:
16
17
- Asynchronous request/response handling (hence the name)
18
19
- Requests and responses are both key/value pairs
20
21
- Binary transfer of all data: all data is length-prefixed. Your
22
application will never need to worry about quoting.
23
24
- Command dispatching (like HTTP Verbs): the protocol is extensible, and
25
multiple AMP sub-protocols can be grouped together easily.
26
27
The protocol implementation also provides a few additional features which are
28
not part of the core wire protocol, but are nevertheless very useful:
29
30
- Tight TLS integration, with an included StartTLS command.
31
32
- Handshaking to other protocols: because AMP has well-defined message
33
boundaries and maintains all incoming and outgoing requests for you, you
34
can start a connection over AMP and then switch to another protocol.
35
This makes it ideal for firewall-traversal applications where you may
36
have only one forwarded port but multiple applications that want to use
37
it.
38
39
Using AMP with Twisted is simple. Each message is a command, with a response.
40
You begin by defining a command type. Commands specify their input and output
41
in terms of the types that they expect to see in the request and response
42
key-value pairs. Here's an example of a command that adds two integers, 'a'
43
and 'b'::
44
45
class Sum(amp.Command):
46
arguments = [('a', amp.Integer()),
47
('b', amp.Integer())]
48
response = [('total', amp.Integer())]
49
50
Once you have specified a command, you need to make it part of a protocol, and
51
define a responder for it. Here's a 'JustSum' protocol that includes a
52
responder for our 'Sum' command::
53
54
class JustSum(amp.AMP):
55
def sum(self, a, b):
56
total = a + b
57
print 'Did a sum: %d + %d = %d' % (a, b, total)
58
return {'total': total}
59
Sum.responder(sum)
60
61
Later, when you want to actually do a sum, the following expression will return
62
a Deferred which will fire with the result::
63
64
ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
65
lambda p: p.callRemote(Sum, a=13, b=81)).addCallback(
66
lambda result: result['total'])
67
68
You can also define the propagation of specific errors in AMP. For example,
69
for the slightly more complicated case of division, we might have to deal with
70
division by zero::
71
72
class Divide(amp.Command):
73
arguments = [('numerator', amp.Integer()),
74
('denominator', amp.Integer())]
75
response = [('result', amp.Float())]
76
errors = {ZeroDivisionError: 'ZERO_DIVISION'}
77
78
The 'errors' mapping here tells AMP that if a responder to Divide emits a
79
L{ZeroDivisionError}, then the other side should be informed that an error of
80
the type 'ZERO_DIVISION' has occurred. Writing a responder which takes
81
advantage of this is very simple - just raise your exception normally::
82
83
class JustDivide(amp.AMP):
84
def divide(self, numerator, denominator):
85
result = numerator / denominator
86
print 'Divided: %d / %d = %d' % (numerator, denominator, total)
87
return {'result': result}
88
Divide.responder(divide)
89
90
On the client side, the errors mapping will be used to determine what the
91
'ZERO_DIVISION' error means, and translated into an asynchronous exception,
92
which can be handled normally as any L{Deferred} would be::
93
94
def trapZero(result):
95
result.trap(ZeroDivisionError)
96
print "Divided by zero: returning INF"
97
return 1e1000
98
ClientCreator(reactor, amp.AMP).connectTCP(...).addCallback(
99
lambda p: p.callRemote(Divide, numerator=1234,
100
denominator=0)
101
).addErrback(trapZero)
102
103
For a complete, runnable example of both of these commands, see the files in
104
the Twisted repository::
105
106
doc/core/examples/ampserver.py
107
doc/core/examples/ampclient.py
108
109
On the wire, AMP is a protocol which uses 2-byte lengths to prefix keys and
110
values, and empty keys to separate messages::
111
112
<2-byte length><key><2-byte length><value>
113
<2-byte length><key><2-byte length><value>
114
...
115
<2-byte length><key><2-byte length><value>
116
<NUL><NUL> # Empty Key == End of Message
117
118
And so on. Because it's tedious to refer to lengths and NULs constantly, the
119
documentation will refer to packets as if they were newline delimited, like
120
so::
121
122
C: _command: sum
123
C: _ask: ef639e5c892ccb54
124
C: a: 13
125
C: b: 81
126
127
S: _answer: ef639e5c892ccb54
128
S: total: 94
129
130
Notes:
131
132
Values are limited to the maximum encodable size in a 16-bit length, 65535
133
bytes.
134
135
Keys are limited to the maximum encodable size in a 8-bit length, 255 bytes.
136
Note that we still use 2-byte lengths to encode keys. This small redundancy
137
has several features:
138
139
- If an implementation becomes confused and starts emitting corrupt data,
140
or gets keys confused with values, many common errors will be
141
signalled immediately instead of delivering obviously corrupt packets.
142
143
- A single NUL will separate every key, and a double NUL separates
144
messages. This provides some redundancy when debugging traffic dumps.
145
146
- NULs will be present at regular intervals along the protocol, providing
147
some padding for otherwise braindead C implementations of the protocol,
148
so that <stdio.h> string functions will see the NUL and stop.
149
150
- This makes it possible to run an AMP server on a port also used by a
151
plain-text protocol, and easily distinguish between non-AMP clients (like
152
web browsers) which issue non-NUL as the first byte, and AMP clients,
153
which always issue NUL as the first byte.
154
155
"""
156
157
__metaclass__ = type
158
159
import types, warnings
160
161
from cStringIO import StringIO
162
from struct import pack
163
164
from zope.interface import Interface, implements
165
166
from twisted.python.reflect import accumulateClassDict
167
from twisted.python.failure import Failure
168
from twisted.python import log, filepath
169
170
from twisted.internet.main import CONNECTION_LOST
171
from twisted.internet.error import ConnectionLost
172
from twisted.internet.defer import Deferred, maybeDeferred, fail
173
from twisted.protocols.basic import Int16StringReceiver, StatefulStringProtocol
174
175
#from twisted.internet._sslverify import problemsFromTransport
176
177
# I'd like this to use the exposed public API, but for some reason, when it was
178
# moved, these names were not exposed by internet.ssl.
179
180
#from twisted.internet.ssl import CertificateOptions, Certificate, DN, KeyPair
181
182
ASK = '_ask'
183
ANSWER = '_answer'
184
COMMAND = '_command'
185
ERROR = '_error'
186
ERROR_CODE = '_error_code'
187
ERROR_DESCRIPTION = '_error_description'
188
UNKNOWN_ERROR_CODE = 'UNKNOWN'
189
UNHANDLED_ERROR_CODE = 'UNHANDLED'
190
191
MAX_KEY_LENGTH = 0xff
192
MAX_VALUE_LENGTH = 0xffff
193
194
195
class IBoxSender(Interface):
196
"""
197
A transport which can send L{AmpBox} objects.
198
"""
199
200
def sendBox(box):
201
"""
202
Send an L{AmpBox}.
203
204
@raise ProtocolSwitched: if the underlying protocol has been
205
switched.
206
207
@raise ConnectionLost: if the underlying connection has already been
208
lost.
209
"""
210
211
def unhandledError(failure):
212
"""
213
An unhandled error occurred in response to a box. Log it
214
appropriately.
215
216
@param failure: a L{Failure} describing the error that occurred.
217
"""
218
219
220
221
class IBoxReceiver(Interface):
222
"""
223
An application object which can receive L{AmpBox} objects and dispatch them
224
appropriately.
225
"""
226
227
def startReceivingBoxes(boxSender):
228
"""
229
The L{ampBoxReceived} method will start being called; boxes may be
230
responded to by responding to the given L{IBoxSender}.
231
232
@param boxSender: an L{IBoxSender} provider.
233
"""
234
235
236
def ampBoxReceived(box):
237
"""
238
A box was received from the transport; dispatch it appropriately.
239
"""
240
241
242
def stopReceivingBoxes(reason):
243
"""
244
No further boxes will be received on this connection.
245
246
@type reason: L{Failure}
247
"""
248
249
250
251
class IResponderLocator(Interface):
252
"""
253
An application object which can look up appropriate responder methods for
254
AMP commands.
255
"""
256
257
def locateResponder(self, name):
258
"""
259
Locate a responder method appropriate for the named command.
260
261
@param name: the wire-level name (commandName) of the AMP command to be
262
responded to.
263
264
@return: a 1-argument callable that takes an L{AmpBox} with argument
265
values for the given command, and returns an L{AmpBox} containing
266
argument values for the named command, or a L{Deferred} that fires the
267
same.
268
"""
269
270
271
272
class AmpError(Exception):
273
"""
274
Base class of all Amp-related exceptions.
275
"""
276
277
278
279
class ProtocolSwitched(Exception):
280
"""
281
Connections which have been switched to other protocols can no longer
282
accept traffic at the AMP level. This is raised when you try to send it.
283
"""
284
285
286
287
class OnlyOneTLS(AmpError):
288
"""
289
This is an implementation limitation; TLS may only be started once per
290
connection.
291
"""
292
293
294
295
class NoEmptyBoxes(AmpError):
296
"""
297
You can't have empty boxes on the connection. This is raised when you
298
receive or attempt to send one.
299
"""
300
301
302
303
class InvalidSignature(AmpError):
304
"""
305
You didn't pass all the required arguments.
306
"""
307
308
309
310
class TooLong(AmpError):
311
"""
312
One of the protocol's length limitations was violated.
313
314
@ivar isKey: true if the string being encoded in a key position, false if
315
it was in a value position.
316
317
@ivar isLocal: Was the string encoded locally, or received too long from
318
the network? (It's only physically possible to encode "too long" values on
319
the network for keys.)
320
321
@ivar value: The string that was too long.
322
323
@ivar keyName: If the string being encoded was in a value position, what
324
key was it being encoded for?
325
"""
326
327
def __init__(self, isKey, isLocal, value, keyName=None):
328
AmpError.__init__(self)
329
self.isKey = isKey
330
self.isLocal = isLocal
331
self.value = value
332
self.keyName = keyName
333
334
335
def __repr__(self):
336
hdr = self.isKey and "key" or "value"
337
if not self.isKey:
338
hdr += ' ' + repr(self.keyName)
339
lcl = self.isLocal and "local" or "remote"
340
return "%s %s too long: %d" % (lcl, hdr, len(self.value))
341
342
343
344
class BadLocalReturn(AmpError):
345
"""
346
A bad value was returned from a local command; we were unable to coerce it.
347
"""
348
def __init__(self, message, enclosed):
349
AmpError.__init__(self)
350
self.message = message
351
self.enclosed = enclosed
352
353
354
def __repr__(self):
355
return self.message + " " + self.enclosed.getBriefTraceback()
356
357
__str__ = __repr__
358
359
360
361
class RemoteAmpError(AmpError):
362
"""
363
This error indicates that something went wrong on the remote end of the
364
connection, and the error was serialized and transmitted to you.
365
"""
366
def __init__(self, errorCode, description, fatal=False, local=None):
367
"""Create a remote error with an error code and description.
368
369
@param errorCode: the AMP error code of this error.
370
371
@param description: some text to show to the user.
372
373
@param fatal: a boolean, true if this error should terminate the
374
connection.
375
376
@param local: a local Failure, if one exists.
377
"""
378
if local:
379
localwhat = ' (local)'
380
othertb = local.getBriefTraceback()
381
else:
382
localwhat = ''
383
othertb = ''
384
Exception.__init__(self, "Code<%s>%s: %s%s" % (
385
errorCode, localwhat,
386
description, othertb))
387
self.local = local
388
self.errorCode = errorCode
389
self.description = description
390
self.fatal = fatal
391
392
393
394
class UnknownRemoteError(RemoteAmpError):
395
"""
396
This means that an error whose type we can't identify was raised from the
397
other side.
398
"""
399
def __init__(self, description):
400
errorCode = UNKNOWN_ERROR_CODE
401
RemoteAmpError.__init__(self, errorCode, description)
402
403
404
405
class MalformedAmpBox(AmpError):
406
"""
407
This error indicates that the wire-level protocol was malformed.
408
"""
409
410
411
412
class UnhandledCommand(AmpError):
413
"""
414
A command received via amp could not be dispatched.
415
"""
416
417
418
419
class IncompatibleVersions(AmpError):
420
"""
421
It was impossible to negotiate a compatible version of the protocol with
422
the other end of the connection.
423
"""
424
425
426
PROTOCOL_ERRORS = {UNHANDLED_ERROR_CODE: UnhandledCommand}
427
428
class AmpBox(dict):
429
"""
430
I am a packet in the AMP protocol, much like a regular str:str dictionary.
431
"""
432
__slots__ = [] # be like a regular dictionary, don't magically
433
# acquire a __dict__...
434
435
436
def copy(self):
437
"""
438
Return another AmpBox just like me.
439
"""
440
newBox = self.__class__()
441
newBox.update(self)
442
return newBox
443
444
445
def serialize(self):
446
"""
447
Convert me into a wire-encoded string.
448
449
@return: a str encoded according to the rules described in the module
450
docstring.
451
"""
452
i = self.items()
453
i.sort()
454
L = []
455
w = L.append
456
for k, v in i:
457
if len(k) > MAX_KEY_LENGTH:
458
raise TooLong(True, True, k, None)
459
if len(v) > MAX_VALUE_LENGTH:
460
raise TooLong(False, True, v, k)
461
for kv in k, v:
462
w(pack("!H", len(kv)))
463
w(kv)
464
w(pack("!H", 0))
465
return ''.join(L)
466
467
468
def _sendTo(self, proto):
469
"""
470
Serialize and send this box to a Amp instance. By the time it is being
471
sent, several keys are required. I must have exactly ONE of::
472
473
_ask
474
_answer
475
_error
476
477
If the '_ask' key is set, then the '_command' key must also be
478
set.
479
480
@param proto: an AMP instance.
481
"""
482
proto.sendBox(self)
483
484
def __repr__(self):
485
return 'AmpBox(%s)' % (dict.__repr__(self),)
486
487
# amp.Box => AmpBox
488
489
Box = AmpBox
490
491
class QuitBox(AmpBox):
492
"""
493
I am an AmpBox that, upon being sent, terminates the connection.
494
"""
495
__slots__ = []
496
497
498
def __repr__(self):
499
return 'QuitBox(**%s)' % (super(QuitBox, self).__repr__(),)
500
501
502
def _sendTo(self, proto):
503
"""
504
Immediately call loseConnection after sending.
505
"""
506
super(QuitBox, self)._sendTo(proto)
507
proto.transport.loseConnection()
508
509
510
511
class _SwitchBox(AmpBox):
512
"""
513
Implementation detail of ProtocolSwitchCommand: I am a AmpBox which sets
514
up state for the protocol to switch.
515
"""
516
517
# DON'T set __slots__ here; we do have an attribute.
518
519
def __init__(self, innerProto, **kw):
520
"""
521
Create a _SwitchBox with the protocol to switch to after being sent.
522
523
@param innerProto: the protocol instance to switch to.
524
@type innerProto: an IProtocol provider.
525
"""
526
super(_SwitchBox, self).__init__(**kw)
527
self.innerProto = innerProto
528
529
530
def __repr__(self):
531
return '_SwitchBox(%r, **%s)' % (self.innerProto,
532
dict.__repr__(self),)
533
534
535
def _sendTo(self, proto):
536
"""
537
Send me; I am the last box on the connection. All further traffic will be
538
over the new protocol.
539
"""
540
super(_SwitchBox, self)._sendTo(proto)
541
proto._lockForSwitch()
542
proto._switchTo(self.innerProto)
543
544
545
546
class BoxDispatcher:
547
"""
548
A L{BoxDispatcher} dispatches '_ask', '_answer', and '_error' L{AmpBox}es,
549
both incoming and outgoing, to their appropriate destinations.
550
551
Outgoing commands are converted into L{Deferred}s and outgoing boxes, and
552
associated tracking state to fire those L{Deferred} when '_answer' boxes
553
come back. Incoming '_answer' and '_error' boxes are converted into
554
callbacks and errbacks on those L{Deferred}s, respectively.
555
556
Incoming '_ask' boxes are converted into method calls on a supplied method
557
locator.
558
559
@ivar _outstandingRequests: a dictionary mapping request IDs to
560
L{Deferred}s which were returned for those requests.
561
562
@ivar locator: an object with a L{locateResponder} method that locates a
563
responder function that takes a Box and returns a result (either a Box or a
564
Deferred which fires one).
565
566
@ivar boxSender: an object which can send boxes, via the L{_sendBox}
567
method, such as an L{AMP} instance.
568
@type boxSender: L{IBoxSender}
569
"""
570
571
implements(IBoxReceiver)
572
573
_failAllReason = None
574
_outstandingRequests = None
575
_counter = 0L
576
boxSender = None
577
578
def __init__(self, locator):
579
self._outstandingRequests = {}
580
self.locator = locator
581
582
583
def startReceivingBoxes(self, boxSender):
584
"""
585
The given boxSender is going to start calling boxReceived on this
586
L{BoxDispatcher}.
587
588
@param boxSender: The L{IBoxSender} to send command responses to.
589
"""
590
self.boxSender = boxSender
591
592
593
def stopReceivingBoxes(self, reason):
594
"""
595
No further boxes will be received here. Terminate all currently
596
oustanding command deferreds with the given reason.
597
"""
598
self.failAllOutgoing(reason)
599
600
601
def failAllOutgoing(self, reason):
602
"""
603
Call the errback on all outstanding requests awaiting responses.
604
605
@param reason: the Failure instance to pass to those errbacks.
606
"""
607
self._failAllReason = reason
608
OR = self._outstandingRequests.items()
609
self._outstandingRequests = None # we can never send another request
610
for key, value in OR:
611
value.errback(reason)
612
613
614
def _nextTag(self):
615
"""
616
Generate protocol-local serial numbers for _ask keys.
617
618
@return: a string that has not yet been used on this connection.
619
"""
620
self._counter += 1
621
return '%x' % (self._counter,)
622
623
624
def _sendBoxCommand(self, command, box, requiresAnswer=True):
625
"""
626
Send a command across the wire with the given C{amp.Box}.
627
628
Mutate the given box to give it any additional keys (_command, _ask)
629
required for the command and request/response machinery, then send it.
630
631
If requiresAnswer is True, returns a C{Deferred} which fires when a
632
response is received. The C{Deferred} is fired with an C{amp.Box} on
633
success, or with an C{amp.RemoteAmpError} if an error is received.
634
635
If the Deferred fails and the error is not handled by the caller of
636
this method, the failure will be logged and the connection dropped.
637
638
@param command: a str, the name of the command to issue.
639
640
@param box: an AmpBox with the arguments for the command.
641
642
@param requiresAnswer: a boolean. Defaults to True. If True, return a
643
Deferred which will fire when the other side responds to this command.
644
If False, return None and do not ask the other side for acknowledgement.
645
646
@return: a Deferred which fires the AmpBox that holds the response to
647
this command, or None, as specified by requiresAnswer.
648
649
@raise ProtocolSwitched: if the protocol has been switched.
650
"""
651
if self._failAllReason is not None:
652
return fail(self._failAllReason)
653
box[COMMAND] = command
654
tag = self._nextTag()
655
if requiresAnswer:
656
box[ASK] = tag
657
box._sendTo(self.boxSender)
658
if requiresAnswer:
659
result = self._outstandingRequests[tag] = Deferred()
660
else:
661
result = None
662
return result
663
664
665
def callRemoteString(self, command, requiresAnswer=True, **kw):
666
"""
667
This is a low-level API, designed only for optimizing simple messages
668
for which the overhead of parsing is too great.
669
670
@param command: a str naming the command.
671
672
@param kw: arguments to the amp box.
673
674
@param requiresAnswer: a boolean. Defaults to True. If True, return a
675
Deferred which will fire when the other side responds to this command.
676
If False, return None and do not ask the other side for acknowledgement.
677
678
@return: a Deferred which fires the AmpBox that holds the response to
679
this command, or None, as specified by requiresAnswer.
680
"""
681
box = Box(kw)
682
return self._sendBoxCommand(command, box)
683
684
685
def callRemote(self, commandType, *a, **kw):
686
"""
687
This is the primary high-level API for sending messages via AMP. Invoke it
688
with a command and appropriate arguments to send a message to this
689
connection's peer.
690
691
@param commandType: a subclass of Command.
692
@type commandType: L{type}
693
694
@param a: Positional (special) parameters taken by the command.
695
Positional parameters will typically not be sent over the wire. The
696
only command included with AMP which uses positional parameters is
697
L{ProtocolSwitchCommand}, which takes the protocol that will be
698
switched to as its first argument.
699
700
@param kw: Keyword arguments taken by the command. These are the
701
arguments declared in the command's 'arguments' attribute. They will
702
be encoded and sent to the peer as arguments for the L{commandType}.
703
704
@return: If L{commandType} has a C{requiresAnswer} attribute set to
705
L{False}, then return L{None}. Otherwise, return a L{Deferred} which
706
fires with a dictionary of objects representing the result of this
707
call. Additionally, this L{Deferred} may fail with an exception
708
representing a connection failure, with L{UnknownRemoteError} if the
709
other end of the connection fails for an unknown reason, or with any
710
error specified as a key in L{commandType}'s C{errors} dictionary.
711
"""
712
713
# XXX this takes command subclasses and not command objects on purpose.
714
# There's really no reason to have all this back-and-forth between
715
# command objects and the protocol, and the extra object being created
716
# (the Command instance) is pointless. Command is kind of like
717
# Interface, and should be more like it.
718
719
# In other words, the fact that commandType is instantiated here is an
720
# implementation detail. Don't rely on it.
721
722
co = commandType(*a, **kw)
723
return co._doCommand(self)
724
725
726
def unhandledError(self, failure):
727
"""
728
This is a terminal callback called after application code has had a
729
chance to quash any errors.
730
"""
731
return self.boxSender.unhandledError(failure)
732
733
734
def _answerReceived(self, box):
735
"""
736
An AMP box was received that answered a command previously sent with
737
L{callRemote}.
738
739
@param box: an AmpBox with a value for its L{ANSWER} key.
740
"""
741
question = self._outstandingRequests.pop(box[ANSWER])
742
question.addErrback(self.unhandledError)
743
question.callback(box)
744
745
746
def _errorReceived(self, box):
747
"""
748
An AMP box was received that answered a command previously sent with
749
L{callRemote}, with an error.
750
751
@param box: an L{AmpBox} with a value for its L{ERROR}, L{ERROR_CODE},
752
and L{ERROR_DESCRIPTION} keys.
753
"""
754
question = self._outstandingRequests.pop(box[ERROR])
755
question.addErrback(self.unhandledError)
756
errorCode = box[ERROR_CODE]
757
description = box[ERROR_DESCRIPTION]
758
if errorCode in PROTOCOL_ERRORS:
759
exc = PROTOCOL_ERRORS[errorCode](errorCode, description)
760
else:
761
exc = RemoteAmpError(errorCode, description)
762
question.errback(Failure(exc))
763
764
765
def _commandReceived(self, box):
766
"""
767
@param box: an L{AmpBox} with a value for its L{COMMAND} and L{ASK}
768
keys.
769
"""
770
cmd = box[COMMAND]
771
def formatAnswer(answerBox):
772
answerBox[ANSWER] = box[ASK]
773
return answerBox
774
def formatError(error):
775
if error.check(RemoteAmpError):
776
code = error.value.errorCode
777
desc = error.value.description
778
if error.value.fatal:
779
errorBox = QuitBox()
780
else:
781
errorBox = AmpBox()
782
else:
783
errorBox = QuitBox()
784
log.err(error) # here is where server-side logging happens
785
# if the error isn't handled
786
code = UNKNOWN_ERROR_CODE
787
desc = "Unknown Error"
788
errorBox[ERROR] = box[ASK]
789
errorBox[ERROR_DESCRIPTION] = desc
790
errorBox[ERROR_CODE] = code
791
return errorBox
792
deferred = self.dispatchCommand(box)
793
if ASK in box:
794
deferred.addCallbacks(formatAnswer, formatError)
795
deferred.addCallback(self._safeEmit)
796
deferred.addErrback(self.unhandledError)
797
798
799
def ampBoxReceived(self, box):
800
"""
801
An AmpBox was received, representing a command, or an answer to a
802
previously issued command (either successful or erroneous). Respond to
803
it according to its contents.
804
805
@param box: an AmpBox
806
807
@raise NoEmptyBoxes: when a box is received that does not contain an
808
'_answer', '_command' / '_ask', or '_error' key; i.e. one which does not
809
fit into the command / response protocol defined by AMP.
810
"""
811
if ANSWER in box:
812
self._answerReceived(box)
813
elif ERROR in box:
814
self._errorReceived(box)
815
elif COMMAND in box:
816
self._commandReceived(box)
817
else:
818
raise NoEmptyBoxes(box)
819
820
821
def _safeEmit(self, aBox):
822
"""
823
Emit a box, ignoring L{ProtocolSwitched} and L{ConnectionLost} errors
824
which cannot be usefully handled.
825
"""
826
try:
827
aBox._sendTo(self.boxSender)
828
except (ProtocolSwitched, ConnectionLost):
829
pass
830
831
832
def dispatchCommand(self, box):
833
"""
834
A box with a _command key was received.
835
836
Dispatch it to a local handler call it.
837
838
@param proto: an AMP instance.
839
@param box: an AmpBox to be dispatched.
840
"""
841
cmd = box[COMMAND]
842
responder = self.locator.locateResponder(cmd)
843
if responder is None:
844
return fail(RemoteAmpError(
845
UNHANDLED_ERROR_CODE,
846
"Unhandled Command: %r" % (cmd,),
847
False,
848
local=Failure(UnhandledCommand())))
849
return maybeDeferred(responder, box)
850
851
852
853
class CommandLocator:
854
"""
855
A L{CommandLocator} is a collection of responders to AMP L{Command}s, with
856
the help of the L{Command.responder} decorator.
857
"""
858
859
class __metaclass__(type):
860
"""
861
This metaclass keeps track of all of the Command.responder-decorated
862
methods defined since the last CommandLocator subclass was defined. It
863
assumes (usually correctly, but unfortunately not necessarily so) that
864
those commands responders were all declared as methods of the class
865
being defined. Note that this list can be incorrect if users use the
866
Command.responder decorator outside the context of a CommandLocator
867
class declaration.
868
869
The Command.responder decorator explicitly cooperates with this
870
metaclass.
871
"""
872
873
_currentClassCommands = []
874
def __new__(cls, name, bases, attrs):
875
commands = cls._currentClassCommands[:]
876
cls._currentClassCommands[:] = []
877
cd = attrs['_commandDispatch'] = {}
878
for base in bases:
879
cls._grabFromBase(cd, base)
880
for commandClass, responderFunc in commands:
881
cd[commandClass.commandName] = (commandClass, responderFunc)
882
subcls = type.__new__(cls, name, bases, attrs)
883
if (bases and (
884
subcls.lookupFunction != CommandLocator.lookupFunction)):
885
def locateResponder(self, name):
886
warnings.warn(
887
"Override locateResponder, not lookupFunction.",
888
category=PendingDeprecationWarning,
889
stacklevel=2)
890
return self.lookupFunction(name)
891
subcls.locateResponder = locateResponder
892
return subcls
893
894
def _grabFromBase(cls, cd, base):
895
if hasattr(base, "_commandDispatch"):
896
cd.update(base._commandDispatch)
897
for subbase in base.__bases__:
898
cls._grabFromBase(cd, subbase)
899
_grabFromBase = classmethod(_grabFromBase)
900
901
implements(IResponderLocator)
902
903
904
def _wrapWithSerialization(self, aCallable, command):
905
"""
906
Wrap aCallable with its command's argument de-serialization
907
and result serialization logic.
908
909
@param aCallable: a callable with a 'command' attribute, designed to be
910
called with keyword arguments.
911
912
@param command: the command class whose serialization to use.
913
914
@return: a 1-arg callable which, when invoked with an AmpBox, will
915
deserialize the argument list and invoke appropriate user code for the
916
callable's command, returning a Deferred which fires with the result or
917
fails with an error.
918
"""
919
def doit(box):
920
kw = command.parseArguments(box, self)
921
def checkKnownErrors(error):
922
key = error.trap(*command.allErrors)
923
code = command.allErrors[key]
924
desc = str(error.value)
925
return Failure(RemoteAmpError(
926
code, desc, key in command.fatalErrors, local=error))
927
def makeResponseFor(objects):
928
try:
929
return command.makeResponse(objects, self)
930
except:
931
# let's helpfully log this.
932
originalFailure = Failure()
933
raise BadLocalReturn(
934
"%r returned %r and %r could not serialize it" % (
935
aCallable,
936
objects,
937
command),
938
originalFailure)
939
return maybeDeferred(aCallable, **kw).addCallback(
940
makeResponseFor).addErrback(
941
checkKnownErrors)
942
return doit
943
944
945
def lookupFunction(self, name):
946
"""
947
Deprecated synonym for L{locateResponder}
948
"""
949
if self.__class__.lookupFunction != CommandLocator.lookupFunction:
950
return CommandLocator.locateResponder(self, name)
951
else:
952
warnings.warn("Call locateResponder, not lookupFunction.",
953
category=PendingDeprecationWarning,
954
stacklevel=2)
955
return self.locateResponder(name)
956
957
958
def locateResponder(self, name):
959
"""
960
Locate a callable to invoke when executing the named command.
961
962
@param name: the normalized name (from the wire) of the command.
963
964
@return: a 1-argument function that takes a Box and returns a box or a
965
Deferred which fires a Box, for handling the command identified by the
966
given name, or None, if no appropriate responder can be found.
967
"""
968
# Try to find a high-level method to invoke, and if we can't find one,
969
# fall back to a low-level one.
970
cd = self._commandDispatch
971
if name in cd:
972
commandClass, responderFunc = cd[name]
973
responderMethod = types.MethodType(
974
responderFunc, self, self.__class__)
975
return self._wrapWithSerialization(responderMethod, commandClass)
976
977
978
979
class SimpleStringLocator(object):
980
"""
981
Implement the L{locateResponder} method to do simple, string-based
982
dispatch.
983
"""
984
985
implements(IResponderLocator)
986
987
baseDispatchPrefix = 'amp_'
988
989
def locateResponder(self, name):
990
"""
991
Locate a callable to invoke when executing the named command.
992
993
@return: a function with the name C{"amp_" + name} on L{self}, or None
994
if no such function exists. This function will then be called with the
995
L{AmpBox} itself as an argument.
996
997
@param name: the normalized name (from the wire) of the command.
998
"""
999
fName = self.baseDispatchPrefix + (name.upper())
1000
return getattr(self, fName, None)
1001
1002
1003
1004
PYTHON_KEYWORDS = [
1005
'and', 'del', 'for', 'is', 'raise', 'assert', 'elif', 'from', 'lambda',
1006
'return', 'break', 'else', 'global', 'not', 'try', 'class', 'except',
1007
'if', 'or', 'while', 'continue', 'exec', 'import', 'pass', 'yield',
1008
'def', 'finally', 'in', 'print']
1009
1010
1011
1012
def _wireNameToPythonIdentifier(key):
1013
"""
1014
(Private) Normalize an argument name from the wire for use with Python
1015
code. If the return value is going to be a python keyword it will be
1016
capitalized. If it contains any dashes they will be replaced with
1017
underscores.
1018
1019
The rationale behind this method is that AMP should be an inherently
1020
multi-language protocol, so message keys may contain all manner of bizarre
1021
bytes. This is not a complete solution; there are still forms of arguments
1022
that this implementation will be unable to parse. However, Python
1023
identifiers share a huge raft of properties with identifiers from many
1024
other languages, so this is a 'good enough' effort for now. We deal
1025
explicitly with dashes because that is the most likely departure: Lisps
1026
commonly use dashes to separate method names, so protocols initially
1027
implemented in a lisp amp dialect may use dashes in argument or command
1028
names.
1029
1030
@param key: a str, looking something like 'foo-bar-baz' or 'from'
1031
1032
@return: a str which is a valid python identifier, looking something like
1033
'foo_bar_baz' or 'From'.
1034
"""
1035
lkey = key.replace("-", "_")
1036
if lkey in PYTHON_KEYWORDS:
1037
return lkey.title()
1038
return lkey
1039
1040
1041
1042
class Argument:
1043
"""
1044
Base-class of all objects that take values from Amp packets and convert
1045
them into objects for Python functions.
1046
"""
1047
optional = False
1048
1049
1050
def __init__(self, optional=False):
1051
"""
1052
Create an Argument.
1053
1054
@param optional: a boolean indicating whether this argument can be
1055
omitted in the protocol.
1056
"""
1057
self.optional = optional
1058
1059
1060
def retrieve(self, d, name, proto):
1061
"""
1062
Retrieve the given key from the given dictionary, removing it if found.
1063
1064
@param d: a dictionary.
1065
1066
@param name: a key in L{d}.
1067
1068
@param proto: an instance of an AMP.
1069
1070
@raise KeyError: if I am not optional and no value was found.
1071
1072
@return: d[name].
1073
"""
1074
if self.optional:
1075
value = d.get(name)
1076
if value is not None:
1077
del d[name]
1078
else:
1079
value = d.pop(name)
1080
return value
1081
1082
1083
def fromBox(self, name, strings, objects, proto):
1084
"""
1085
Populate an 'out' dictionary with mapping names to Python values
1086
decoded from an 'in' AmpBox mapping strings to string values.
1087
1088
@param name: the argument name to retrieve
1089
@type name: str
1090
1091
@param strings: The AmpBox to read string(s) from, a mapping of
1092
argument names to string values.
1093
@type strings: AmpBox
1094
1095
@param objects: The dictionary to write object(s) to, a mapping of
1096
names to Python objects.
1097
@type objects: dict
1098
1099
@param proto: an AMP instance.
1100
"""
1101
st = self.retrieve(strings, name, proto)
1102
nk = _wireNameToPythonIdentifier(name)
1103
if self.optional and st is None:
1104
objects[nk] = None
1105
else:
1106
objects[nk] = self.fromStringProto(st, proto)
1107
1108
1109
def toBox(self, name, strings, objects, proto):
1110
"""
1111
Populate an 'out' AmpBox with strings encoded from an 'in' dictionary
1112
mapping names to Python values.
1113
1114
@param name: the argument name to retrieve
1115
@type name: str
1116
1117
@param strings: The AmpBox to write string(s) to, a mapping of
1118
argument names to string values.
1119
@type strings: AmpBox
1120
1121
@param objects: The dictionary to read object(s) from, a mapping of
1122
names to Python objects.
1123
1124
@type objects: dict
1125
1126
@param proto: the protocol we are converting for.
1127
@type proto: AMP
1128
"""
1129
obj = self.retrieve(objects, _wireNameToPythonIdentifier(name), proto)
1130
if self.optional and obj is None:
1131
# strings[name] = None
1132
pass
1133
else:
1134
strings[name] = self.toStringProto(obj, proto)
1135
1136
1137
def fromStringProto(self, inString, proto):
1138
"""
1139
Convert a string to a Python value.
1140
1141
@param inString: the string to convert.
1142
1143
@param proto: the protocol we are converting for.
1144
@type proto: AMP
1145
1146
@return: a Python object.
1147
"""
1148
return self.fromString(inString)
1149
1150
1151
def toStringProto(self, inObject, proto):
1152
"""
1153
Convert a Python object to a string.
1154
1155
@param inObject: the object to convert.
1156
1157
@param proto: the protocol we are converting for.
1158
@type proto: AMP
1159
"""
1160
return self.toString(inObject)
1161
1162
1163
def fromString(self, inString):
1164
"""
1165
Convert a string to a Python object. Subclasses must implement this.
1166
1167
@param inString: the string to convert.
1168
@type inString: str
1169
1170
@return: the decoded value from inString
1171
"""
1172
1173
1174
def toString(self, inObject):
1175
"""
1176
Convert a Python object into a string for passing over the network.
1177
1178
@param inObject: an object of the type that this Argument is intended
1179
to deal with.
1180
1181
@return: the wire encoding of inObject
1182
@rtype: str
1183
"""
1184
1185
1186
1187
class Integer(Argument):
1188
"""
1189
Convert to and from 'int'.
1190
"""
1191
fromString = int
1192
def toString(self, inObject):
1193
return str(int(inObject))
1194
1195
1196
1197
class String(Argument):
1198
"""
1199
Don't do any conversion at all; just pass through 'str'.
1200
"""
1201
def toString(self, inObject):
1202
return inObject
1203
1204
1205
def fromString(self, inString):
1206
return inString
1207
1208
1209
1210
class Float(Argument):
1211
"""
1212
Encode floating-point values on the wire as their repr.
1213
"""
1214
fromString = float
1215
toString = repr
1216
1217
1218
1219
class Boolean(Argument):
1220
"""
1221
Encode True or False as "True" or "False" on the wire.
1222
"""
1223
def fromString(self, inString):
1224
if inString == 'True':
1225
return True
1226
elif inString == 'False':
1227
return False
1228
else:
1229
raise TypeError("Bad boolean value: %r" % (inString,))
1230
1231
1232
def toString(self, inObject):
1233
if inObject:
1234
return 'True'
1235
else:
1236
return 'False'
1237
1238
1239
1240
class Unicode(String):
1241
"""
1242
Encode a unicode string on the wire as UTF-8.
1243
"""
1244
1245
def toString(self, inObject):
1246
# assert isinstance(inObject, unicode)
1247
return String.toString(self, inObject.encode('utf-8'))
1248
1249
1250
def fromString(self, inString):
1251
# assert isinstance(inString, str)
1252
return String.fromString(self, inString).decode('utf-8')
1253
1254
1255
1256
class Path(Unicode):
1257
"""
1258
Encode and decode L{filepath.FilePath} instances as paths on the wire.
1259
1260
This is really intended for use with subprocess communication tools:
1261
exchanging pathnames on different machines over a network is not generally
1262
meaningful, but neither is it disallowed; you can use this to communicate
1263
about NFS paths, for example.
1264
"""
1265
def fromString(self, inString):
1266
return filepath.FilePath(Unicode.fromString(self, inString))
1267
1268
1269
def toString(self, inObject):
1270
return Unicode.toString(self, inObject.path)
1271
1272
1273
1274
class AmpList(Argument):
1275
"""
1276
Convert a list of dictionaries into a list of AMP boxes on the wire.
1277
1278
For example, if you want to pass::
1279
1280
[{'a': 7, 'b': u'hello'}, {'a': 9, 'b': u'goodbye'}]
1281
1282
You might use an AmpList like this in your arguments or response list::
1283
1284
AmpList([('a', Integer()),
1285
('b', Unicode())])
1286
"""
1287
def __init__(self, subargs):
1288
"""
1289
Create an AmpList.
1290
1291
@param subargs: a list of 2-tuples of ('name', argument) describing the
1292
schema of the dictionaries in the sequence of amp boxes.
1293
"""
1294
self.subargs = subargs
1295
1296
1297
def fromStringProto(self, inString, proto):
1298
boxes = parseString(inString)
1299
values = [_stringsToObjects(box, self.subargs, proto)
1300
for box in boxes]
1301
return values
1302
1303
1304
def toStringProto(self, inObject, proto):
1305
return ''.join([_objectsToStrings(
1306
objects, self.subargs, Box(), proto
1307
).serialize() for objects in inObject])
1308
1309
class Command:
1310
"""
1311
Subclass me to specify an AMP Command.
1312
1313
@cvar arguments: A list of 2-tuples of (name, Argument-subclass-instance),
1314
specifying the names and values of the parameters which are required for
1315
this command.
1316
1317
@cvar response: A list like L{arguments}, but instead used for the return
1318
value.
1319
1320
@cvar errors: A mapping of subclasses of L{Exception} to wire-protocol tags
1321
for errors represented as L{str}s. Responders which raise keys from this
1322
dictionary will have the error translated to the corresponding tag on the
1323
wire. Invokers which receive Deferreds from invoking this command with
1324
L{AMP.callRemote} will potentially receive Failures with keys from this
1325
mapping as their value. This mapping is inherited; if you declare a
1326
command which handles C{FooError} as 'FOO_ERROR', then subclass it and
1327
specify C{BarError} as 'BAR_ERROR', responders to the subclass may raise
1328
either C{FooError} or C{BarError}, and invokers must be able to deal with
1329
either of those exceptions.
1330
1331
@cvar fatalErrors: like 'errors', but errors in this list will always
1332
terminate the connection, despite being of a recognizable error type.
1333
1334
@cvar commandType: The type of Box used to issue commands; useful only for
1335
protocol-modifying behavior like startTLS or protocol switching. Defaults
1336
to a plain vanilla L{Box}.
1337
1338
@cvar responseType: The type of Box used to respond to this command; only
1339
useful for protocol-modifying behavior like startTLS or protocol switching.
1340
Defaults to a plain vanilla L{Box}.
1341
1342
@ivar requiresAnswer: a boolean; defaults to True. Set it to False on your
1343
subclass if you want callRemote to return None. Note: this is a hint only
1344
to the client side of the protocol. The return-type of a command responder
1345
method must always be a dictionary adhering to the contract specified by
1346
L{response}, because clients are always free to request a response if they
1347
want one.
1348
"""
1349
1350
class __metaclass__(type):
1351
"""
1352
Metaclass hack to establish reverse-mappings for 'errors' and
1353
'fatalErrors' as class vars.
1354
"""
1355
def __new__(cls, name, bases, attrs):
1356
re = attrs['reverseErrors'] = {}
1357
er = attrs['allErrors'] = {}
1358
if 'commandName' not in attrs:
1359
attrs['commandName'] = name
1360
newtype = type.__new__(cls, name, bases, attrs)
1361
errors = {}
1362
fatalErrors = {}
1363
accumulateClassDict(newtype, 'errors', errors)
1364
accumulateClassDict(newtype, 'fatalErrors', fatalErrors)
1365
for v, k in errors.iteritems():
1366
re[k] = v
1367
er[v] = k
1368
for v, k in fatalErrors.iteritems():
1369
re[k] = v
1370
er[v] = k
1371
return newtype
1372
1373
arguments = []
1374
response = []
1375
extra = []
1376
errors = {}
1377
fatalErrors = {}
1378
1379
commandType = Box
1380
responseType = Box
1381
1382
requiresAnswer = True
1383
1384
1385
def __init__(self, **kw):
1386
"""
1387
Create an instance of this command with specified values for its
1388
parameters.
1389
1390
@param kw: a dict containing an appropriate value for each name
1391
specified in the L{arguments} attribute of my class.
1392
1393
@raise InvalidSignature: if you forgot any required arguments.
1394
"""
1395
self.structured = kw
1396
givenArgs = kw.keys()
1397
forgotten = []
1398
for name, arg in self.arguments:
1399
pythonName = _wireNameToPythonIdentifier(name)
1400
if pythonName not in givenArgs and not arg.optional:
1401
forgotten.append(pythonName)
1402
if forgotten:
1403
raise InvalidSignature("forgot %s for %s" % (
1404
', '.join(forgotten), self.commandName))
1405
forgotten = []
1406
1407
1408
def makeResponse(cls, objects, proto):
1409
"""
1410
Serialize a mapping of arguments using this L{Command}'s
1411
response schema.
1412
1413
@param objects: a dict with keys matching the names specified in
1414
self.response, having values of the types that the Argument objects in
1415
self.response can format.
1416
1417
@param proto: an L{AMP}.
1418
1419
@return: an L{AmpBox}.
1420
"""
1421
return _objectsToStrings(objects, cls.response, cls.responseType(),
1422
proto)
1423
makeResponse = classmethod(makeResponse)
1424
1425
1426
def makeArguments(cls, objects, proto):
1427
"""
1428
Serialize a mapping of arguments using this L{Command}'s
1429
argument schema.
1430
1431
@param objects: a dict with keys similar to the names specified in
1432
self.arguments, having values of the types that the Argument objects in
1433
self.arguments can parse.
1434
1435
@param proto: an L{AMP}.
1436
1437
@return: An instance of this L{Command}'s C{commandType}.
1438
"""
1439
return _objectsToStrings(objects, cls.arguments, cls.commandType(),
1440
proto)
1441
makeArguments = classmethod(makeArguments)
1442
1443
1444
def parseResponse(cls, box, protocol):
1445
"""
1446
Parse a mapping of serialized arguments using this
1447
L{Command}'s response schema.
1448
1449
@param box: A mapping of response-argument names to the
1450
serialized forms of those arguments.
1451
@param protocol: The L{AMP} protocol.
1452
1453
@return: A mapping of response-argument names to the parsed
1454
forms.
1455
"""
1456
return _stringsToObjects(box, cls.response, protocol)
1457
parseResponse = classmethod(parseResponse)
1458
1459
1460
def parseArguments(cls, box, protocol):
1461
"""
1462
Parse a mapping of serialized arguments using this
1463
L{Command}'s argument schema.
1464
1465
@param box: A mapping of argument names to the seralized forms
1466
of those arguments.
1467
@param protocol: The L{AMP} protocol.
1468
1469
@return: A mapping of argument names to the parsed forms.
1470
"""
1471
return _stringsToObjects(box, cls.arguments, protocol)
1472
parseArguments = classmethod(parseArguments)
1473
1474
1475
def responder(cls, methodfunc):
1476
"""
1477
Declare a method to be a responder for a particular command.
1478
1479
This is a decorator.
1480
1481
Use like so::
1482
1483
class MyCommand(Command):
1484
arguments = [('a', ...), ('b', ...)]
1485
1486
class MyProto(AMP):
1487
def myFunMethod(self, a, b):
1488
...
1489
MyCommand.responder(myFunMethod)
1490
1491
Notes: Although decorator syntax is not used within Twisted, this
1492
function returns its argument and is therefore safe to use with
1493
decorator syntax.
1494
1495
This is not thread safe. Don't declare AMP subclasses in other
1496
threads. Don't declare responders outside the scope of AMP subclasses;
1497
the behavior is undefined.
1498
1499
@param methodfunc: A function which will later become a method, which
1500
has a keyword signature compatible with this command's L{argument} list
1501
and returns a dictionary with a set of keys compatible with this
1502
command's L{response} list.
1503
1504
@return: the methodfunc parameter.
1505
"""
1506
CommandLocator._currentClassCommands.append((cls, methodfunc))
1507
return methodfunc
1508
responder = classmethod(responder)
1509
1510
1511
# Our only instance method
1512
def _doCommand(self, proto):
1513
"""
1514
Encode and send this Command to the given protocol.
1515
1516
@param proto: an AMP, representing the connection to send to.
1517
1518
@return: a Deferred which will fire or error appropriately when the
1519
other side responds to the command (or error if the connection is lost
1520
before it is responded to).
1521
"""
1522
1523
def _massageError(error):
1524
error.trap(RemoteAmpError)
1525
rje = error.value
1526
errorType = self.reverseErrors.get(rje.errorCode,
1527
UnknownRemoteError)
1528
return Failure(errorType(rje.description))
1529
1530
d = proto._sendBoxCommand(self.commandName,
1531
self.makeArguments(self.structured, proto),
1532
self.requiresAnswer)
1533
1534
if self.requiresAnswer:
1535
d.addCallback(self.parseResponse, proto)
1536
d.addErrback(_massageError)
1537
1538
return d
1539
1540
1541
1542
class _NoCertificate:
1543
"""
1544
This is for peers which don't want to use a local certificate. Used by
1545
AMP because AMP's internal language is all about certificates and this
1546
duck-types in the appropriate place; this API isn't really stable though,
1547
so it's not exposed anywhere public.
1548
1549
For clients, it will use ephemeral DH keys, or whatever the default is for
1550
certificate-less clients in OpenSSL. For servers, it will generate a
1551
temporary self-signed certificate with garbage values in the DN and use
1552
that.
1553
"""
1554
1555
def __init__(self, client):
1556
"""
1557
Create a _NoCertificate which either is or isn't for the client side of
1558
the connection.
1559
1560
@param client: True if we are a client and should truly have no
1561
certificate and be anonymous, False if we are a server and actually
1562
have to generate a temporary certificate.
1563
1564
@type client: bool
1565
"""
1566
self.client = client
1567
1568
1569
def options(self, *authorities):
1570
"""
1571
Behaves like L{twisted.internet.ssl.PrivateCertificate.options}().
1572
"""
1573
if not self.client:
1574
# do some crud with sslverify to generate a temporary self-signed
1575
# certificate. This is SLOOOWWWWW so it is only in the absolute
1576
# worst, most naive case.
1577
1578
# We have to do this because OpenSSL will not let both the server
1579
# and client be anonymous.
1580
sharedDN = DN(CN='TEMPORARY CERTIFICATE')
1581
key = KeyPair.generate()
1582
cr = key.certificateRequest(sharedDN)
1583
sscrd = key.signCertificateRequest(sharedDN, cr, lambda dn: True, 1)
1584
cert = key.newCertificate(sscrd)
1585
return cert.options(*authorities)
1586
options = dict()
1587
if authorities:
1588
options.update(dict(verify=True,
1589
requireCertificate=True,
1590
caCerts=[auth.original for auth in authorities]))
1591
occo = CertificateOptions(**options)
1592
return occo
1593
1594
1595
1596
class _TLSBox(AmpBox):
1597
"""
1598
I am an AmpBox that, upon being sent, initiates a TLS connection.
1599
"""
1600
__slots__ = []
1601
1602
def _keyprop(k, default):
1603
return property(lambda self: self.get(k, default))
1604
1605
1606
# These properties are described in startTLS
1607
certificate = _keyprop('tls_localCertificate', _NoCertificate(False))
1608
verify = _keyprop('tls_verifyAuthorities', None)
1609
1610
def _sendTo(self, proto):
1611
"""
1612
Send my encoded value to the protocol, then initiate TLS.
1613
"""
1614
ab = AmpBox(self)
1615
for k in ['tls_localCertificate',
1616
'tls_verifyAuthorities']:
1617
ab.pop(k, None)
1618
ab._sendTo(proto)
1619
proto._startTLS(self.certificate, self.verify)
1620
1621
1622
1623
class _LocalArgument(String):
1624
"""
1625
Local arguments are never actually relayed across the wire. This is just a
1626
shim so that StartTLS can pretend to have some arguments: if arguments
1627
acquire documentation properties, replace this with something nicer later.
1628
"""
1629
1630
def fromBox(self, name, strings, objects, proto):
1631
pass
1632
1633
1634
1635
class StartTLS(Command):
1636
"""
1637
Use, or subclass, me to implement a command that starts TLS.
1638
1639
Callers of StartTLS may pass several special arguments, which affect the
1640
TLS negotiation:
1641
1642
- tls_localCertificate: This is a
1643
twisted.internet.ssl.PrivateCertificate which will be used to secure
1644
the side of the connection it is returned on.
1645
1646
- tls_verifyAuthorities: This is a list of
1647
twisted.internet.ssl.Certificate objects that will be used as the
1648
certificate authorities to verify our peer's certificate.
1649
1650
Each of those special parameters may also be present as a key in the
1651
response dictionary.
1652
"""
1653
1654
arguments = [("tls_localCertificate", _LocalArgument(optional=True)),
1655
("tls_verifyAuthorities", _LocalArgument(optional=True))]
1656
1657
response = [("tls_localCertificate", _LocalArgument(optional=True)),
1658
("tls_verifyAuthorities", _LocalArgument(optional=True))]
1659
1660
responseType = _TLSBox
1661
1662
def __init__(self, **kw):
1663
"""
1664
Create a StartTLS command. (This is private. Use AMP.callRemote.)
1665
1666
@param tls_localCertificate: the PrivateCertificate object to use to
1667
secure the connection. If it's None, or unspecified, an ephemeral DH
1668
key is used instead.
1669
1670
@param tls_verifyAuthorities: a list of Certificate objects which
1671
represent root certificates to verify our peer with.
1672
"""
1673
self.certificate = kw.pop('tls_localCertificate', _NoCertificate(True))
1674
self.authorities = kw.pop('tls_verifyAuthorities', None)
1675
Command.__init__(self, **kw)
1676
1677
1678
def _doCommand(self, proto):
1679
"""
1680
When a StartTLS command is sent, prepare to start TLS, but don't actually
1681
do it; wait for the acknowledgement, then initiate the TLS handshake.
1682
"""
1683
d = Command._doCommand(self, proto)
1684
proto._prepareTLS(self.certificate, self.authorities)
1685
# XXX before we get back to user code we are going to start TLS...
1686
def actuallystart(response):
1687
proto._startTLS(self.certificate, self.authorities)
1688
return response
1689
d.addCallback(actuallystart)
1690
return d
1691
1692
1693
1694
class ProtocolSwitchCommand(Command):
1695
"""
1696
Use this command to switch from something Amp-derived to a different
1697
protocol mid-connection. This can be useful to use amp as the
1698
connection-startup negotiation phase. Since TLS is a different layer
1699
entirely, you can use Amp to negotiate the security parameters of your
1700
connection, then switch to a different protocol, and the connection will
1701
remain secured.
1702
"""
1703
1704
def __init__(self, _protoToSwitchToFactory, **kw):
1705
"""
1706
Create a ProtocolSwitchCommand.
1707
1708
@param _protoToSwitchToFactory: a ProtocolFactory which will generate
1709
the Protocol to switch to.
1710
1711
@param kw: Keyword arguments, encoded and handled normally as
1712
L{Command} would.
1713
"""
1714
1715
self.protoToSwitchToFactory = _protoToSwitchToFactory
1716
super(ProtocolSwitchCommand, self).__init__(**kw)
1717
1718
1719
def makeResponse(cls, innerProto, proto):
1720
return _SwitchBox(innerProto)
1721
makeResponse = classmethod(makeResponse)
1722
1723
1724
def _doCommand(self, proto):
1725
"""
1726
When we emit a ProtocolSwitchCommand, lock the protocol, but don't actually
1727
switch to the new protocol unless an acknowledgement is received. If
1728
an error is received, switch back.
1729
"""
1730
d = super(ProtocolSwitchCommand, self)._doCommand(proto)
1731
proto._lockForSwitch()
1732
def switchNow(ign):
1733
innerProto = self.protoToSwitchToFactory.buildProtocol(
1734
proto.transport.getPeer())
1735
proto._switchTo(innerProto, self.protoToSwitchToFactory)
1736
return ign
1737
def handle(ign):
1738
proto._unlockFromSwitch()
1739
self.protoToSwitchToFactory.clientConnectionFailed(
1740
None, Failure(CONNECTION_LOST))
1741
return ign
1742
return d.addCallbacks(switchNow, handle)
1743
1744
1745
1746
class BinaryBoxProtocol(StatefulStringProtocol, Int16StringReceiver):
1747
"""
1748
A protocol for receving L{Box}es - key/value pairs - via length-prefixed
1749
strings. A box is composed of:
1750
1751
- any number of key-value pairs, described by:
1752
- a 2-byte network-endian packed key length (of which the first
1753
byte must be null, and the second must be non-null: i.e. the
1754
value of the length must be 1-255)
1755
- a key, comprised of that many bytes
1756
- a 2-byte network-endian unsigned value length (up to the maximum
1757
of 65535)
1758
- a value, comprised of that many bytes
1759
- 2 null bytes
1760
1761
In other words, an even number of strings prefixed with packed unsigned
1762
16-bit integers, and then a 0-length string to indicate the end of the box.
1763
1764
This protocol also implements 2 extra private bits of functionality related
1765
to the byte boundaries between messages; it can start TLS between two given
1766
boxes or switch to an entirely different protocol. However, due to some
1767
tricky elements of the implementation, the public interface to this
1768
functionality is L{ProtocolSwitchCommand} and L{StartTLS}.
1769
1770
@ivar boxReceiver: an L{IBoxReceiver} provider, whose L{ampBoxReceived}
1771
method will be invoked for each L{Box} that is received.
1772
"""
1773
1774
implements(IBoxSender)
1775
1776
_justStartedTLS = False
1777
_startingTLSBuffer = None
1778
_locked = False
1779
_currentKey = None
1780
_currentBox = None
1781
1782
hostCertificate = None
1783
noPeerCertificate = False # for tests
1784
innerProtocol = None
1785
innerProtocolClientFactory = None
1786
1787
_sslVerifyProblems = ()
1788
# ^ Later this will become a mutable list - we can't get the handle during
1789
# connection shutdown thanks to the fact that Twisted destroys the socket
1790
# on our transport before notifying us of a lost connection (which I guess
1791
# is reasonable - the socket is dead by then) See a few lines below in
1792
# startTLS for details. --glyph
1793
1794
1795
def __init__(self, boxReceiver):
1796
self.boxReceiver = boxReceiver
1797
1798
1799
def _switchTo(self, newProto, clientFactory=None):
1800
"""
1801
Switch this BinaryBoxProtocol's transport to a new protocol. You need
1802
to do this 'simultaneously' on both ends of a connection; the easiest
1803
way to do this is to use a subclass of ProtocolSwitchCommand.
1804
1805
@param newProto: the new protocol instance to switch to.
1806
1807
@param clientFactory: the ClientFactory to send the
1808
L{clientConnectionLost} notification to.
1809
"""
1810
# All the data that Int16Receiver has not yet dealt with belongs to our
1811
# new protocol: luckily it's keeping that in a handy (although
1812
# ostensibly internal) variable for us:
1813
newProtoData = self.recvd
1814
# We're quite possibly in the middle of a 'dataReceived' loop in
1815
# Int16StringReceiver: let's make sure that the next iteration, the
1816
# loop will break and not attempt to look at something that isn't a
1817
# length prefix.
1818
self.recvd = ''
1819
# Finally, do the actual work of setting up the protocol and delivering
1820
# its first chunk of data, if one is available.
1821
self.innerProtocol = newProto
1822
self.innerProtocolClientFactory = clientFactory
1823
newProto.makeConnection(self.transport)
1824
newProto.dataReceived(newProtoData)
1825
1826
1827
def sendBox(self, box):
1828
"""
1829
Send a amp.Box to my peer.
1830
1831
Note: transport.write is never called outside of this method.
1832
1833
@param box: an AmpBox.
1834
1835
@raise ProtocolSwitched: if the protocol has previously been switched.
1836
1837
@raise ConnectionLost: if the connection has previously been lost.
1838
"""
1839
if self._locked:
1840
raise ProtocolSwitched(
1841
"This connection has switched: no AMP traffic allowed.")
1842
if self.transport is None:
1843
raise ConnectionLost()
1844
if self._startingTLSBuffer is not None:
1845
self._startingTLSBuffer.append(box)
1846
else:
1847
self.transport.write(box.serialize())
1848
1849
1850
def makeConnection(self, transport):
1851
"""
1852
Notify L{boxReceiver} that it is about to receive boxes from this
1853
protocol by invoking L{startReceivingBoxes}.
1854
"""
1855
self.boxReceiver.startReceivingBoxes(self)
1856
Int16StringReceiver.makeConnection(self, transport)
1857
1858
1859
def dataReceived(self, data):
1860
"""
1861
Either parse incoming data as L{AmpBox}es or relay it to our nested
1862
protocol.
1863
"""
1864
if self._justStartedTLS:
1865
self._justStartedTLS = False
1866
# If we already have an inner protocol, then we don't deliver data to
1867
# the protocol parser any more; we just hand it off.
1868
if self.innerProtocol is not None:
1869
self.innerProtocol.dataReceived(data)
1870
return
1871
return Int16StringReceiver.dataReceived(self, data)
1872
1873
1874
def connectionLost(self, reason):
1875
"""
1876
The connection was lost; notify any nested protocol.
1877
"""
1878
if self.innerProtocol is not None:
1879
self.innerProtocol.connectionLost(reason)
1880
if self.innerProtocolClientFactory is not None:
1881
self.innerProtocolClientFactory.clientConnectionLost(None, reason)
1882
# XXX this may be a slight oversimplification, but I believe that if
1883
# there are pending SSL errors, they _are_ the reason that the
1884
# connection was lost. a totally correct implementation of this would
1885
# set up a simple state machine to track whether any bytes were
1886
# received after startTLS was called. --glyph
1887
problems = self._sslVerifyProblems
1888
if problems:
1889
failReason = Failure(problems[0])
1890
elif self._justStartedTLS:
1891
# We just started TLS and haven't received any data. This means
1892
# the other connection didn't like our cert (although they may not
1893
# have told us why - later Twisted should make 'reason' into a TLS
1894
# error.)
1895
failReason = PeerVerifyError(
1896
"Peer rejected our certificate for an unknown reason.")
1897
else:
1898
failReason = reason
1899
self.boxReceiver.stopReceivingBoxes(failReason)
1900
1901
1902
1903
def proto_init(self, string):
1904
"""
1905
String received in the 'init' state.
1906
"""
1907
self._currentBox = AmpBox()
1908
return self.proto_key(string)
1909
1910
1911
def proto_key(self, string):
1912
"""
1913
String received in the 'key' state. If the key is empty, a complete
1914
box has been received.
1915
"""
1916
if string:
1917
self._currentKey = string
1918
return 'value'
1919
else:
1920
self.boxReceiver.ampBoxReceived(self._currentBox)
1921
self._currentBox = None
1922
return 'init'
1923
1924
1925
def proto_value(self, string):
1926
"""
1927
String received in the 'value' state.
1928
"""
1929
self._currentBox[self._currentKey] = string
1930
self._currentKey = None
1931
return 'key'
1932
1933
1934
def _lockForSwitch(self):
1935
"""
1936
Lock this binary protocol so that no further boxes may be sent. This
1937
is used when sending a request to switch underlying protocols. You
1938
probably want to subclass ProtocolSwitchCommand rather than calling
1939
this directly.
1940
"""
1941
self._locked = True
1942
1943
1944
def _unlockFromSwitch(self):
1945
"""
1946
Unlock this locked binary protocol so that further boxes may be sent
1947
again. This is used after an attempt to switch protocols has failed
1948
for some reason.
1949
"""
1950
if self.innerProtocol is not None:
1951
raise ProtocolSwitched("Protocol already switched. Cannot unlock.")
1952
self._locked = False
1953
1954
1955
def _prepareTLS(self, certificate, verifyAuthorities):
1956
"""
1957
Used by StartTLSCommand to put us into the state where we don't
1958
actually send things that get sent, instead we buffer them. see
1959
L{_sendBox}.
1960
"""
1961
self._startingTLSBuffer = []
1962
if self.hostCertificate is not None:
1963
raise OnlyOneTLS(
1964
"Previously authenticated connection between %s and %s "
1965
"is trying to re-establish as %s" % (
1966
self.hostCertificate,
1967
self.peerCertificate,
1968
(certificate, verifyAuthorities)))
1969
1970
1971
def _startTLS(self, certificate, verifyAuthorities):
1972
"""
1973
Used by TLSBox to initiate the SSL handshake.
1974
1975
@param certificate: a L{twisted.internet.ssl.PrivateCertificate} for
1976
use locally.
1977
1978
@param verifyAuthorities: L{twisted.internet.ssl.Certificate} instances
1979
representing certificate authorities which will verify our peer.
1980
"""
1981
self.hostCertificate = certificate
1982
self._justStartedTLS = True
1983
if verifyAuthorities is None:
1984
verifyAuthorities = ()
1985
self.transport.startTLS(certificate.options(*verifyAuthorities))
1986
# Remember that mutable list that we were just talking about? Here
1987
# it is. sslverify.py takes care of populating this list as
1988
# necessary. --glyph
1989
self._sslVerifyProblems = problemsFromTransport(self.transport)
1990
stlsb = self._startingTLSBuffer
1991
if stlsb is not None:
1992
self._startingTLSBuffer = None
1993
for box in stlsb:
1994
self.sendBox(box)
1995
1996
1997
def _getPeerCertificate(self):
1998
if self.noPeerCertificate:
1999
return None
2000
return Certificate.peerFromTransport(self.transport)
2001
peerCertificate = property(_getPeerCertificate)
2002
2003
2004
def unhandledError(self, failure):
2005
"""
2006
The buck stops here. This error was completely unhandled, time to
2007
terminate the connection.
2008
"""
2009
log.msg("Amp server or network failure "
2010
"unhandled by client application:")
2011
log.err(failure)
2012
log.msg(
2013
"Dropping connection! "
2014
"To avoid, add errbacks to ALL remote commands!")
2015
if self.transport is not None:
2016
self.transport.loseConnection()
2017
2018
2019
def _defaultStartTLSResponder(self):
2020
"""
2021
The default TLS responder doesn't specify any certificate or anything.
2022
2023
From a security perspective, it's little better than a plain-text
2024
connection - but it is still a *bit* better, so it's included for
2025
convenience.
2026
2027
You probably want to override this by providing your own StartTLS.responder.
2028
"""
2029
return {}
2030
StartTLS.responder(_defaultStartTLSResponder)
2031
2032
2033
2034
class AMP(BinaryBoxProtocol, BoxDispatcher,
2035
CommandLocator, SimpleStringLocator):
2036
"""
2037
This protocol is an AMP connection. See the module docstring for protocol
2038
details.
2039
"""
2040
2041
_ampInitialized = False
2042
2043
def __init__(self, boxReceiver=None, locator=None):
2044
# For backwards compatibility. When AMP did not separate parsing logic
2045
# (L{BinaryBoxProtocol}), request-response logic (L{BoxDispatcher}) and
2046
# command routing (L{CommandLocator}), it did not have a constructor.
2047
# Now it does, so old subclasses might have defined their own that did
2048
# not upcall. If this flag isn't set, we'll call the constructor in
2049
# makeConnection before anything actually happens.
2050
self._ampInitialized = True
2051
if boxReceiver is None:
2052
boxReceiver = self
2053
if locator is None:
2054
locator = self
2055
boxSender = self
2056
BoxDispatcher.__init__(self, locator)
2057
BinaryBoxProtocol.__init__(self, boxReceiver)
2058
2059
2060
def locateResponder(self, name):
2061
"""
2062
Unify the implementations of L{CommandLocator} and
2063
L{SimpleStringLocator} to perform both kinds of dispatch, preferring
2064
L{CommandLocator}.
2065
"""
2066
firstResponder = CommandLocator.locateResponder(self, name)
2067
if firstResponder is not None:
2068
return firstResponder
2069
secondResponder = SimpleStringLocator.locateResponder(self, name)
2070
return secondResponder
2071
2072
2073
def __repr__(self):
2074
"""
2075
A verbose string representation which gives us information about this
2076
AMP connection.
2077
"""
2078
return '<%s %s at 0x%x>' % (
2079
self.__class__.__name__,
2080
self.innerProtocol, id(self))
2081
2082
2083
def makeConnection(self, transport):
2084
"""
2085
Emit a helpful log message when the connection is made.
2086
"""
2087
if not self._ampInitialized:
2088
# See comment in the constructor re: backward compatibility. I
2089
# should probably emit a deprecation warning here.
2090
AMP.__init__(self)
2091
# Save these so we can emit a similar log message in L{connectionLost}.
2092
self._transportPeer = transport.getPeer()
2093
self._transportHost = transport.getHost()
2094
log.msg("%s connection established (HOST:%s PEER:%s)" % (
2095
self.__class__.__name__,
2096
self._transportHost,
2097
self._transportPeer))
2098
BinaryBoxProtocol.makeConnection(self, transport)
2099
2100
2101
def connectionLost(self, reason):
2102
"""
2103
Emit a helpful log message when the connection is lost.
2104
"""
2105
log.msg("%s connection lost (HOST:%s PEER:%s)" %
2106
(self.__class__.__name__,
2107
self._transportHost,
2108
self._transportPeer))
2109
BinaryBoxProtocol.connectionLost(self, reason)
2110
self.transport = None
2111
2112
2113
2114
class _ParserHelper:
2115
"""
2116
A box receiver which records all boxes received.
2117
"""
2118
def __init__(self):
2119
self.boxes = []
2120
2121
2122
def getPeer(self):
2123
return 'string'
2124
2125
2126
def getHost(self):
2127
return 'string'
2128
2129
disconnecting = False
2130
2131
2132
def startReceivingBoxes(self, sender):
2133
"""
2134
No initialization is required.
2135
"""
2136
2137
2138
def ampBoxReceived(self, box):
2139
self.boxes.append(box)
2140
2141
2142
# Synchronous helpers
2143
def parse(cls, fileObj):
2144
"""
2145
Parse some amp data stored in a file.
2146
2147
@param fileObj: a file-like object.
2148
2149
@return: a list of AmpBoxes encoded in the given file.
2150
"""
2151
parserHelper = cls()
2152
bbp = BinaryBoxProtocol(boxReceiver=parserHelper)
2153
bbp.makeConnection(parserHelper)
2154
bbp.dataReceived(fileObj.read())
2155
return parserHelper.boxes
2156
parse = classmethod(parse)
2157
2158
2159
def parseString(cls, data):
2160
"""
2161
Parse some amp data stored in a string.
2162
2163
@param data: a str holding some amp-encoded data.
2164
2165
@return: a list of AmpBoxes encoded in the given string.
2166
"""
2167
return cls.parse(StringIO(data))
2168
parseString = classmethod(parseString)
2169
2170
2171
2172
parse = _ParserHelper.parse
2173
parseString = _ParserHelper.parseString
2174
2175
def _stringsToObjects(strings, arglist, proto):
2176
"""
2177
Convert an AmpBox to a dictionary of python objects, converting through a
2178
given arglist.
2179
2180
@param strings: an AmpBox (or dict of strings)
2181
2182
@param arglist: a list of 2-tuples of strings and Argument objects, as
2183
described in L{Command.arguments}.
2184
2185
@param proto: an L{AMP} instance.
2186
2187
@return: the converted dictionary mapping names to argument objects.
2188
"""
2189
objects = {}
2190
myStrings = strings.copy()
2191
for argname, argparser in arglist:
2192
argparser.fromBox(argname, myStrings, objects, proto)
2193
return objects
2194
2195
2196
2197
def _objectsToStrings(objects, arglist, strings, proto):
2198
"""
2199
Convert a dictionary of python objects to an AmpBox, converting through a
2200
given arglist.
2201
2202
@param objects: a dict mapping names to python objects
2203
2204
@param arglist: a list of 2-tuples of strings and Argument objects, as
2205
described in L{Command.arguments}.
2206
2207
@param strings: [OUT PARAMETER] An object providing the L{dict}
2208
interface which will be populated with serialized data.
2209
2210
@param proto: an L{AMP} instance.
2211
2212
@return: The converted dictionary mapping names to encoded argument
2213
strings (identical to C{strings}).
2214
"""
2215
myObjects = {}
2216
for (k, v) in objects.items():
2217
myObjects[k] = v
2218
2219
for argname, argparser in arglist:
2220
argparser.toBox(argname, strings, myObjects, proto)
2221
return strings
2222
2223
Loggerhead is a web-based interface for Breezy
Version: 2.0.1