1
An Introduction to boto's SQS interface
2
---------------------------------------
4
This tutorial focuses on the boto interface to the Simple Queue Service
5
from Amazon Web Services. This tutorial assumes that you have already
6
downloaded and installed boto.
10
The first step in accessing SQS is to create a connection to the service.
11
There are two ways to do this in boto. The first is:
13
>>> from boto.sqs.connection import SQSConnection
14
>>> conn = SQSConnection('<aws access key>', '<aws secret key>')
16
At this point the variable conn will point to an SQSConnection object. In
17
this example, the AWS access key and AWS secret key are passed in to the
18
method explicitely. Alternatively, you can set the environment variables:
20
AWS_ACCESS_KEY_ID - Your AWS Access Key ID
21
AWS_SECRET_ACCESS_KEY - Your AWS Secret Access Key
23
and then call the constructor without any arguments, like this:
25
>>> conn = SQSConnection()
27
There is also a shortcut function in the boto package, called connect_sqs
28
that may provide a slightly easier means of creating a connection:
31
>>> conn = boto.connect_sqs()
33
In either case, conn will point to an SQSConnection object which we will
34
use throughout the remainder of this tutorial.
39
Once you have a connection established with SQS, you will probably want to
40
create a queue. That can be accomplished like this:
42
>>> q = conn.create_queue('myqueue')
44
The create_queue method will create the requested queue if it does not
45
exist or will return the existing queue if it does exist. There is an
46
optional parameter to create_queue called visibility_timeout. This basically
47
controls how long a message will remain invisible to other queue readers
48
once it has been read (see SQS documentation for more detailed explanation).
49
If this is not explicitly specified the queue will be created with whatever
50
default value SQS provides (currently 30 seconds). If you would like to
51
specify another value, you could do so like this:
53
>>> q = conn.create_queue('myqueue', 120)
55
This would establish a default visibility timeout for this queue of 120
56
seconds. As you will see later on, this default value for the queue can
57
also be overridden each time a message is read from the queue. If you want
58
to check what the default visibility timeout is for a queue:
67
Once you have a queue, presumably you will want to write some messages
68
to it. SQS doesn't care what kind of information you store in your messages
69
or what format you use to store it. As long as the amount of data per
70
message is less than or equal to 256Kb, it's happy.
72
However, you may have a lot of specific requirements around the format of
73
that data. For example, you may want to store one big string or you might
74
want to store something that looks more like RFC822 messages or you might want
75
to store a binary payload such as pickled Python objects.
77
The way boto deals with this is to define a simple Message object that
78
treats the message data as one big string which you can set and get. If that
79
Message object meets your needs, you're good to go. However, if you need to
80
incorporate different behavior in your message or handle different types of
81
data you can create your own Message class. You just need to register that
82
class with the queue so that it knows that when you read a message from the
83
queue that it should create one of your message objects rather than the
84
default boto Message object. To register your message class, you would:
86
>>> q.set_message_class(MyMessage)
88
where MyMessage is the class definition for your message class. Your
89
message class should subclass the boto Message because there is a small
90
bit of Python magic happening in the __setattr__ method of the boto Message
93
For this tutorial, let's just assume that we are using the boto Message
94
class. So, first we need to create a Message object:
96
>>> from boto.sqs.message import Message
98
>>> m.set_body('This is my first message.')
101
The value returned by the write method of a queue object is a ResultSet. In
102
fact, most method calls return a ResultSet object. The purpose of the
103
ResultSet is to communicate both status and results from an operation. In
104
this case, there are no results returned because we are storing a message
105
in the queue rather than asking for data from the queue. However, the
106
ResultSet does contain some potentially useful status information. For
110
u'0ZPEY5AD356SP2WMFHYA|3H4AA8J7EJKM0DQZR7E1|FDDWV0TXFGBY4SK1E171'
112
u'3370b6fd-2be3-47e4-80b3-a1f08798e560'
116
The MessageId attribute is the unique identifier assigned by SQS to this
117
particular message. This value is also available in the Message object as
118
the id attribute, once the message has actually been written to queue:
121
u'0ZPEY5AD356SP2WMFHYA|3H4AA8J7EJKM0DQZR7E1|FDDWV0TXFGBY4SK1E171'
123
The RequestId is a unique identifier assigned by Amazon Web Services
124
to this particular request. If you ever experience problems with SQS
125
or have specific questions about a request, this information is critical
126
because it allows AWS to track down the specific request. Finally, the
127
StatusCode attribute is just the status code returned in the response. All
128
of these attributes map directly to elements in the XML response returned
134
So, now we have a message in our queue. How would we go about reading it?
137
>>> rs = q.get_messages()
142
u'This is my first message'
144
The get_messages method also returns a ResultSet object as described
145
above. In addition to the special attributes that we already talked
146
about the ResultSet object also contains any results returned by the
147
request. To get at the results you can treat the ResultSet as a
148
sequence object (e.g. a list). We can check the length (how many results)
149
and access particular items within the list using the slice notation
150
familiar to Python programmers.
152
At this point, we have read the message from the queue and SQS will make
153
sure that this message remains invisible to other readers of the queue
154
until the visibility timeout period for the queue expires. If I delete
155
the message before the timeout period expires then no one will ever see
156
the message again. However, if I don't delete it (maybe because I crashed
157
or failed in some way, for example) it will magically reappear in my queue
158
for someone else to read. If you aren't happy with the default visibility
159
timeout defined for the queue, you can override it when you read a message:
161
>>> q.get_messages(visibility_timeout=60)
163
This means that regardless of what the default visibility timeout is for
164
the queue, this message will remain invisible to other readers for 60
167
The get_messages method can also return more than a single message. By
168
passing a num_messages parameter (defaults to 1) you can control the maximum
169
number of messages that will be returned by the method. To show this
170
feature off, first let's load up a few more messages.
172
>>> for i in range(1, 11):
173
... m = Message('This is message %d' % i)
176
>>> rs = q.get_messages(10)
180
Don't be alarmed if the length of the result set returned by the get_messages
181
call is less than 10. Sometimes it takes some time for new messages to become
182
visible in the queue. Give it a minute or two and they will all show up.
184
If you want a slightly simpler way to read messages from a queue, you
185
can use the read method. It will either return the message read or
186
it will return None if no messages were available. You can also pass
187
a visibility_timeout parameter to read, if you desire:
191
u'This is my first message'
193
Deleting Messages and Queues
194
----------------------------
196
Note that the first message we put in the queue is still there, even though
197
we have read it a number of times. That's because we never deleted it. To
198
remove a message from a queue:
200
>>> q.delete_message(m)
203
If I want to delete the entire queue, I would use:
205
>>> conn.delete_queue(q)
207
However, this won't succeed unless the queue is empty.
209
Listing All Available Queues
210
----------------------------
211
In addition to accessing specific queues via the create_queue method
212
you can also get a list of all available queues that you have created.
214
>>> rs = conn.get_all_queues()
216
This returns a ResultSet object, as described above. The ResultSet
217
can be used as a sequence or list type object to retrieve Queue objects.
224
<listing of available queues>
230
That covers the basic operations of creating queues, writing messages,
231
reading messages, deleting messages, and deleting queues. There are a
232
few utility methods in boto that might be useful as well. For example,
233
to count the number of messages in a queue:
238
This can be handy but is command as well as the other two utility methods
239
I'll describe in a minute are inefficient and should be used with caution
240
on queues with lots of messages (e.g. many hundreds or more). Similarly,
241
you can clear (delete) all messages in a queue with:
245
Be REAL careful with that one! Finally, if you want to dump all of the
246
messages in a queue to a local file:
248
>>> q.dump('messages.txt', sep='\n------------------\n')
250
This will read all of the messages in the queue and write the bodies of
251
each of the messages to the file messages.txt. The option sep argument
252
is a separator that will be printed between each message body in the file.
added
removed
doc/sqs_tut.txt
