8
This is a plug-in for RabbitMQ that shovels messages from a queue on
9
one broker to an exchange on another broker. The two brokers may be
10
the same. The plug-in allows several shovels to be specified at the
11
same time. Each shovel may have a number of source and destination
12
brokers specified, and one of each is chosen whenever the shovel
13
attempts to make a connection: this permits simple round-rabbit load
16
Resources can be declared upon connection to both the source and
17
destination brokers, and parameters can be specified for both the
18
reception and publishing of messages.
24
You can build and install it like any other plugin (see
25
http://www.rabbitmq.com/plugin-development.html).
31
The RabbitMQ configuration file specifies the shovel
32
configurations. This exists by default, in
33
/etc/rabbitmq/rabbitmq.config under Linux systems,
34
%RABBITMQ_BASE%\rabbitmq.config under Windows or somewhere else under
35
OS X. This file configures both RabbitMQ-server and all the plugins
36
installed in it. It is an Erlang-syntax file of the form:
38
[{section1, [section1-config]},
39
{section2, [section2-config]},
41
{sectionN, [sectionN-config]}
44
thus a list of tuples, where the left element of each tuple names the
45
applications being configured. Don't forget the last element of the
46
list doesn't have a trailing comma, and don't forget the full-stop is
47
needed after closing the list. Hence if you configure RabbitMQ-server
48
and the RabbitMQ-shovel, then the configuration file may have a
51
[{rabbit, [configuration-for-RabbitMQ-server]},
52
{rabbit-shovel, [configuration-for-RabbitMQ-shovel]}
55
A full example of the shovel configuration is:
61
["amqp://fred:secret@host1.domain/my_vhost",
62
"amqp://john:secret@host2.domain/my_vhost"
67
[{exchange, <<"my_exchange">>},
70
{destinations, [{broker, "amqp://"},
73
[{exchange, <<"my_exchange">>},
79
{ack_mode, on_confirm},
80
{publish_properties, [{delivery_mode, 2}]},
81
{publish_fields, [{exchange, <<"my_exchange">>},
82
{routing_key, <<"from_shovel">>}]},
89
Firstly, all shovels are named. Here we have one shovel, called
90
'my_first_shovel'. We can have multiple shovels if you wish. Secondly,
91
every shovel must have the subfields sources, destinations and queue
92
specified, whereas the other fields (prefetch_count, ack_mode,
93
publish_properties, publish_fields, reconnect_delay) are optional and
94
have defaults as follows:
97
{ack_mode, on_confirm}
98
{publish_properties, []}
103
Sources and Destinations
104
------------------------
106
Sources and destinations specify respectively where messages are
107
fetched from and delivered too. One of 'broker' and 'brokers' must be
108
specified, and 'broker' is simply shorthand for when only one broker
109
needs specifying. Using 'brokers' allows a list of brokers to be
110
specified: whenever the connection to a broker is lost, another one is
111
chosen at random from the list and a connection attempt is made to
112
that. The syntax for broker URIs is:
114
amqp://username:password@host:port/vhost?key1=value1&key2=value2...
116
If username or password are omitted, the default values of guest and
117
guest are used. If the vhost is omitted, the default value of / is
118
used. If the host is omitted, then the plugin uses the "direct"
119
connection internally rather than a network connection: this means it
120
connects to the RabbitMQ-server node on which it is running without
121
going through the network stack. This is much more efficient. If port
122
is omitted then the default value is used (5672 or 5671 if SSL is
125
SSL is implemented, for which additional parameters are needed:
127
amqps://username:password@host:port/vhost?cacertfile=/path/to/cacert.pem&certfile=/path/to/certfile.pem&keyfile=/path/to/keyfile.pem&verify=verifyOption&fail_if_no_peer_cert=failOption
129
All five parameters (3 paths: cacertfile, certfile and keyfile; 2
130
options: verify, fail_if_no_peer_cert) must be specified. See the SSL
131
guide at http://www.rabbitmq.com/ssl.html#configure-erlang for details
132
of SSL in RabbitMQ in general and specifically for the Erlang client
133
(on which the shovel is built).
135
Note that SSL cannot be used with the direct connection (i.e. a host
136
must be specified when using SSL), and that it is preferable to use
137
the non-SSL direct connection when connecting to the same node that's
140
The query part of the URI permits the configuration of additional
141
connection parameters, permitting heartbeat, channel_max, and
142
frame_max to be specified. These can be given in any order, and
143
omitted fields assume default values. For example:
145
amqp://myhost?heartbeat=5&frame_max=8192
147
Specifies a non-encrypted network connection to the host 'myhost',
148
using default username, password, port, vhost and channel_max, but
149
specifying the heartbeat interval of 5 seconds, and the maximum frame
153
Resource Declarations
154
---------------------
156
Both sources and destinations can have an optional 'declarations'
157
clause. The value of this is a list, consisting of AMQP Methods. If
158
default values are sufficient, then the method name alone can be
159
specified - e.g. 'queue.declare'. If parameters need to be set then
160
the method should be given as a tuple, with the right hand side a
161
proplist specifying which fields need altering from their default
163
{'exchange.declare',[{exchange, <<"my_exchange">>},
164
{type, <<"direct">>},
167
One very useful feature here is the Most-Recently-Declared-Queue
168
feature, in which RabbitMQ remembers the name of the most recently
169
declared queue. This means that you can declare a private queue, and
170
then bind it to exchanges without ever needing to know its name.
176
This parameter specifies the name of the queue on the source brokers
177
to consume from. This queue must exist. Use the resource declarations
178
to create the queue (or ensure it exists) first. Note again that the
179
Most-Recently-Declared-Queue feature can be used here, thus an
180
anonymous queue can be used: use <<>> to indicate the
181
Most-Recently-Declared-Queue.
184
prefetch_count :: non-negative-integer
185
--------------------------------------
187
The shovel consumes from a queue. This parameter imposes a limit on
188
the number of messages which are sent to the shovel in advance of the
189
message the shovel is currently processing.
192
ack_mode :: 'no_ack' | 'on_publish' | 'on_confirm'
193
--------------------------------------------------
195
This setting controls if or when acknowledgements to the source broker
196
are issued by the shovel. The default is 'on_confirm'.
198
'no_ack' - The shovel consumes from its queue with no_ack = 'true':
199
i.e. the shovel does not issue explicit acks for messages it receives,
200
and the source broker considers messages acknowledged as soon as it
201
has sent them to the shovel.
203
'on_publish' - the shovel consumes from its queue with no_ack =
204
'false', and a message is acknowledged to the source broker as soon as
205
it has been published to the destination broker.
207
'on_confirm' - the shovel consumes from its queue with no_ack =
208
'false' and it requests publisher confirmations from the destination
209
broker. It only issues acknowledgements to the source broker when it
210
has received confirmation from the destination broker that each
211
message has been successfully received. This setting gives a guarantee
212
that messages will not be discarded from the source broker until after
213
the destination broker has confirmed receipt of the message.
215
'on_confirm' is strongly recommended. However the destination broker
216
must support publish confirms, meaning it can only be used with
217
RabbitMQ 2.3.1 and up.
222
This is a list of tuples which override fields in the basic class
223
properties when publishing to the destination. This can be used to
224
override any of the following fields: content_type, content_encoding,
225
headers, delivery_mode, priority, correlation_id, reply_to,
226
expiration, message_id, timestamp, type, user_id, app_id, cluster_id.
228
By default, all the properties of the basic class that are received
229
with the delivery are passed through to the destination, but this
230
field can be used to override them.
236
This is a list of tuples which override fields in the publish method
237
when publishing to the destination. This can be used to direct
238
messages to a particular exchange on the destination, for example, or
239
change the routing key. By default, the exchange and the routing key
240
of the message as it is received by the shovel is passed through, but
241
this can be overridden as necessary.
244
reconnect_delay :: non-negative-number
245
--------------------------------------
247
When an error occurs, the shovel will disconnect from both the source
248
and destination broker immediately. This will force uncommitted
249
transactions at the destination to be rolled back, and delivered but
250
unacknowledged messages from the source to be requeued. The shovel
251
will then try connecting again. If this is unsuccessful, then it's not
252
a good idea for the shovel to very quickly and repeatedly try to
253
reconnect. The value specified here is determines the delay, in
254
seconds, between each connection attempt.
256
Note that if set to 0, the shovel will never try to reconnect: it'll
257
stop after the first error.
259
Also, the value can be a floating point number, thus permitting the
260
specification of delays at a granularity smaller than whole seconds.
263
Obtaining shovel statuses
264
-------------------------
266
From the broker Erlang prompt, call
267
rabbit_shovel_status:status(). This will return a list, with one row
268
for each configured shovel. Each row has three fields: the shovel
269
name, the shovel status, and the timestamp (a local calendar time of
270
{{YYYY,MM,DD},{HH,MM,SS}}). There are 3 possible statuses:
272
'starting': The shovel is starting up, connecting and creating
275
{'running'|'blocked', {'source', Source},
276
{'destination', Destination}}:
278
When 'running', the shovel is up and running, shovelling
279
messages. When 'blocked', the destination has raised channel.flow,
280
preventing the shovel from sending messages to the
281
destination. The shovel will raise channel.flow to the source,
282
asking the source to stop sending further messages to the
283
shovel. Any messages that are received by the shovel before the
284
source observes the channel.flow are correctly buffered and
285
maintained in order, and are published to the destination as soon
286
as the destination drops the channel.flow block.
288
Source and Destination give the connection parameters used to
289
connect to the endpoints.
291
{'terminated', Reason}: Something's gone wrong. The Reason should give
292
a further indication of where the fault lies.
1
Generic build instructions are at:
2
http://www.rabbitmq.com/plugin-development.html
4
See the http://www.rabbitmq.com/shovel.html page for full instructions.
added
removed
plugins-src/rabbitmq-shovel/README
