1
<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
2
"http://www.w3.org/TR/html4/loose.dtd">
8
<title>Postfix Bottleneck Analysis</title>
10
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
16
<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix Bottleneck Analysis</h1>
20
<h2>Purpose of this document </h2>
22
<p> This document describes the qshape(1) program which helps the
23
administrator understand the Postfix queue message distribution
24
sorted by time and by sender or recipient domain. qshape(1) is
25
bundled with the Postfix 2.1 source under the "auxiliary" directory.
28
<p> In order to understand the output of qshape(1), it useful to
29
understand the various Postfix queues. To this end the role of each
30
Postfix queue directory is described briefly in the "Background
31
info: Postfix queue directories" section near the end of this
34
<p> This document covers the following topics: </p>
38
<li><a href="#qshape">Introducing the qshape tool</a>
40
<li><a href="#trouble_shooting">Trouble shooting with qshape</a>
42
<li><a href="#healthy">Example 1: Healthy queue</a>
44
<li><a href="#dictionary_bounce">Example 2: Deferred queue full of
45
dictionary attack bounces</a></li>
47
<li><a href="#active_congestion">Example 3: Congestion in the active
50
<li><a href="#backlog">Example 4: High volume destination backlog</a>
52
<li><a href="#queues">Background info: Postfix queue directories</a>
56
<li> <a href="#maildrop_queue"> The "maildrop" queue </a>
58
<li> <a href="#hold_queue"> The "hold" queue </a>
60
<li> <a href="#incoming_queue"> The "incoming" queue </a>
62
<li> <a href="#active_queue"> The "active" queue </a>
64
<li> <a href="#deferred_queue"> The "deferred" queue </a>
68
<li><a href="#credits">Credits</a>
72
<h2><a name="qshape">Introducing the qshape tool</a></h2>
75
<p> When mail is draining slowly or the queue is unexpectedly large,
76
run qshape(1) as the super-user (root) to help zero in on the problem.
77
The qshape(1) program displays a tabular view of the Postfix queue
82
<li> <p> On the horizontal axis, it displays the queue age with
83
fine granularity for recent messages and (geometrically) less fine
84
granularity for older messages. </p>
86
<li> <p> The vertical axis displays the destination (or with the
87
"-s" switch the sender) domain. Domains with the most messages are
92
<p> For example, in the output below we see the top 10 lines of
93
the (mostly forged) sender domain distribution for captured spam
94
in the "hold" queue: </p>
98
$ qshape -s hold | head
99
T 5 10 20 40 80 160 320 640 1280 1280+
100
TOTAL 486 0 0 1 0 0 2 4 20 40 419
101
yahoo.com 14 0 0 1 0 0 0 0 1 0 12
102
extremepricecuts.net 13 0 0 0 0 0 0 0 2 0 11
103
ms35.hinet.net 12 0 0 0 0 0 0 0 0 1 11
104
winnersdaily.net 12 0 0 0 0 0 0 0 2 0 10
105
hotmail.com 11 0 0 0 0 0 0 0 0 1 10
106
worldnet.fr 6 0 0 0 0 0 0 0 0 0 6
107
ms41.hinet.net 6 0 0 0 0 0 0 0 0 0 6
108
osn.de 5 0 0 0 0 0 1 0 0 0 4
114
<li> <p> The "T" column shows the total (in this case sender) count
115
for each domain. The columns with numbers above them, show counts
116
for messages aged fewer than that many minutes, but not younger
117
than the age limit for the previous column. The row labeled "TOTAL"
118
shows the total count for all domains. </p>
120
<li> <p> In this example, there are 14 messages allegedly from
121
yahoo.com, 1 between 10 and 20 minutes old, 1 between 320 and 640
122
minutes old and 12 older than 1280 minutes (1440 minutes in a day).
127
<p> By default, qshape shows statistics for the union of both the
128
incoming and active queues which are the most relevant queues to
129
look at when analyzing performance. </p>
131
<p> One can request an alternate list of queues: </p>
135
$ qshape deferred | less
136
$ qshape incoming active deferred | less
140
<p> this will show the age distribution of the deferred queue or
141
the union of the incoming active and deferred queues. </p>
143
<p> Command line options control the number of display "buckets",
144
the age limit for the smallest bucket, display of parent domain
145
counts and so on. The "-h" option outputs a summary of the available
148
<h2><a name="trouble_shooting">Trouble shooting with qshape</a>
151
<p> Large numbers in the qshape output represent a large number of
152
messages that are destined to (or alleged to come from) a particular
153
domain. It should be possible to tell at a glance which domains
154
dominate the queue sender or recipient counts, approximately when
155
a burst of mail started, and when it stopped. </p>
157
<p> The problem destinations or sender domains appear near the top
158
left corner of the output table. Remember that the active queue
159
can accommodate up to 20000 ($qmgr_message_active_limit) messages.
160
To check wether this limit has been reached, use: </p>
164
$ qshape -s active | head <i>(show sender statistics)</i>
168
<p> If the total sender count is below 20000 the active queue is
169
not yet saturated, any high volume sender domains show near the
172
<p> The active queue is also limited to at most 20000 recipient
173
addresses ($qmgr_message_recipient_limit). To check for exhaustion
174
of this limit use: </p>
178
$ qshape active | head <i>(show recipient statistics)</i>
182
<p> Having found the high volume domains, it is often useful to
183
search the logs for recent messages pertaining to the domains in
188
# Find deliveries to example.com
190
$ tail -10000 /var/log/maillog |
191
egrep -i ': to=<.*@example\.com>,' |
194
# Find messages from example.com
196
$ tail -10000 /var/log/maillog |
197
egrep -i ': from=<.*@example\.com>,' |
202
<p> You may want to drill in on some specific queue ids: </p>
206
# Find all messages for a specific queue id.
208
$ tail -10000 /var/log/maillog | egrep ': 2B2173FF68: '
212
<p> Also look for queue manager warning messages in the log. These
213
warnings can suggest strategies to reduce congestion. </p>
217
$ egrep 'qmgr.*(panic|fatal|error|warning):' /var/log/maillog
221
<p> When all else fails try the Postfix mailing list for help, but
222
please don't forget to include the top 10 or 20 lines of qshape(1)
225
<h2><a name="healthy">Example 1: Healthy queue</a></h2>
227
<p> When looking at just the incoming and active queues, under
228
normal conditions (no congestion) the incoming and active queues
229
are nearly empty. Mail leaves the system almost as quickly as it
230
comes in or is deferred without congestion in the active queue.
235
$ qshape <i>(show incoming and active queue status)</i>
237
T 5 10 20 40 80 160 320 640 1280 1280+
238
TOTAL 5 0 0 0 1 0 0 0 1 1 2
239
meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2
243
<p> If one looks at the two queues separately, the incoming queue
244
is empty or perhaps briefly has one or two messages, while the
245
active queue holds more messages and for a somewhat longer time:
252
T 5 10 20 40 80 160 320 640 1280 1280+
253
TOTAL 0 0 0 0 0 0 0 0 0 0 0
257
T 5 10 20 40 80 160 320 640 1280 1280+
258
TOTAL 5 0 0 0 1 0 0 0 1 1 2
259
meri.uwasa.fi 5 0 0 0 1 0 0 0 1 1 2
263
<h2><a name="dictionary_bounce">Example 2: Deferred queue full of
264
dictionary attack bounces</a></h2>
266
<p> This is from a server where recipient validation is not yet
267
available for some of the hosted domains. Dictionary attacks on
268
the unvalidated domains result in bounce backscatter. The bounces
269
dominate the queue, but with proper tuning they do not saturate the
270
incoming or active queues. The high volume of deferred mail is not
271
a direct cause for alarm. </p>
275
$ qshape deferred | head
277
T 5 10 20 40 80 160 320 640 1280 1280+
278
TOTAL 2234 4 2 5 9 31 57 108 201 464 1353
279
heyhihellothere.com 207 0 0 1 1 6 6 8 25 68 92
280
pleazerzoneprod.com 105 0 0 0 0 0 0 0 5 44 56
281
groups.msn.com 63 2 1 2 4 4 14 14 14 8 0
282
orion.toppoint.de 49 0 0 0 1 0 2 4 3 16 23
283
kali.com.cn 46 0 0 0 0 1 0 2 6 12 25
284
meri.uwasa.fi 44 0 0 0 0 1 0 2 8 11 22
285
gjr.paknet.com.pk 43 1 0 0 1 1 3 3 6 12 16
286
aristotle.algonet.se 41 0 0 0 0 0 1 2 11 12 15
290
<p> The domains shown are mostly bulk-mailers and all the volume
291
is the tail end of the time distribution, showing that short term
292
arrival rates are moderate. Larger numbers and lower message ages
293
are more indicative of current trouble. Old mail still going nowhere
294
is largely harmless so long as the active and incoming queues are
295
short. We can also see that the groups.msn.com undeliverables are
296
low rate steady stream rather than a concentrated dictionary attack
297
that is now over. </p>
301
$ qshape -s deferred | head
303
T 5 10 20 40 80 160 320 640 1280 1280+
304
TOTAL 2193 4 4 5 8 33 56 104 205 465 1309
305
MAILER-DAEMON 1709 4 4 5 8 33 55 101 198 452 849
306
example.com 263 0 0 0 0 0 0 0 0 2 261
307
example.org 209 0 0 0 0 0 1 3 6 11 188
308
example.net 6 0 0 0 0 0 0 0 0 0 6
309
example.edu 3 0 0 0 0 0 0 0 0 0 3
310
example.gov 2 0 0 0 0 0 0 0 1 0 1
311
example.mil 1 0 0 0 0 0 0 0 0 0 1
315
<p> Looking at the sender distribution, we see that as expected
316
most of the messages are bounces. </p>
318
<h2><a name="active_congestion">Example 3: Congestion in the active
321
<p> This example is taken from a Feb 2004 discussion on the Postfix
322
Users list. Congestion was reported with the active and incoming
323
queues large and not shrinking despite very large delivery agent
324
process limits. The thread is archived at:
325
http://groups.google.com/groups?th=636626c645f5bbde </p>
327
<p> Using an older version of qshape(1) it was quickly determined
328
that all the messages were for just a few destinations: </p>
332
$ qshape <i>(show incoming and active queue status)</i>
334
T A 5 10 20 40 80 160 320 320+
335
TOTAL 11775 9996 0 0 1 1 42 94 221 1420
336
user.sourceforge.net 7678 7678 0 0 0 0 0 0 0 0
337
lists.sourceforge.net 2313 2313 0 0 0 0 0 0 0 0
338
gzd.gotdns.com 102 0 0 0 0 0 0 0 2 100
342
<p> The "A" column showed the count of messages in the active queue,
343
and the numbered columns showed totals for the deferred queue. At
344
10000 messages (Postfix 1.x active queue size limit) the active
345
queue is full. The incoming was growing rapidly. </p>
347
<p> With the trouble destinations clearly identified, the administrator
348
quickly found and fixed the problem. It is substantially harder to
349
glean the same information from the logs. While a careful reading
350
of mailq(1) output should yield similar results, it is much harder
351
to gauge the magnitude of the problem by looking at the queue
352
one message at a time. </p>
354
<h2><a name="backlog">Example 4: High volume destination backlog</a></h2>
356
<p> When a site you send a lot of email to is down or slow, mail
357
messages will rapidly build up in the deferred queue, or worse, in
358
the active queue. The qshape output will show large numbers for
359
the destination domain in all age buckets that overlap the starting
360
time of the problem: </p>
364
$ qshape deferred | head
366
T 5 10 20 40 80 160 320 640 1280 1280+
367
TOTAL 5000 200 200 400 800 1600 1000 200 200 200 200
368
highvolume.com 4000 160 160 320 640 1280 1440 0 0 0 0
373
<p> Here the "highvolume.com" destination is continuing to accumulate
374
deferred mail. The incoming and active queues are fine, but the
375
deferred queue started growing some time between 1 and 2 hours ago
376
and continues to grow. </p>
378
<p> If the high volume destination is not down, but is instead
379
slow, one might see similar congestion in the active queue. Active
380
queue congestion is a greater cause for alarm; one might need to
381
take measures to ensure that the mail is deferred instead or even
382
add an access(5) rule asking the sender to try again later. </p>
384
<p> If a high volume destination exhibits frequent bursts of
385
consecutive connections refused by all MX hosts or "421 Server busy
386
errors", it is possible for the queue manager to mark the destination
387
as "dead" despite the transient nature of the errors. The destination
388
will be retried again after the expiration of a $minimal_backoff_time
389
timer. If the error bursts are frequent enough it may be that only
390
a small quantity of email is delivered before the destination is
391
again marked "dead". </p>
393
<p> The MTA that has been observed most frequently to exhibit such
394
bursts of errors is Microsoft Exchange, which refuses connections
395
under load. Some proxy virus scanners in front of the Exchange
396
server propagate the refused connection to the client as a "421"
399
<p> Note that it is now possible to configure Postfix to exhibit
400
similarly erratic behavior by misconfiguring the anvil(8) server
401
(not included in Postfix 2.1.). Do not use anvil(8) for steady-state
402
rate limiting, its purpose is DoS prevention and the rate limits
403
set should be very generous! </p>
405
<p> In the long run it is hoped that the Postfix dead host detection
406
and concurrency control mechanism will be tuned to be more "noise"
407
tolerant. If one finds oneself needing to deliver a high volume
408
of mail to a destination that exhibits frequent brief bursts of
409
errors, there is a subtle workaround. </p>
413
<li> <p> In master.cf set up a dedicated clone of the "smtp"
414
transport for the destination in question. </p>
416
<li> <p> In master.cf configure a reasonable process limit for the
417
transport (a number in the 10-20 range is typical). </p>
419
<li> <p> IMPORTANT!!! In main.cf configure a very large initial
420
and destination concurrency limit for this transport (say 200). </p>
423
/etc/postfix/main.cf:
424
initial_destination_concurrency = 200
425
<i>transportname</i>_destination_concurrency_limit = 200
428
<p> Where <i>transportname</i> is the name of the master.cf entry
433
<p> The effect of this surprising configuration is that up to 200
434
consecutive errors are tolerated without marking the destination
435
dead, while the total concurrency remains reasonable (10-20
436
processes). This trick is only for a very specialized situation:
437
high volume delivery into a channel with multi-error bursts
438
that is capable of high throughput, but is repeatedly throttled by
439
the bursts of errors.
441
<p> When a destination is unable to handle the load even after the
442
Postfix process limit is reduced to 1, a desperate measure is to
443
insert brief delays between delivery attempts. </p>
447
<li> <p> In the transport map entry for the problem destination,
448
specify a dead host as the primary nexthop. </p>
450
<li> <p> In the master.cf entry for the transport specify the
451
problem destination as the fallback_relay and specify a small
452
smtp_connect_timeout value. </p>
455
/etc/postfix/transport:
456
problem.example.com slow:[dead.host]
458
/etc/postfix/master.cf:
459
# service type private unpriv chroot wakeup maxproc command
460
slow unix - - n - 1 smtp
461
-o fallback_relay=problem.example.com
462
-o smtp_connect_timeout=1
467
<p> This solution forces the Postfix smtp(8) client to wait for
468
$smtp_connect_timeout seconds between deliveries. The solution
469
depends on Postfix connection management details, and needs to be
470
updated when SMTP connection caching is introduced. </p>
472
<p> Hopefully a more elegant solution to these problems will be
473
found in the future. </p>
475
<h2><a name="queues">Background info: Postfix queue directories</a></h2>
477
<p> The following sections describe Postfix queues: their purpose,
478
what normal behavior looks like, and how to diagnose abnormal
481
<h3> <a name="maildrop_queue"> The "maildrop" queue </a> </h3>
483
<p> Messages that have been submitted via the Postfix sendmail(1)
484
command, but not yet brought into the main Postfix queue by the
485
pickup(8) service, await processing in the "maildrop" queue. Messages
486
can be added to the "maildrop" queue even when the Postfix system
487
is not running. They will begin to be processed once Postfix is
490
<p> The "maildrop" queue is drained by the single threaded pickup(8)
491
service scanning the queue directory periodically or when notified
492
of new message arrival by the postdrop(1) program. The postdrop(1)
493
program is a setgid helper that allows the unprivileged Postfix
494
sendmail(1) program to inject mail into the "maildrop" queue and
495
to notify the pickup(8) service of its arrival. </p>
497
<p> All mail that enters the main Postfix queue does so via the
498
cleanup(8) service. The cleanup service is responsible for envelope
499
and header rewriting, header and body regular expression checks,
500
automatic bcc recipient processing and guaranteed insertion of the
501
message into the Postfix "incoming" queue. </p>
503
<p> In the absence of excessive CPU consumption in cleanup(8) header
504
or body regular expression checks or other software consuming all
505
available CPU resources, Postfix performance is disk I/O bound.
506
The rate at which the pickup(8) service can inject messages into
507
the queue is largely determined by disk access times, since the
508
cleanup(8) service must commit the message to stable storage before
509
returning success. The same is true of the postdrop(1) program
510
writing the message to the "maildrop" directory. </p>
512
<p> As the pickup service is single threaded, it can only deliver
513
one message at a time at a rate that does not exceed the reciprocal
514
disk I/O latency (+ CPU if not negligible) of the cleanup service.
517
<p> Congestion in this queue is indicative of an excessive local
518
message submission rate or perhaps excessive CPU consumption in
519
the cleanup(8) service due to excessive body_checks. </p>
521
<p> Note, that once the active queue is full, the cleanup service
522
will attempt to slow down message injection by pausing $in_flow_delay
523
for each message. In this case "maildrop" queue congestion may be
524
a consequence of congestion downstream, rather than a problem in
527
<p> Note also, that one should not attempt to deliver large volumes
528
of mail via the pickup(8) service. High volume sites must avoid
529
using content filters that reinject scanned mail via Postfix
530
sendmail(1) and postdrop(1). </p>
532
<p> A high arrival rate of locally submitted mail may be an indication
533
of an uncaught forwarding loop, or a run-away notification program.
534
Try to keep the volume of local mail injection to a moderate level.
537
<p> The "postsuper -r" command can place selected messages into
538
the "maildrop" queue for reprocessing. This is most useful for
539
resetting any stale content_filter settings. Requeuing a large number
540
of messages using "postsuper -r" can clearly cause a spike in the
541
size of the "maildrop" queue. </p>
543
<h3> <a name="hold_queue"> The "hold" queue </a> </h3>
545
<p> The administrator can define "smtpd" access(5) policies, or
546
cleanup(8) header/body checks that cause messages to be automatically
547
diverted from normal processing and placed indefinitely in the
548
"hold" queue. Messages placed in the "hold" queue stay there until
549
the administrator intervenes. No periodic delivery attempts are
550
made for messages in the "hold" queue. The postsuper(1) command
551
can be used to manually release messages into the "deferred" queue.
554
<p> Messages can potentially stay in the "hold" queue for a time
555
exceeding the normal maximal queue lifetime (after which undelivered
556
messages are bounced back to the sender). If such "old" messages
557
need to be released from the "hold" queue, they should typically
558
be moved into the "maildrop" queue, so that the message gets a new
559
timestamp and is given more than one opportunity to be delivered.
560
Messages that are "young" can be moved directly into the "deferred"
563
<p> The "hold" queue plays little role in Postfix performance, and
564
monitoring of the "hold" queue is typically more closely motivated
565
by tracking spam and malware, than by performance issues. </p>
567
<h3> <a name="incoming_queue"> The "incoming" queue </a> </h3>
569
<p> All new mail entering the Postfix queue is written by the
570
cleanup(8) service into the "incoming" queue. New queue files are
571
created owned by the "postfix" user with an access bitmask (or
572
mode) of 0600. Once a queue file is ready for further processing
573
the cleanup(8) service changes the queue file mode to 0700 and
574
notifies the queue manager of new mail arrival. The queue manager
575
ignores incomplete queue files whose mode is 0600, as these are
576
still being written by cleanup. </p>
578
<p> The queue manager scans the incoming queue bringing any new
579
mail into the "active" queue if the active queue resource limits
580
have not been exceeded. By default, the active queue accommodates
581
at most 20000 messages. Once the active queue message limit is
582
reached, the queue manager stops scanning the incoming (and deferred,
583
see below) queue. </p>
585
<p> Under normal conditions the incoming queue is nearly empty (has
586
only mode 0600 files), with the queue manager able to import new
587
messages into the active queue as soon as they become available.
590
<p> The incoming queue grows when the message input rate spikes
591
above the rate at which the queue manager can import messages into
592
the active queue. The main factor slowing down the queue manager
593
is transport queries to the trivial-rewrite service. If the queue
594
manager is routinely not keeping up, consider not using "slow"
595
lookup services (MySQL, LDAP, ...) for transport lookups or speeding
596
up the hosts that provide the lookup service. </p>
598
<p> The in_flow_delay parameter is used to clamp the input rate
599
when the queue manager starts to fall behind. The cleanup(8) service
600
will pause for $in_flow_delay seconds before creating a new queue
601
file if it cannot obtain a "token" from the queue manager. </p>
603
<p> Since the number of cleanup(8) processes is limited in most
604
cases by the SMTP server concurrency, the input rate can exceed
605
the output rate by at most "SMTP connection count" / $in_flow_delay
606
messages per second. </p>
608
<p> With a default process limit of 100, and an in_flow_delay of
609
1s, the coupling is strong enough to limit a single run-away injector
610
to 1 message per second, but is not strong enough to deflect an
611
excessive input rate from many sources at the same time. </p>
613
<p> If a server is being hammered from multiple directions, consider
614
raising the in_flow_delay to 10 seconds, but only if the incoming
615
queue is growing even while the active queue is not full and the
616
trivial-rewrite service is using a fast transport lookup mechanism.
619
<h3> <a name="active_queue"> The "active" queue </a> </h3>
621
<p> The queue manager is a delivery agent scheduler; it works to
622
ensure fast and fair delivery of mail to all destinations within
623
designated resource limits. </p>
625
<p> The active queue is somewhat analogous to an operating system's
626
process run queue. Messages in the active queue are ready to be
627
sent (runnable), but are not necessarily in the process of being
630
<p> While most Postfix administrators think of the "active" queue
631
as a directory on disk, the real "active" queue is a set of data
632
structures in the memory of the queue manager process. </p>
634
<p> Messages in the "maildrop", "hold", "incoming" and "deferred"
635
queues (see below) do not occupy memory; they are safely stored on
636
disk waiting for their turn to be processed. The envelope information
637
for messages in the "active" queue is managed in memory, allowing
638
the queue manager to do global scheduling, allocating available
639
delivery agent processes to an appropriate message in the active
642
<p> Within the active queue, (multi-recipient) messages are broken
643
up into groups of recipients that share the same transport/nexthop
644
combination; the group size is capped by the transport's recipient
645
concurrency limit. </p>
647
<p> Multiple recipient groups (from one or more messages) are queued
648
for delivery via the common transport/nexthop combination. The
649
destination concurrency limit for the transports caps the number
650
of simultaneous delivery attempts for each nexthop. Transports with
651
a recipient concurrency limit of 1 are special: these are grouped
652
by the actual recipient address rather than the nexthop, thereby
653
enabling per-recipient concurrency limits rather than per-domain
654
concurrency limits. Per-recipient limits are appropriate when
655
performing final delivery to mailboxes rather than when relaying
656
to a remote server. </p>
658
<p> Congestion occurs in the active queue when one or more destinations
659
drain slower than the corresponding message input rate. If a
660
destination is down for some time, the queue manager will mark it
661
dead, and immediately defer all mail for the destination without
662
trying to assign it to a delivery agent. In this case the messages
663
will quickly leave the active queue and end up in the deferred
664
queue. If the destination is instead simply slow, or there is a
665
problem causing an excessive arrival rate the active queue will
666
grow and will become dominated by mail to the congested destination.
669
<p> The only way to reduce congestion is to either reduce the input
670
rate or increase the throughput. Increasing the throughput requires
671
either increasing the concurrency or reducing the latency of
674
<p> For high volume sites a key tuning parameter is the number of
675
"smtp" delivery agents allocated to the "smtp" and "relay" transports.
676
High volume sites tend to send to many different destinations, many
677
of which may be down or slow, so a good fraction of the available
678
delivery agents will be blocked waiting for slow sites. Also mail
679
destined across the globe will incur large SMTP command-response
680
latencies, so high message throughput can only be achieved with
681
more concurrent delivery agents. </p>
683
<p> The default "smtp" process limit of 100 is good enough for most
684
sites, and may even need to be lowered for sites with low bandwidth
685
connections (no use increasing concurrency once the network pipe
686
is full). When one finds that the queue is growing on an "idle"
687
system (CPU, disk I/O and network not exhausted) the remaining
688
reason for congestion is insufficient concurrency in the face of
689
a high average latency. If the number of outbound SMTP connections
690
(either ESTABLISHED or SYN_SENT) reaches the process limit, mail
691
is draining slowly and the system and network are not loaded, raise
692
the "smtp" and/or "relay" process limits! </p>
694
<p> Especially for the "relay" transport, consider lower SMTP
695
connection timeouts (1-5 seconds) and higher than default destination
696
concurrency limits. Compute the expected latency when 1 out of N
697
of the MX hosts for a high volume site is down and not responding,
698
and make sure that the configured concurrency divided by this
699
latency exceeds the required steady-state message rate. If the
700
destination is managed by you, consider load balancers in front of
701
groups of MX hosts. Load balancers have higher uptime and will be
702
able to hide individual MX host failures. </p>
704
<p> If necessary, dedicate and tune custom transports for high
705
volume destinations. </p>
707
<p> Another common cause of congestion is unwarranted flushing of
708
the entire deferred queue. The deferred queue holds messages that
709
are likely to fail to be delivered and are also likely to be slow
710
to fail delivery (timeouts). This means that the most common reaction
711
to a large deferred queue (flush it!) is more than likely counter-
712
productive, and is likely to make the problem worse. Do not flush
713
the deferred queue unless you expect that most of its content has
714
recently become deliverable (e.g. relayhost back up after an outage)!
717
<p> Note that whenever the queue manager is restarted, there may
718
already be messages in the active queue directory, but the "real"
719
active queue in memory is empty. In order to recover the in-memory
720
state, the queue manager moves all the active queue messages
721
back into the incoming queue, and then uses its normal incoming
722
queue scan to refill the active queue. The process of moving all
723
the messages back and forth, redoing transport table (trivial-rewrite(8)
724
resolve service) lookups, and re-importing the messages back into
725
memory is expensive. At all costs, avoid frequent restarts of the
728
<h3> <a name="deferred_queue"> The "deferred" queue </a> </h3>
730
<p> When all the deliverable recipients for a message are delivered,
731
and for some recipients delivery failed for a transient reason (it
732
might succeed later), the message is placed in the deferred queue.
735
<p> The queue manager scans the deferred queue periodically. The
736
scan interval is controlled by the queue_run_delay parameter.
737
While a deferred queue scan is in progress, if an incoming queue
738
scan is also in progress (ideally these are brief since the incoming
739
queue should be short), the queue manager alternates between bringing
740
a new "incoming" message and a new "deferred" message into the
741
queue. This "round-robin" strategy prevents starvation of either
742
the incoming or the deferred queues. </p>
744
<p> Each deferred queue scan only brings a fraction of the deferred
745
queue back into the active queue for a retry. This is because each
746
message in the deferred queue is assigned a "cool-off" time when
747
it is deferred. This is done by time-warping the modification
748
times of the queue file into the future. The queue file is not
749
eligible for a retry if its modification time is not yet reached.
752
<p> The "cool-off" time is at least $minimal_backoff_time and at
753
most $maximal_backoff_time. The next retry time is set by doubling
754
the message's age in the queue, and adjusting up or down to lie
755
within the limits. This means that young messages are initially
756
retried more often than old messages. </p>
758
<p> If a high volume site routinely has large deferred queues, it
759
may be useful to adjust the queue_run_delay, minimal_backoff_time
760
and maximal_backoff_time to provide short enough delays on first
761
failure, with perhaps longer delays after multiple failures, to
762
reduce the retransmission rate of old messages and thereby reduce
763
the quantity of previously deferred mail in the active queue. </p>
765
<p> One common cause of large deferred queues is failure to validate
766
recipients at the SMTP input stage. Since spammers routinely launch
767
dictionary attacks from unrepliable sender addresses, the bounces
768
for invalid recipient addresses clog the deferred queue (and at
769
high volumes proportionally clog the active queue). Recipient
770
validation is strongly recommended through use of the local_recipient_maps
771
and relay_recipient_maps parameters. </p>
773
<p> When a host with lots of deferred mail is down for some time,
774
it is possible for the entire deferred queue to reach its retry
775
time simultaneously. This can lead to a very full active queue once
776
the host comes back up. The phenomenon can repeat approximately
777
every maximal_backoff_time seconds if the messages are again deferred
778
after a brief burst of congestion. Ideally, in the future Postfix
779
will add a random offset to the retry time (or use a combination
780
of strategies) to reduce the chances of repeated complete deferred
783
<h2><a name="credits">Credits</a></h2>
785
<p> The qshape(1) program was developed by Victor Duchovni of Morgan
786
Stanley, who also wrote the initial version of this document. </p>
added
removed
proto/QSHAPE_README.html
