« back to all changes in this revision
Viewing changes to docs/user/pt-tcp-model.rst
- browse files at revision 123
- compare with another revision
- download diff
- download tarball
- view history from revision 123
- Committer: Daniel Nichter
- Date: 2011-12-29 22:16:13 UTC
- Revision ID: daniel@percona.com-20111229221613-oprw0q2td34r6zkb
Update release date.
- files removed:
- docs/user/authors.rst
- files modified:
- Changelog
added
removed
docs/user/pt-tcp-model.rst
1
2
############
3
pt-tcp-model
4
############
5
6
.. highlight:: perl
7
8
9
****
10
NAME
11
****
12
13
14
pt-tcp-model - Transform tcpdump into metrics that permit performance and scalability modeling.
15
16
17
********
18
SYNOPSIS
19
********
20
21
22
Usage: pt-tcp-model [OPTION...] [FILE]
23
24
pt-tcp-model parses and analyzes tcpdump files. With no FILE, or when
25
FILE is -, it read standard input.
26
27
Dump TCP requests and responses to a file, capturing only the packet headers to
28
avoid dropped packets, and ignoring any packets without a payload (such as
29
ack-only packets). Capture port 3306 (MySQL database traffic). Note that to
30
avoid line breaking in terminals and man pages, the TCP filtering expression
31
that follows has a line break at the end of the second line; you should omit
32
this from your tcpdump command.
33
34
35
.. code-block:: perl
36
37
tcpdump -s 384 -i any -nnq -tttt \
38
'tcp port 3306 and (((ip[2:2] - ((ip[0]&0xf)<<2))
39
- ((tcp[12]&0xf0)>>2)) != 0)' \
40
> /path/to/tcp-file.txt
41
42
43
Extract individual response times, sorted by end time:
44
45
46
.. code-block:: perl
47
48
pt-tcp-model /path/to/tcp-file.txt > requests.txt
49
50
51
Sort the result by arrival time, for input to the next step:
52
53
54
.. code-block:: perl
55
56
sort -n -k1,1 requests.txt > sorted.txt
57
58
59
Slice the result into 10-second intervals and emit throughput, concurrency, and
60
response time metrics for each interval:
61
62
63
.. code-block:: perl
64
65
pt-tcp-model --type=requests --run-time=10 sorted.txt > sliced.txt
66
67
68
Transform the result for modeling with Aspersa's usl tool, discarding the first
69
and last line of each file if you specify multiple files (the first and last
70
line are normally incomplete observation periods and are aberrant):
71
72
73
.. code-block:: perl
74
75
for f in sliced.txt; do
76
tail -n +2 "$f" | head -n -1 | awk '{print $2, $3, $7/$4}'
77
done > usl-input.txt
78
79
80
81
*****
82
RISKS
83
*****
84
85
86
The following section is included to inform users about the potential risks,
87
whether known or unknown, of using this tool. The two main categories of risks
88
are those created by the nature of the tool (e.g. read-only tools vs. read-write
89
tools) and those created by bugs.
90
91
pt-tcp-model merely reads and transforms its input, printing it to the output.
92
It should be very low risk.
93
94
At the time of this release, we know of no bugs that could cause serious harm
95
to users.
96
97
The authoritative source for updated information is always the online issue
98
tracking system. Issues that affect this tool will be marked as such. You can
99
see a list of such issues at the following URL:
100
`http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.
101
102
See also "BUGS" for more information on filing bugs and getting help.
103
104
105
***********
106
DESCRIPTION
107
***********
108
109
110
This tool recognizes requests and responses in a TCP stream, and extracts the
111
"conversations". You can use it to capture the response times of individual
112
queries to a database, for example. It expects the TCP input to be in the
113
following format, which should result from the sample shown in the SYNOPSIS:
114
115
116
.. code-block:: perl
117
118
<date> <time.microseconds> IP <IP.port> > <IP.port>: <junk>
119
120
121
The tool watches for "incoming" packets to the port you specify with the
122
"--watch-server" option. This begins a request. If multiple inbound packets
123
follow each other, then by default the last inbound packet seen determines the
124
time at which the request is assumed to begin. This is logical if one assumes
125
that a server must receive the whole SQL statement before beginning execution,
126
for example.
127
128
When the first outbound packet is seen, the server is considered to have
129
responded to the request. The tool might see an inbound packet, but never see a
130
response. This can happen when the kernel drops packets, for example. As a
131
result, the tool never prints a request unless it sees the response to it.
132
However, the tool actually does not print any request until it sees the "last"
133
outbound packet. It determines this by waiting for either another inbound
134
packet, or EOF, and then considers the previous inbound/outbound pair to be
135
complete. As a result, the tool prints requests in a relatively random order.
136
Most types of analysis require processing in either arrival or completion order.
137
Therefore, the second type of processing this tool can do requires that you sort
138
the output from the first stage and supply it as input.
139
140
The second type of processing is selected with the "--type" option set to
141
"requests". In this mode, the tool reads a group of requests and aggregates
142
them, then emits the aggregated metrics.
143
144
145
******
146
OUTPUT
147
******
148
149
150
In the default mode (parsing tcpdump output), requests are printed out one per
151
line, in the following format:
152
153
154
.. code-block:: perl
155
156
<id> <start> <end> <elapsed> <IP:port>
157
158
159
The ID is an incrementing number, assigned in arrival order in the original TCP
160
traffic. The start and end timestamps, and the elapsed time, can be customized
161
with the "--start-end" option.
162
163
In "--type=requests" mode, the tool prints out one line per time interval as
164
defined by "--run-time", with the following columns: ts, concurrency,
165
throughput, arrivals, completions, busy_time, weighted_time, sum_time,
166
variance_mean, quantile_time, obs_time. A detailed explanation follows:
167
168
169
ts
170
171
The timestamp that defines the beginning of the interval.
172
173
174
175
concurrency
176
177
The average number of requests resident in the server during the interval.
178
179
180
181
throughput
182
183
The number of arrivals per second during the interval.
184
185
186
187
arrivals
188
189
The number of arrivals during the interval.
190
191
192
193
completions
194
195
The number of completions during the interval.
196
197
198
199
busy_time
200
201
The total amount of time during which at least one request was resident in
202
the server during the interval.
203
204
205
206
weighted_time
207
208
The total response time of all the requests resident in the server during the
209
interval, including requests that neither arrived nor completed during the
210
interval.
211
212
213
214
sum_time
215
216
The total response time of all the requests that arrived in the interval.
217
218
219
220
variance_mean
221
222
The variance-to-mean ratio (index of dispersion) of the response times of the
223
requests that arrived in the interval.
224
225
226
227
quantile_time
228
229
The Nth percentile response time for all the requests that arrived in the
230
interval. See also "--quantile".
231
232
233
234
obs_time
235
236
The length of the observation time window. This will usually be the same as the
237
interval length, except for the first and last intervals in a file, which might
238
have a shorter observation time.
239
240
241
242
243
*******
244
OPTIONS
245
*******
246
247
248
This tool accepts additional command-line arguments. Refer to the
249
"SYNOPSIS" and usage information for details.
250
251
252
--config
253
254
type: Array
255
256
Read this comma-separated list of config files; if specified, this must be the
257
first option on the command line.
258
259
260
261
--help
262
263
Show help and exit.
264
265
266
267
--progress
268
269
type: array; default: time,30
270
271
Print progress reports to STDERR. The value is a comma-separated list with two
272
parts. The first part can be percentage, time, or iterations; the second part
273
specifies how often an update should be printed, in percentage, seconds, or
274
number of iterations.
275
276
277
278
--quantile
279
280
type: float
281
282
The percentile for the last column when "--type" is "requests" (default .99).
283
284
285
286
--run-time
287
288
type: float
289
290
The size of the aggregation interval in seconds when "--type" is "requests"
291
(default 1). Fractional values are permitted.
292
293
294
295
--start-end
296
297
type: Array; default: ts,end
298
299
Define how the arrival and completion timestamps of a query, and thus its
300
response time (elapsed time) are computed. Recall that there may be multiple
301
inbound and outbound packets per request and response, and refer to the
302
following ASCII diagram. Suppose that a client sends a series of three inbound
303
(I) packets to the server, whch computes the result and then sends two outbound
304
(O) packets back:
305
306
307
.. code-block:: perl
308
309
I I I ..................... O O
310
|<---->|<---response time----->|<-->|
311
ts0 ts end end1
312
313
314
By default, the query is considered to arrive at time ts, and complete at time
315
end. However, this might not be what you want. Perhaps you do not want to
316
consider the query to have completed until time end1. You can accomplish this
317
by setting this option to \ ``ts,end1``\ .
318
319
320
321
--type
322
323
type: string
324
325
The type of input to parse (default tcpdump). The permitted types are
326
327
328
tcpdump
329
330
The parser expects the input to be formatted with the following options: \ ``-x -n
331
-q -tttt``\ . For example, if you want to capture output from your local machine,
332
you can do something like the following (the port must come last on FreeBSD):
333
334
335
.. code-block:: perl
336
337
tcpdump -s 65535 -x -nn -q -tttt -i any -c 1000 port 3306 \
338
> mysql.tcp.txt
339
pt-query-digest --type tcpdump mysql.tcp.txt
340
341
342
The other tcpdump parameters, such as -s, -c, and -i, are up to you. Just make
343
sure the output looks like this (there is a line break in the first line to
344
avoid man-page problems):
345
346
347
.. code-block:: perl
348
349
2009-04-12 09:50:16.804849 IP 127.0.0.1.42167
350
> 127.0.0.1.3306: tcp 37
351
352
353
All MySQL servers running on port 3306 are automatically detected in the
354
tcpdump output. Therefore, if the tcpdump out contains packets from
355
multiple servers on port 3306 (for example, 10.0.0.1:3306, 10.0.0.2:3306,
356
etc.), all packets/queries from all these servers will be analyzed
357
together as if they were one server.
358
359
If you're analyzing traffic for a protocol that is not running on port
360
3306, see "--watch-server".
361
362
363
364
365
366
--version
367
368
Show version and exit.
369
370
371
372
--watch-server
373
374
type: string; default: 10.10.10.10:3306
375
376
This option tells pt-tcp-model which server IP address and port (such as
377
"10.0.0.1:3306") to watch when parsing tcpdump for "--type" tcpdump. If you
378
don't specify it, the tool watches all servers by looking for any IP address
379
using port 3306. If you're watching a server with a non-standard port, this
380
won't work, so you must specify the IP address and port to watch.
381
382
Currently, IP address filtering isn't implemented; so even though you must
383
specify the option in IP:port form, it ignores the IP and only looks at the port
384
number.
385
386
387
388
389
***********
390
ENVIRONMENT
391
***********
392
393
394
The environment variable \ ``PTDEBUG``\ enables verbose debugging output to STDERR.
395
To enable debugging and capture all output to a file, run the tool like:
396
397
398
.. code-block:: perl
399
400
PTDEBUG=1 pt-tcp-model ... > FILE 2>&1
401
402
403
Be careful: debugging output is voluminous and can generate several megabytes
404
of output.
405
406
407
*******************
408
SYSTEM REQUIREMENTS
409
*******************
410
411
412
You need Perl, DBI, DBD::mysql, and some core packages that ought to be
413
installed in any reasonably new version of Perl.
414
415
416
****
417
BUGS
418
****
419
420
421
For a list of known bugs, see `http://www.percona.com/bugs/pt-tcp-model <http://www.percona.com/bugs/pt-tcp-model>`_.
422
423
Please report bugs at `https://bugs.launchpad.net/percona-toolkit <https://bugs.launchpad.net/percona-toolkit>`_.
424
Include the following information in your bug report:
425
426
427
\* Complete command-line used to run the tool
428
429
430
431
\* Tool "--version"
432
433
434
435
\* MySQL version of all servers involved
436
437
438
439
\* Output from the tool including STDERR
440
441
442
443
\* Input files (log/dump/config files, etc.)
444
445
446
447
If possible, include debugging output by running the tool with \ ``PTDEBUG``\ ;
448
see "ENVIRONMENT".
449
450
451
***********
452
DOWNLOADING
453
***********
454
455
456
Visit `http://www.percona.com/software/percona-toolkit/ <http://www.percona.com/software/percona-toolkit/>`_ to download the
457
latest release of Percona Toolkit. Or, get the latest release from the
458
command line:
459
460
461
.. code-block:: perl
462
463
wget percona.com/get/percona-toolkit.tar.gz
464
465
wget percona.com/get/percona-toolkit.rpm
466
467
wget percona.com/get/percona-toolkit.deb
468
469
470
You can also get individual tools from the latest release:
471
472
473
.. code-block:: perl
474
475
wget percona.com/get/TOOL
476
477
478
Replace \ ``TOOL``\ with the name of any tool.
479
480
481
*******
482
AUTHORS
483
*******
484
485
486
Baron Schwartz
487
488
489
*********************
490
ABOUT PERCONA TOOLKIT
491
*********************
492
493
494
This tool is part of Percona Toolkit, a collection of advanced command-line
495
tools developed by Percona for MySQL support and consulting. Percona Toolkit
496
was forked from two projects in June, 2011: Maatkit and Aspersa. Those
497
projects were created by Baron Schwartz and developed primarily by him and
498
Daniel Nichter, both of whom are employed by Percona. Visit
499
`http://www.percona.com/software/ <http://www.percona.com/software/>`_ for more software developed by Percona.
500
501
502
********************************
503
COPYRIGHT, LICENSE, AND WARRANTY
504
********************************
505
506
507
This program is copyright 2011 Baron Schwartz, 2011 Percona Inc.
508
Feedback and improvements are welcome.
509
510
THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
511
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
512
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
513
514
This program is free software; you can redistribute it and/or modify it under
515
the terms of the GNU General Public License as published by the Free Software
516
Foundation, version 2; OR the Perl Artistic License. On UNIX and similar
517
systems, you can issue \`man perlgpl' or \`man perlartistic' to read these
518
licenses.
519
520
You should have received a copy of the GNU General Public License along with
521
this program; if not, write to the Free Software Foundation, Inc., 59 Temple
522
Place, Suite 330, Boston, MA 02111-1307 USA.
523
524
525
*******
526
VERSION
527
*******
528
529
530
pt-tcp-model 1.0.2
531
Loggerhead is a web-based interface for Breezy
Version: 2.0.1