1
//-----------------------------------------------------------------------------
3
// Statistics Processor
5
// Last Update: 04/11/2001
7
//-----------------------------------------------------------------------------
9
//-----------------------------------------------------------------------------
11
//-----------------------------------------------------------------------------
13
This statistics processor is a Cougar II (Tsunami) feature.
14
It is designed to replace the StatAggregate.cc and portion of WebOverview.cc
15
for scalability, maintainability, and ability to customize reasons.
17
The statistics processor aggregate/calculate traffic server statistics based
18
on a XML-based configuration file. At the processor's initialization, the
19
configuration file is read and parsed. Each set of calculation is then stored
20
as a C++ object, called StatObject; and each StatObject is linked into a list,
21
called StatObjectList.
23
The mgmt2/traffic_manager threads will invoke the StatObjectList->Eval() to
24
perform the statistics aggregation and calculation within its event loop. As
25
Eval() is call, each StatObject in StatObjectList is evaluated.
27
Recall: StatAggregate.cc aggregates/calculates/copies proxy.process.* TS
28
variables to proxy.node.* variables. Similarly, WebOverview.cc aggregate
29
proxy.node.* variables to their corresponding proxy.cluster.* variable.
31
So there are two types of calculations in the statistics processor: NodeEval
32
and ClusterEval. As their names imply, they aggregate node-based statistics
33
and clsuter-based statistics, respectively. We call the different basis of
34
statistics aggregation as "scope". (See "Destination Attributes")
36
In the cluster-based statistics, the aggregation is further divided into two
37
types: sum and re-calculate. Sum refers calculating the proxy.cluster.*
38
variable by simply summing all required proxy.node.* variables from nodes in
39
the cluster. Re-calculate refers to summing all proxy.nodes.* variables that
40
are used in the process of calculation before performing the calculation.
41
An analogy would be, summing all open connection in the cluster vs. the
42
average hit rate in the cluster.
44
//-----------------------------------------------------------------------------
45
// Destination Attributes
46
//-----------------------------------------------------------------------------
54
summing the corresponding node variable across all nodes in the cluster.
62
//-----------------------------------------------------------------------------
63
// Predefined Constants and Functions
64
//-----------------------------------------------------------------------------
68
. BYTES_TO_MB_SCALE (1/(1024*1024.0))
69
- convert bytes to mega-bytes
71
. MBIT_TO_KBIT_SCALE (1000.0)
72
- convert mega-bits to kilo-bits
74
. SECOND_TO_MILLISECOND_SCALE (1000.0)
75
- convert seconds to milliseconds
77
. PCT_TO_INTPCT_SCALE (100.0)
78
- convert ratios to percentage
81
- converting milli-seconds to seconds
83
. BYTES_TO_MBIT_SCALE (8/1000000.0)
84
- convert bytes to mega-bits
88
- the number of milliseconds since last update. Usually used in
89
combination of HRTIME_SECOND which computes the number of seconds
92
//-----------------------------------------------------------------------------
94
//-----------------------------------------------------------------------------
96
The statistics processor is designed to replace StatAggregate.cc and part of
97
the WebOverview. The first thing to test StatProcessor is to comment the
98
aggregateNodeRecords() and doClusterAg() calls from proxy/mgmt2/Main.cc.
100
The next step is to replace the above function calls with StatProcessor::
103
This statistics processor is a rather complicated module in traffic manager.
104
Hence it can't be easily tested. We divided the test into multiple sections.
106
1) Node-based Simple Aggregation
107
- simply performs those aggregation that are node-based and the aggregation
108
is performed every time statProcess() is invoked.
109
E.g.: hit rate = doc. hit / doc. served.
111
2) Node-based Time-Delta Aggregation
112
- performs those aggregation that are node-based but the operation is only
113
perform in a regular interval AND one of more variables in the
114
calculation is obtained by calculating the difference between the last
115
updated value and the current value. E.g. average connections per second
116
is calculated by subtracting the 10 seconds ago connection count from the
117
current connection count and divide the quotient by 10.
119
Repeat the about 2 testes with cluster-based variables. So, we have, at least,
122
Developing a PASS/FAIL unit test that will test the statistics processor is not
123
cost-efficient. The approach we are going to use is to display the input value
124
and the output value before and after the calculation is done.
126
Let's subdivide the testes in two stages:
128
Stage 1 : Synthetic Data
129
------------------------
130
We control the testing environment by setting the input values. This will test
131
the correctness of the statistics processor in a controlled environment. PASS/
132
FAIL is determined by matching the input/output values.
136
Submitting network traffic through traffic server with load tools like jtest and
137
ftest, dumps the statistics to a text file, periodically and examines the
140
//-----------------------------------------------------------------------------
142
//-----------------------------------------------------------------------------
143
The most concerning question for QA engineers is "how can I tell if the
144
Statistics Processor is working correctly?"
146
Recall, the new Statistics Processor is meant to replace the StatAggregate.cc
147
and part of the WebOverview.cc. In essence, you should not see any apparent
150
If you ever see a value of -9999.0 (or -9999), then there is an error in
151
computing that value.
160
- specifies the variable that stores that result value.
162
built-in: variables that are built-in/defined in traffic server
163
custom: variables that are introducted to be temporary storage or
164
variables that are introdcuted by the client.
165
- default attributes:
169
- variable need to computer the <dst>
171
node: this is a proxy.node.* variables
172
cluster: the is a proxy.node.* variables but summing over all
173
nodes in the cluster.
174
- default attributes:
178
- specifics what is the smallest possible value for <dst>. For values
179
smaller than <min>, the <defualt> is used.
182
- specifics what is the largest possible value for <dst>. For values
183
larger than <max>, the <defualt> is used.
186
- specifics what value to be assigned to <dst> is the result <dst>
187
value is smaller then <min> or larger then <max>
189
RULES: (some of these are enfored by the DTD anyways)
190
- all operator and operand in <expr> MUST BE separated by a single space.
191
- the order of the tags matters
192
- the order of each entry matters
193
- each statistics entry has have at most 1 <dst>
196
DEFINED CONSTANT (in alphabetical order)
198
* Origin: utils/WebMgmtUtils.h
199
* Value: (1/(1024*1024.0))
205
* _MBIT_TO_KBIT_SCALE
206
* Origin: utils/WebMgmtUtils.h
209
* _PCT_TO_INTPCT_SCALE
210
* Origin: utils/WebMgmtUtils.h
213
* _SECOND_TO_MILLISECOND_SCALE
214
* Origin: utils/WebMgmtUtils.h
added
removed
proxy/mgmt2/stats/spec
