1
<!-- $PostgreSQL: pgsql/doc/src/sgml/regress.sgml,v 1.45 2004-12-28 19:08:58 tgl Exp $ -->
4
<title id="regress-title">Regression Tests</title>
6
<indexterm zone="regress">
7
<primary>regression tests</primary>
10
<indexterm zone="regress">
11
<primary>test</primary>
15
The regression tests are a comprehensive set of tests for the SQL
16
implementation in <productname>PostgreSQL</productname>. They test
17
standard SQL operations as well as the extended capabilities of
18
<productname>PostgreSQL</productname>.
21
<sect1 id="regress-run">
22
<title>Running the Tests</title>
25
The regression tests can be run against an already installed and
26
running server, or using a temporary installation within the build
27
tree. Furthermore, there is a <quote>parallel</quote> and a
28
<quote>sequential</quote> mode for running the tests. The
29
sequential method runs each test script in turn, whereas the
30
parallel method starts up multiple server processes to run groups
31
of tests in parallel. Parallel testing gives confidence that
32
interprocess communication and locking are working correctly. For
33
historical reasons, the sequential test is usually run against an
34
existing installation and the parallel method against a temporary
35
installation, but there are no technical reasons for this.
39
To run the regression tests after building but before installation,
44
in the top-level directory. (Or you can change to
45
<filename>src/test/regress</filename> and run the command there.)
46
This will first build several auxiliary files, such as
47
some sample user-defined trigger functions, and then run the test driver
48
script. At the end you should see something like
51
======================
53
======================
56
or otherwise a note about which tests failed. See <xref
57
linkend="regress-evaluation"> below before assuming that a
58
<quote>failure</> represents a serious problem.
62
Because this test method runs a temporary server, it will not work
63
when you are the root user (since the server will not start as root).
64
If you already did the build as root, you do not have to start all
65
over. Instead, make the regression test directory writable by
66
some other user, log in as that user, and restart the tests.
69
<prompt>root# </prompt><userinput>chmod -R a+w src/test/regress</userinput>
70
<prompt>root# </prompt><userinput>chmod -R a+w contrib/spi</userinput>
71
<prompt>root# </prompt><userinput>su - joeuser</userinput>
72
<prompt>joeuser$ </prompt><userinput>cd <replaceable>top-level build directory</></userinput>
73
<prompt>joeuser$ </prompt><userinput>gmake check</userinput>
75
(The only possible <quote>security risk</quote> here is that other
76
users might be able to alter the regression test results behind
77
your back. Use common sense when managing user permissions.)
80
Alternatively, run the tests after installation.
84
If you have configured <productname>PostgreSQL</productname> to install
85
into a location where an older <productname>PostgreSQL</productname>
86
installation already exists, and you perform <literal>gmake check</>
87
before installing the new version, you may find that the tests fail
88
because the new programs try to use the already-installed shared
89
libraries. (Typical symptoms are complaints about undefined symbols.)
90
If you wish to run the tests before overwriting the old installation,
91
you'll need to build with <literal>configure --disable-rpath</>.
92
It is not recommended that you use this option for the final installation,
97
The parallel regression test starts quite a few processes under your
98
user ID. Presently, the maximum concurrency is twenty parallel test
99
scripts, which means sixty processes: there's a server process, a
100
<application>psql</>, and usually a shell parent process for the
101
<application>psql</> for each test script.
102
So if your system enforces a per-user limit on the number of processes,
103
make sure this limit is at least seventy-five or so, else you may get
104
random-seeming failures in the parallel test. If you are not in
105
a position to raise the limit, you can cut down the degree of parallelism
106
by setting the <literal>MAX_CONNECTIONS</> parameter. For example,
108
gmake MAX_CONNECTIONS=10 check
110
runs no more than ten tests concurrently.
114
On some systems, the default Bourne-compatible shell
115
(<filename>/bin/sh</filename>) gets confused when it has to manage
116
too many child processes in parallel. This may cause the parallel
117
test run to lock up or fail. In such cases, specify a different
118
Bourne-compatible shell on the command line, for example:
120
gmake SHELL=/bin/ksh check
122
If no non-broken shell is available, you may be able to work around the
123
problem by limiting the number of connections, as shown above.
127
To run the tests after installation<![%standalone-ignore;[ (see <xref linkend="installation">)]]>,
128
initialize a data area and start the
129
server, <![%standalone-ignore;[as explained in <xref linkend="runtime">, ]]> then type
133
or for a parallel test
135
gmake installcheck-parallel
137
The tests will expect to contact the server at the local host and the
138
default port number, unless directed otherwise by <envar>PGHOST</envar> and
139
<envar>PGPORT</envar> environment variables.
143
<sect1 id="regress-evaluation">
144
<title>Test Evaluation</title>
147
Some properly installed and fully functional
148
<productname>PostgreSQL</productname> installations can
149
<quote>fail</quote> some of these regression tests due to
150
platform-specific artifacts such as varying floating-point representation
151
and time zone support. The tests are currently evaluated using a simple
152
<command>diff</command> comparison against the outputs
153
generated on a reference system, so the results are sensitive to
154
small system differences. When a test is reported as
155
<quote>failed</quote>, always examine the differences between
156
expected and actual results; you may well find that the
157
differences are not significant. Nonetheless, we still strive to
158
maintain accurate reference files across all supported platforms,
159
so it can be expected that all tests pass.
163
The actual outputs of the regression tests are in files in the
164
<filename>src/test/regress/results</filename> directory. The test
165
script uses <command>diff</command> to compare each output
166
file against the reference outputs stored in the
167
<filename>src/test/regress/expected</filename> directory. Any
168
differences are saved for your inspection in
169
<filename>src/test/regress/regression.diffs</filename>. (Or you
170
can run <command>diff</command> yourself, if you prefer.)
174
<title>Error message differences</title>
177
Some of the regression tests involve intentional invalid input
178
values. Error messages can come from either the
179
<productname>PostgreSQL</productname> code or from the host
180
platform system routines. In the latter case, the messages may
181
vary between platforms, but should reflect similar
182
information. These differences in messages will result in a
183
<quote>failed</quote> regression test that can be validated by
189
<title>Locale differences</title>
192
If you run the tests against an already-installed server that was
193
initialized with a collation-order locale other than C, then
194
there may be differences due to sort order and follow-up
195
failures. The regression test suite is set up to handle this
196
problem by providing alternative result files that together are
197
known to handle a large number of locales. For example, for the
198
<literal>char</literal> test, the expected file
199
<filename>char.out</filename> handles the <literal>C</> and <literal>POSIX</> locales,
200
and the file <filename>char_1.out</filename> handles many other
201
locales. The regression test driver will automatically pick the
202
best file to match against when checking for success and for
203
computing failure differences. (This means that the regression
204
tests cannot detect whether the results are appropriate for the
205
configured locale. The tests will simply pick the one result
206
file that works best.)
210
If for some reason the existing expected files do not cover some
211
locale, you can add a new file. The naming scheme is
212
<literal><replaceable>testname</>_<replaceable>digit</>.out</>.
213
The actual digit is not significant. Remember that the
214
regression test driver will consider all such files to be equally
215
valid test results. If the test results are platform-specific,
216
the technique described in <xref linkend="regress-platform">
217
should be used instead.
222
<title>Date and time differences</title>
225
A few of the queries in the <filename>horology</filename> test will
226
fail if you run the test on the day of a daylight-saving time
227
changeover, or the day after one. These queries expect that
228
the intervals between midnight yesterday, midnight today and
229
midnight tomorrow are exactly twenty-four hours — which is wrong
230
if daylight-saving time went into or out of effect meanwhile.
235
Because USA daylight-saving time rules are used, this problem always
236
occurs on the first Sunday of April, the last Sunday of October,
237
and their following Mondays, regardless of when daylight-saving time
238
is in effect where you live. Also note that the problem appears or
239
disappears at midnight Pacific time (UTC-7 or UTC-8), not midnight
240
your local time. Thus the failure may appear late on Saturday or
241
persist through much of Tuesday, depending on where you live.
246
Most of the date and time results are dependent on the time zone
247
environment. The reference files are generated for time zone
248
<literal>PST8PDT</literal> (Berkeley, California), and there will be
249
apparent failures if the tests are not run with that time zone setting.
250
The regression test driver sets environment variable
251
<envar>PGTZ</envar> to <literal>PST8PDT</literal>, which normally
252
ensures proper results.
257
<title>Floating-point differences</title>
260
Some of the tests involve computing 64-bit floating-point numbers (<type>double
261
precision</type>) from table columns. Differences in
262
results involving mathematical functions of <type>double
263
precision</type> columns have been observed. The <literal>float8</> and
264
<literal>geometry</> tests are particularly prone to small differences
265
across platforms, or even with different compiler optimization options.
266
Human eyeball comparison is needed to determine the real
267
significance of these differences which are usually 10 places to
268
the right of the decimal point.
272
Some systems display minus zero as <literal>-0</>, while others
273
just show <literal>0</>.
277
Some systems signal errors from <function>pow()</function> and
278
<function>exp()</function> differently from the mechanism
279
expected by the current <productname>PostgreSQL</productname>
285
<title>Row ordering differences</title>
288
You might see differences in which the same rows are output in a
289
different order than what appears in the expected file. In most cases
290
this is not, strictly speaking, a bug. Most of the regression test
291
scripts are not so pedantic as to use an <literal>ORDER BY</> for every single
292
<literal>SELECT</>, and so their result row orderings are not well-defined
293
according to the letter of the SQL specification. In practice, since we are
294
looking at the same queries being executed on the same data by the same
295
software, we usually get the same result ordering on all platforms, and
296
so the lack of <literal>ORDER BY</> isn't a problem. Some queries do exhibit
297
cross-platform ordering differences, however. (Ordering differences
298
can also be triggered by non-C locale settings.)
302
Therefore, if you see an ordering difference, it's not something to
303
worry about, unless the query does have an <literal>ORDER BY</> that your result
304
is violating. But please report it anyway, so that we can add an
305
<literal>ORDER BY</> to that particular query and thereby eliminate the bogus
306
<quote>failure</quote> in future releases.
310
You might wonder why we don't order all the regression test queries explicitly to
311
get rid of this issue once and for all. The reason is that that would
312
make the regression tests less useful, not more, since they'd tend
313
to exercise query plan types that produce ordered results to the
314
exclusion of those that don't.
319
<title>The <quote>random</quote> test</title>
322
The <literal>random</literal> test script is intended to produce
323
random results. In rare cases, this causes the random regression
326
diff results/random.out expected/random.out
328
should produce only one or a few lines of differences. You need
329
not worry unless the random test fails repeatedly.
334
<!-- We might want to move the following section into the developer's guide. -->
335
<sect1 id="regress-platform">
336
<title>Platform-specific comparison files</title>
339
Since some of the tests inherently produce platform-specific
340
results, we have provided a way to supply platform-specific result
341
comparison files. Frequently, the same variation applies to
342
multiple platforms; rather than supplying a separate comparison
343
file for every platform, there is a mapping file that defines
344
which comparison file to use. So, to eliminate bogus test
345
<quote>failures</quote> for a particular platform, you must choose
346
or make a variant result file, and then add a line to the mapping
347
file, which is <filename>src/test/regress/resultmap</filename>.
351
Each line in the mapping file is of the form
353
testname/platformpattern=comparisonfilename
355
The test name is just the name of the particular regression test
356
module. The platform pattern is a pattern in the style of the Unix
357
tool <command>expr</> (that is, a regular expression with an implicit
358
<literal>^</literal> anchor
359
at the start). It is matched against the platform name as printed
360
by <command>config.guess</command> followed by
361
<literal>:gcc</literal> or <literal>:cc</literal>, depending on
362
whether you use the GNU compiler or the system's native compiler
363
(on systems where there is a difference). The comparison file
364
name is the name of the substitute result comparison file.
368
For example: some systems interpret very small floating-point values
369
as zero, rather than reporting an underflow error. This causes a
370
few differences in the <filename>float8</> regression test.
371
Therefore, we provide a variant comparison file,
372
<filename>float8-small-is-zero.out</filename>, which includes
373
the results to be expected on these systems. To silence the bogus
374
<quote>failure</quote> message on <systemitem>OpenBSD</systemitem>
375
platforms, <filename>resultmap</filename> includes
377
float8/i.86-.*-openbsd=float8-small-is-zero
379
which will trigger on any machine for which the output of
380
<command>config.guess</command> matches <literal>i.86-.*-openbsd</literal>.
382
in <filename>resultmap</> select the variant comparison file for other
383
platforms where it's appropriate.
390
<!-- Keep this comment at the end of the file
395
sgml-minimize-attributes:nil
396
sgml-always-quote-attributes:t
399
sgml-parent-document:nil
400
sgml-default-dtd-file:"./reference.ced"
401
sgml-exposed-tags:nil
402
sgml-local-catalogs:("/usr/lib/sgml/catalog")
403
sgml-local-ecat-files:nil
added
removed
doc/src/sgml/regress.sgml
