3
The regression tests are a comprehensive set of tests for the SQL
4
implementation in PostgreSQL. They test standard SQL operations as well
5
as the extended capabilities of PostgreSQL.
6
__________________________________________________________________
10
The regression tests can be run against an already installed and
11
running server, or using a temporary installation within the build
12
tree. Furthermore, there is a "parallel" and a "sequential" mode for
13
running the tests. The sequential method runs each test script alone,
14
while the parallel method starts up multiple server processes to run
15
groups of tests in parallel. Parallel testing gives confidence that
16
interprocess communication and locking are working correctly.
17
__________________________________________________________________
19
Running the Tests Against a Temporary Installation
21
To run the parallel regression tests after building but before
25
in the top-level directory. (Or you can change to "src/test/regress"
26
and run the command there.) This will first build several auxiliary
27
files, such as sample user-defined trigger functions, and then run the
28
test driver script. At the end you should see something like:
29
=======================
31
=======================
33
or otherwise a note about which tests failed. See the Section called
34
Test Evaluation below before assuming that a "failure" represents a
37
Because this test method runs a temporary server, it will not work when
38
you are the root user (since the server will not start as root). If you
39
already did the build as root, you do not have to start all over.
40
Instead, make the regression test directory writable by some other
41
user, log in as that user, and restart the tests. For example:
42
root# chmod -R a+w src/test/regress
44
joeuser$ cd top-level build directory
47
(The only possible "security risk" here is that other users might be
48
able to alter the regression test results behind your back. Use common
49
sense when managing user permissions.)
51
Alternatively, run the tests after installation.
53
If you have configured PostgreSQL to install into a location where an
54
older PostgreSQL installation already exists, and you perform gmake
55
check before installing the new version, you might find that the tests
56
fail because the new programs try to use the already-installed shared
57
libraries. (Typical symptoms are complaints about undefined symbols.)
58
If you wish to run the tests before overwriting the old installation,
59
you'll need to build with configure --disable-rpath. It is not
60
recommended that you use this option for the final installation,
63
The parallel regression test starts quite a few processes under your
64
user ID. Presently, the maximum concurrency is twenty parallel test
65
scripts, which means forty processes: there's a server process and a
66
psql process for each test script. So if your system enforces a
67
per-user limit on the number of processes, make sure this limit is at
68
least fifty or so, else you might get random-seeming failures in the
69
parallel test. If you are not in a position to raise the limit, you can
70
cut down the degree of parallelism by setting the MAX_CONNECTIONS
71
parameter. For example:
72
gmake MAX_CONNECTIONS=10 check
74
runs no more than ten tests concurrently.
75
__________________________________________________________________
77
Running the Tests Against an Existing Installation
79
To run the tests after installation, initialize a data area and start
80
the server, then type:
83
or for a parallel test:
84
gmake installcheck-parallel
86
The tests will expect to contact the server at the local host and the
87
default port number, unless directed otherwise by PGHOST and PGPORT
88
environment variables.
90
The source distribution also contains regression tests for the optional
91
procedural languages and for some of the "contrib" modules. At present,
92
these tests can be used only against an already-installed server. To
93
run the tests for all procedural languages that have been built and
94
installed, change to the "src/pl" directory of the build tree and type:
97
You can also do this in any of the subdirectories of "src/pl" to run
98
tests for just one procedural language. To run the tests for all
99
"contrib" modules that have them, change to the "contrib" directory of
100
the build tree and type:
103
The "contrib" modules must have been built and installed first. You can
104
also do this in a subdirectory of "contrib" to run the tests for just
106
__________________________________________________________________
110
The source distribution also contains regression tests of the static
111
behaviour of Hot Standby. These tests require a running primary server
112
and a running standby server that is accepting new WAL changes from the
113
primary using either file-based log shipping or streaming replication.
114
Those servers are not automatically created for you, nor is the setup
115
documented here. Please check the various sections of the documentation
116
already devoted to the required commands and related issues.
118
First create a database called "regression" on the primary.
119
psql -h primary -c "CREATE DATABASE regression"
121
Next, run a preparatory script on the primary in the regression
122
database: "src/test/regress/sql/hs_primary_setup.sql", and allow for
123
the changes to propagate to the standby, for example
124
psql -h primary -f src/test/regress/sql/hs_primary_setup.sql regression
126
Now confirm that the default connection for the tester is the standby
127
server under test and then run the standbycheck target from the
128
regression directory:
132
Some extreme behaviours can also be generated on the primary using the
133
script: "src/test/regress/sql/hs_primary_extremes.sql" to allow the
134
behaviour of the standby to be tested.
136
Additional automated testing may be available in later releases.
137
__________________________________________________________________
141
By default, the tests against a temporary installation use the locale
142
defined in the current environment and the corresponding database
143
encoding as determined by "initdb". It can be useful to test different
144
locales by setting the appropriate environment variables, for example:
146
gmake check LC_COLLATE=en_US.utf8 LC_CTYPE=fr_CA.utf8
148
For implementation reasons, setting LC_ALL does not work for this
149
purpose; all the other locale-related environment variables do work.
151
When testing against an existing installation, the locale is determined
152
by the existing database cluster and cannot be set separately for the
155
You can also choose the database encoding explicitly by setting the
156
variable ENCODING, for example:
157
gmake check LANG=C ENCODING=EUC_JP
159
Setting the database encoding this way typically only makes sense if
160
the locale is C; otherwise the encoding is chosen automatically from
161
the locale, and specifying an encoding that does not match the locale
162
will result in an error.
164
The encoding can be set for tests against a temporary or an existing
166
__________________________________________________________________
170
The regression test suite contains a few test files that are not run by
171
default, because they might be platform-dependent or take a very long
172
time to run. You can run these or other extra test files by setting the
173
variable EXTRA_TESTS. For example, to run the numeric_big test:
174
gmake check EXTRA_TESTS=numeric_big
176
To run the collation tests:
177
gmake check EXTRA_TESTS=collate.linux.utf8 LANG=en_US.utf8
179
The collate.linux.utf8 test works only on Linux/glibc platforms, and
180
only when run in a database that uses UTF-8 encoding.
181
__________________________________________________________________
185
Some properly installed and fully functional PostgreSQL installations
186
can "fail" some of these regression tests due to platform-specific
187
artifacts such as varying floating-point representation and message
188
wording. The tests are currently evaluated using a simple "diff"
189
comparison against the outputs generated on a reference system, so the
190
results are sensitive to small system differences. When a test is
191
reported as "failed", always examine the differences between expected
192
and actual results; you might find that the differences are not
193
significant. Nonetheless, we still strive to maintain accurate
194
reference files across all supported platforms, so it can be expected
197
The actual outputs of the regression tests are in files in the
198
"src/test/regress/results" directory. The test script uses "diff" to
199
compare each output file against the reference outputs stored in the
200
"src/test/regress/expected" directory. Any differences are saved for
201
your inspection in "src/test/regress/regression.diffs". (Or you can run
202
"diff" yourself, if you prefer.)
204
If for some reason a particular platform generates a "failure" for a
205
given test, but inspection of the output convinces you that the result
206
is valid, you can add a new comparison file to silence the failure
207
report in future test runs. See the Section called Variant Comparison
209
__________________________________________________________________
211
Error Message Differences
213
Some of the regression tests involve intentional invalid input values.
214
Error messages can come from either the PostgreSQL code or from the
215
host platform system routines. In the latter case, the messages can
216
vary between platforms, but should reflect similar information. These
217
differences in messages will result in a "failed" regression test that
218
can be validated by inspection.
219
__________________________________________________________________
223
If you run the tests against a server that was initialized with a
224
collation-order locale other than C, then there might be differences
225
due to sort order and subsequent failures. The regression test suite is
226
set up to handle this problem by providing alternate result files that
227
together are known to handle a large number of locales.
229
To run the tests in a different locale when using the
230
temporary-installation method, pass the appropriate locale-related
231
environment variables on the "make" command line, for example:
232
gmake check LANG=de_DE.utf8
234
(The regression test driver unsets LC_ALL, so it does not work to
235
choose the locale using that variable.) To use no locale, either unset
236
all locale-related environment variables (or set them to C) or use the
237
following special invocation:
238
gmake check NO_LOCALE=1
240
When running the tests against an existing installation, the locale
241
setup is determined by the existing installation. To change it,
242
initialize the database cluster with a different locale by passing the
243
appropriate options to "initdb".
245
In general, it is nevertheless advisable to try to run the regression
246
tests in the locale setup that is wanted for production use, as this
247
will exercise the locale- and encoding-related code portions that will
248
actually be used in production. Depending on the operating system
249
environment, you might get failures, but then you will at least know
250
what locale-specific behaviors to expect when running real
252
__________________________________________________________________
254
Date and Time Differences
256
Most of the date and time results are dependent on the time zone
257
environment. The reference files are generated for time zone PST8PDT
258
(Berkeley, California), and there will be apparent failures if the
259
tests are not run with that time zone setting. The regression test
260
driver sets environment variable PGTZ to PST8PDT, which normally
261
ensures proper results.
262
__________________________________________________________________
264
Floating-Point Differences
266
Some of the tests involve computing 64-bit floating-point numbers
267
(double precision) from table columns. Differences in results involving
268
mathematical functions of double precision columns have been observed.
269
The float8 and geometry tests are particularly prone to small
270
differences across platforms, or even with different compiler
271
optimization setting. Human eyeball comparison is needed to determine
272
the real significance of these differences which are usually 10 places
273
to the right of the decimal point.
275
Some systems display minus zero as -0, while others just show 0.
277
Some systems signal errors from pow() and exp() differently from the
278
mechanism expected by the current PostgreSQL code.
279
__________________________________________________________________
281
Row Ordering Differences
283
You might see differences in which the same rows are output in a
284
different order than what appears in the expected file. In most cases
285
this is not, strictly speaking, a bug. Most of the regression test
286
scripts are not so pedantic as to use an ORDER BY for every single
287
SELECT, and so their result row orderings are not well-defined
288
according to the SQL specification. In practice, since we are looking
289
at the same queries being executed on the same data by the same
290
software, we usually get the same result ordering on all platforms, so
291
the lack of ORDER BY is not a problem. Some queries do exhibit
292
cross-platform ordering differences, however. When testing against an
293
already-installed server, ordering differences can also be caused by
294
non-C locale settings or non-default parameter settings, such as custom
295
values of work_mem or the planner cost parameters.
297
Therefore, if you see an ordering difference, it's not something to
298
worry about, unless the query does have an ORDER BY that your result is
299
violating. However, please report it anyway, so that we can add an
300
ORDER BY to that particular query to eliminate the bogus "failure" in
303
You might wonder why we don't order all the regression test queries
304
explicitly to get rid of this issue once and for all. The reason is
305
that that would make the regression tests less useful, not more, since
306
they'd tend to exercise query plan types that produce ordered results
307
to the exclusion of those that don't.
308
__________________________________________________________________
310
Insufficient Stack Depth
312
If the errors test results in a server crash at the select
313
infinite_recurse() command, it means that the platform's limit on
314
process stack size is smaller than the max_stack_depth parameter
315
indicates. This can be fixed by running the server under a higher stack
316
size limit (4MB is recommended with the default value of
317
max_stack_depth). If you are unable to do that, an alternative is to
318
reduce the value of max_stack_depth.
319
__________________________________________________________________
323
The random test script is intended to produce random results. In rare
324
cases, this causes the random regression test to fail. Typing:
325
diff results/random.out expected/random.out
327
should produce only one or a few lines of differences. You need not
328
worry unless the random test fails repeatedly.
329
__________________________________________________________________
331
Variant Comparison Files
333
Since some of the tests inherently produce environment-dependent
334
results, we have provided ways to specify alternate "expected" result
335
files. Each regression test can have several comparison files showing
336
possible results on different platforms. There are two independent
337
mechanisms for determining which comparison file is used for each test.
339
The first mechanism allows comparison files to be selected for specific
340
platforms. There is a mapping file, "src/test/regress/resultmap", that
341
defines which comparison file to use for each platform. To eliminate
342
bogus test "failures" for a particular platform, you first choose or
343
make a variant result file, and then add a line to the "resultmap"
346
Each line in the mapping file is of the form
347
testname:output:platformpattern=comparisonfilename
349
The test name is just the name of the particular regression test
350
module. The output value indicates which output file to check. For the
351
standard regression tests, this is always out. The value corresponds to
352
the file extension of the output file. The platform pattern is a
353
pattern in the style of the Unix tool "expr" (that is, a regular
354
expression with an implicit ^ anchor at the start). It is matched
355
against the platform name as printed by "config.guess". The comparison
356
file name is the base name of the substitute result comparison file.
358
For example: some systems interpret very small floating-point values as
359
zero, rather than reporting an underflow error. This causes a few
360
differences in the "float8" regression test. Therefore, we provide a
361
variant comparison file, "float8-small-is-zero.out", which includes the
362
results to be expected on these systems. To silence the bogus "failure"
363
message on OpenBSD platforms, "resultmap" includes:
364
float8:out:i.86-.*-openbsd=float8-small-is-zero.out
366
which will trigger on any machine where the output of "config.guess"
367
matches i.86-.*-openbsd. Other lines in "resultmap" select the variant
368
comparison file for other platforms where it's appropriate.
370
The second selection mechanism for variant comparison files is much
371
more automatic: it simply uses the "best match" among several supplied
372
comparison files. The regression test driver script considers both the
373
standard comparison file for a test, testname.out, and variant files
374
named testname_digit.out (where the "digit" is any single digit 0-9).
375
If any such file is an exact match, the test is considered to pass;
376
otherwise, the one that generates the shortest diff is used to create
377
the failure report. (If "resultmap" includes an entry for the
378
particular test, then the base "testname" is the substitute name given
381
For example, for the char test, the comparison file "char.out" contains
382
results that are expected in the C and POSIX locales, while the file
383
"char_1.out" contains results sorted as they appear in many other
386
The best-match mechanism was devised to cope with locale-dependent
387
results, but it can be used in any situation where the test results
388
cannot be predicted easily from the platform name alone. A limitation
389
of this mechanism is that the test driver cannot tell which variant is
390
actually "correct" for the current environment; it will just pick the
391
variant that seems to work best. Therefore it is safest to use this
392
mechanism only for variant results that you are willing to consider
393
equally valid in all contexts.
394
__________________________________________________________________
396
Test Coverage Examination
398
The PostgreSQL source code can be compiled with coverage testing
399
instrumentation, so that it becomes possible to examine which parts of
400
the code are covered by the regression tests or any other test suite
401
that is run with the code. This is currently supported when compiling
402
with GCC and requires the "gcov" and "lcov" programs.
404
A typical workflow would look like this:
405
./configure --enable-coverage ... OTHER OPTIONS ...
407
gmake check # or other test suite
410
Then point your HTML browser to "coverage/index.html". The "gmake"
411
commands also work in subdirectories.
413
To reset the execution counts between test runs, run:
added
removed
src/test/regress/README
