28
59
- TCP/9001 for POP3
29
60
- TCP/9002 for IMAP
30
61
- TCP/9003 for SMTP
62
- TCP/9004 for SMTP IPv6
64
- TCP/9006 for RTSP IPv6
66
- TCP/9008 for GOPHER IPv6
67
- TCP/9008 for HTTPS server with TLS-SRP support
32
71
The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
33
servers on these ports to which it makes requests. For SSL tests, it runs
34
stunnel to handle encryption to the regular servers. For SSH, it runs a
35
standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform the SOCKS
36
functionality and requires a SSH client and server.
38
The base port number shown above can be changed using runtests' -b option
39
to allow running more than one instance of the test suite simultaneously
72
servers on the ports listed above to which it makes requests. For SSL tests,
73
it runs stunnel to handle encryption to the regular servers. For SSH, it
74
runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
75
the SOCKS functionality and requires a SSH client and server.
77
The base port number (8990), which all the individual port numbers are
78
indexed from, can be set explicitly using runtests.pl' -b option to allow
79
running more than one instance of the test suite simultaneously on one
80
machine, or just move the servers in case you have local services on any of
43
85
'make test'. This builds the test suite support code and invokes the
44
86
'runtests.pl' perl script to run all the tests. Edit the top variables
45
87
of that script in case you have some specific needs, or run the script
46
88
manually (after the support code has been built).
48
90
The script breaks on the first test that doesn't do OK. Use -a to prevent
49
the script from abort on the first error. Run the script with -v for more
91
the script from aborting on the first error. Run the script with -v for more
50
92
verbose output. Use -d to run the test servers with debug output enabled as
51
93
well. Specifying -k keeps all the log files generated by the test intact.
56
98
3 to 9. Any test numbers starting with ! are disabled, as are any test
57
99
numbers found in the file data/DISABLED (one per line).
59
Shell startup scripts:
101
When -s is not present, each successful test will display on one line the
102
test number and description and on the next line a set of flags, the test
103
result, current test sequence, total number of tests to be run and an
104
estimated amount of time to complete the test run. The flags consist of
105
these letters describing what is checked in this test:
116
1.5 Shell startup scripts
60
118
Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
61
119
influenced by the output of system wide or user specific shell startup
62
120
scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
71
129
output of a shell startup script. Locate, cleanup or adjust the shell
75
134
The test script will check that all allocated memory is freed properly IF
76
135
curl has been built with the CURLDEBUG define set. The script will
77
automatically detect if that is the case, and it will use the ../memanalyze
78
script to analyze the memory debugging output.
80
The -t option will enable torture testing mode, which runs each test
81
many times but causes a different memory allocation to fail on each
82
successive run. This tests the out of memory error handling code to
83
ensure that memory leaks do not occur even in those situations.
136
automatically detect if that is the case, and it will use the
137
'memanalyze.pl' script to analyze the memory debugging output.
139
Also, if you run tests on a machine where valgrind is found, the script will
140
use valgrind to run the test with (unless you use -n) to further verify
143
runtests.pl's -t option will enable torture testing mode, which runs each
144
test many times and makes each different memory allocation fail on each
145
successive run. This tests the out of memory error handling code to ensure
146
that memory leaks do not occur even in those situations.
86
150
If a test case fails, you can conveniently get the script to invoke the
87
151
debugger (gdb) for you with the server running and the exact same command
88
152
line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
89
153
then just type 'run' in the debugger to perform the command through the
92
If a test case causes a core dump, analyze it by running gdb like:
94
# gdb ../curl/src core
96
... and get a stack trace with the gdb command:
101
All logs are generated in the logs/ subdirectory (it is emptied first
102
in the runtests.pl script). Use runtests.pl -k to keep the temporary files
158
All logs are generated in the logs/ subdirectory (it is emptied first in the
159
runtests.pl script). Use runtests.pl -k to force it to keep the temporary
160
files after the test run since successful runs will clean it up otherwise.
106
164
All test cases are put in the data/ subdirectory. Each test is stored in the
107
165
file named according to the test number.
109
167
See FILEFORMAT for the description of the test case files.
112
171
gcc provides a tool that can determine the code coverage figures for
113
172
the test suite. To use it, configure curl with
114
173
CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
125
184
The text mode tool gcov may also be used, but it doesn't handle object files
126
185
in more than one directory very well.
129
189
The runtests.pl script provides some hooks to allow curl to be tested on a
130
190
machine where perl can not be run. The test framework in this case runs on
131
191
a workstation where perl is available, while curl itself is run on a remote
132
192
system using ssh or some other remote execution method. See the comments at
133
193
the beginning of runtests.pl for details.
137
So far, we've used this system:
144
500 - 599 libcurl source code tests, not using the curl command tool
146
700 - 799 SOCKS4 (even numbers) and SOCK5 (odd numbers)
147
800 - 899 POP3, IMAP, SMTP
148
1000 - 1299 miscellaneous*
149
1300 - 1399 unit tests*
150
1400 - 1999 miscellaneous*
151
2000 - x multiple sequential protocols per test case*
153
Since 30-apr-2003, there's nothing in the system that requires us to keep
154
within these number series, and those sections marked with * actually
155
contain tests for a variety of protocols. Each test case now specifies
156
its own server requirements, independent of test number.
160
* Add tests for TELNET, LDAP, DICT...
161
* SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
162
test mechanism) doesn't support them
197
2.1 Test case numbering
204
500 - 599 libcurl source code tests, not using the curl command tool
206
700 - 799 SOCKS4 (even numbers) and SOCK5 (odd numbers)
207
800 - 899 POP3, IMAP, SMTP
208
1000 - 1299 miscellaneous*
209
1300 - 1399 unit tests*
210
1400 - 1999 miscellaneous*
211
2000 - x multiple sequential protocols per test case*
213
Since 30-apr-2003, there's nothing in the system that requires us to keep
214
within these number series, and those sections marked with * actually
215
contain tests for a variety of protocols. Each test case now specifies its
216
own server requirements, independent of test number.
220
Here's a quick description on writing test cases. We basically have three
221
kinds of tests: the ones that test the curl tool, the ones that build small
222
applications and test libcurl directly and the unit tests that test
223
individual (possibly internal) functions.
227
Each test has a master file that controls all the test data. What to read,
228
what the protocol exchange should look like, what exit code to expect and
229
what command line arguments to use etc.
231
These files are tests/data/test[num] where [num] is described in section 2
232
of this document, and the XML-like file format of them is described in the
233
separate tests/FILEFORMAT document.
237
A test case that runs the curl tool and verifies that it gets the correct
238
data, it sends the correct data, it uses the correct protocol primitives
243
The libcurl tests are identical to the curl ones, except that they use a
244
specific and dedicated custom-built program to run instead of "curl". This
245
tool is built from source code placed in tests/libtest and if you want to
246
make a new libcurl test that is where you add your code.
250
Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
251
There's a tests/unit/README describing the specific set of checks and macros
252
that may be used when writing tests that verify behaviors of specific
253
individual functions.
255
The unit tests depend on curl being built with debug enabled.
261
Add tests for TELNET, LDAP, DICT...
265
SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
266
test mechanism) doesn't support them
added
removed
tests/README
