1
<?xml version="1.0" encoding="latin1" ?>
2
<!DOCTYPE chapter SYSTEM "chapter.dtd">
9
<holder>Ericsson AB, All Rights Reserved</holder>
12
The contents of this file are subject to the Erlang Public License,
13
Version 1.1, (the "License"); you may not use this file except in
14
compliance with the License. You should have received a copy of the
15
Erlang Public License along with this software. If not, it can be
16
retrieved online at http://www.erlang.org/.
18
Software distributed under the License is distributed on an "AS IS"
19
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
20
the License for the specific language governing rights and limitations
23
The Initial Developer of the Original Code is Ericsson AB.
26
<title>Writing Test Suites</title>
27
<prepared>Siri Hansen</prepared>
34
<title>Support for test suite authors</title>
35
<p>The <c>test_server</c> module provides some useful functions
36
to support the test suite author. This includes:
38
<list type="bulleted">
39
<item>Starting and stopping slave or peer nodes</item>
40
<item>Capturing and checking stdout output</item>
41
<item>Retrieving and flushing process message queue</item>
42
<item>Watchdog timers</item>
43
<item>Checking that a function crashes</item>
44
<item>Checking that a function succeds at least m out of n times</item>
45
<item>Checking .app files</item>
47
<p>Please turn to the reference manual for the <c>test_server</c>
48
module for details about these functions.
53
<title>Test suites</title>
54
<p>A test suite is an ordinary Erlang module that contains test
55
cases. It's recommended that the module has a name on the form
56
*_SUITE.erl. Otherwise, the directory function will not find the
59
<p>For some of the test server support, the test server include
60
file <c>test_server.hrl</c> must be included. Never include it
61
with the full path, for portability reasons. Use the compiler
62
include directive instead.
64
<p>The special function <c>all(suite)</c> in each module is called
65
to get the test specification for that module. The function
66
typically returns a list of test cases in that module, but any
67
test specification could be returned. Please see the chapter
68
about test specifications for details about this.
73
<title>Init per test case</title>
74
<p>In each test suite module, the functions
75
<c>init_per_testcase/2</c> and <c>end_per_testcase/2</c> must be
78
<p><c>init_per_testcase</c> is called before each test case in the
79
test suite, giving a (limited) possibility for initialization.
81
<p><c>end_per_testcase/2</c> is called after each test case is
82
completed, giving a possibility to clean up.
84
<p>The first argument to these functions is the name of the test
85
case. This can be used to do individual initialization and cleanup for
88
<p>The second argument is a list of tuples called
89
<c>Config</c>. The first element in a <c>Config</c> tuple
90
should be an atom - a key value to be used for searching.
91
<c>init_per_testcase/2</c> may modify the <c>Config</c>
92
parameter or just return it as is. Whatever is retuned by
93
<c>init_per_testcase/2</c> is given as <c>Config</c> parameter to
96
<p>The return value of <c>end_per_testcase/2</c> is ignored by the
102
<title>Test cases</title>
103
<p>The smallest unit that the test server is concerned with is a
104
test case. Each test case can in turn test many things, for
105
example make several calls to the same interface function with
106
different parameters.
108
<p>It is possible to put many or few tests into each test
109
case. How many things each test case tests is up to the author,
110
but here are some things to keep in mind.
112
<p>Very small test cases often leads to more code, since
113
initialization has to be duplicated. Larger code, especially with
114
a lot of duplication, increases maintenance and reduces
117
<p>Larger test cases make it harder to tell what went wrong if it
118
fails, and force us to skip larger portions of test code if a
119
specific part fails. These effects are accentuated when running on
120
multiple platforms because test cases often have to be skipped.
122
<p>A test case generally consists of three parts, the
123
documentation part, the specification part and the execution
124
part. These are implemented as three clauses of the same function.
126
<p>The documentation clause matches the argument '<c>doc</c>' and
127
returns a list for strings describing what the test case tests.
129
<p>The specification clause matches the argument '<c>suite</c>'
130
and returns the test specification for this particular test
131
case. If the test specification is an empty list, this indicates
132
that the test case is a leaf test case, i.e. one to be executed.
134
<p><em>Note that the specification clause of a test case is executed on the test server controller host. This means that if target is remote, the specification clause is probably executed on a different platform than the one tested.</em></p>
135
<p>The execution clause implements the actual test case. It takes
136
one argument, <c>Config</c>, which contain configuration
137
information like <c>data_dir</c> and <c>priv_dir</c>. See <seealso marker="#data_priv_dir">Data and Private Directories</seealso> for
138
more information about these.
140
<p>The <c>Config</c> variable can also contain the
141
<c>nodenames</c> key, if requested by the <c>require_nodenames</c>
142
command in the test suite specification file. All <c>Config</c>
143
items should be extracted using the <c>?config</c> macro. This is
144
to ensure future compability if the <c>Config</c> format
145
changes. See the reference manual for <c>test_server</c> for
146
details about this macro.
148
<p>If the execution clause crashes or exits, it is considered a
149
failure. If it returns <c>{skip,Reason}</c>, the test case is
150
considered skipped. If it returns <c>{comment,String}</c>,
151
the string will be added in the 'Comment' field on the HTML
152
result page. If the execution clause returns anything else, it is
153
considered a success, unless it is <c>{'EXIT',Reason}</c> or
154
<c>{'EXIT',Pid,Reason}</c> which can't be distinguished from a
155
crash, and thus will be considered a failure.
160
<marker id="data_priv_dir"></marker>
161
<title>Data and Private Directories</title>
162
<p>The data directory (<c>data_dir</c>) is the directory where the test
163
module has its own files needed for the testing. A compiler test
164
case may have source files to feed into the compiler, a release
165
upgrade test case may have some old and new release of
166
something. A graphics test case may have some icons and a test
167
case doing a lot of math with bignums might store the correct
168
answers there. The name of the <c>data_dir</c> is the the name of
169
the test suite and then "_data". For example,
170
<c>"some_path/foo_SUITE.beam"</c> has the data directory
171
<c>"some_path/foo_SUITE_data/"</c>.
173
<p>The <c>priv_dir</c> is the test suite's private directory. This
174
directory should be used when a test case needs to write to
175
files. The name of the private directory is generated by the test
176
server, which also creates the directory.
178
<p><em>Warning:</em> Do not depend on current directory to be
179
writable, or to point to anything in particular. All scratch files
180
are to be written in the <c>priv_dir</c>, and all data files found
181
in <c>data_dir</c>. If the current directory has to be something
182
specific, it must be set with <c>file:set_cwd/1</c>.
187
<title>Execution environment</title>
188
<p>Each time a test case is about to be executed, a new process is
189
created with <c>spawn_link</c>. This is so that the test case will
190
have no dependencies to earlier tests, with respect to process flags,
191
process links, messages in the queue, other processes having registered
192
the process, etc. As little as possible is done to change the initial
193
context of the process (what is created by plain spawn). Here is a
196
<list type="bulleted">
197
<item>It has a link to the test server. If this link is removed,
198
the test server will not know when the test case is finished,
199
just wait infinitely.
201
<item>It often holds a few items in the process dictionary, all
202
with names starting with '<c>test_server_</c>'. This is to keep
203
track of if/where a test case fails.
205
<item>There is a top-level catch. All of the test case code is
206
catched, so that the location of a crash can be reported back to
207
the test server. If the test case process is killed by another
208
process (thus the catch code is never executed) the test server
209
is not able to tell where the test case was executing.
211
<item>It has a special group leader implemented by the test
212
server. This way the test server is able to capture the io that
213
the test case provokes. This is also used by some of the test
214
server support functions.
217
<p>There is no time limit for a test case, unless the test case
218
itself imposes such a limit, by calling
219
<c>test_server:timetrap/1</c> for example. The call can be made
220
in each test case, or in the <c>init_per_testcase/2</c>
221
function. Make sure to call the corresponding
222
<c>test_server:timetrap_cancel/1</c> function as well, e.g in the
223
<c>end_per_testcase/2</c> function, or else the test cases will
added
removed
lib/test_server/doc/src/write_test_chapter.xml
