1
<?xml version="1.0" encoding="UTF-8" ?>
3
<!DOCTYPE erlref SYSTEM "erlref.dtd">
8
<year>2010</year><year>2011</year>
9
<holder>Ericsson AB. All Rights Reserved.</holder>
12
The contents of this file are subject to the Erlang Public License,
13
Version 1.1, (the "License"); you may not use this file except in
14
compliance with the License. You should have received a copy of the
15
Erlang Public License along with this software. If not, it can be
16
retrieved online at http://www.erlang.org/.
18
Software distributed under the License is distributed on an "AS IS"
19
basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
20
the License for the specific language governing rights and limitations
25
<title>Common Test Hooks</title>
26
<prepared>Lukas Larsson</prepared>
27
<responsible>Lukas Larsson</responsible>
31
<date>2010-12-02</date>
33
<file>ct_hooks.sgml</file>
35
<module>ct_hooks</module>
36
<modulesummary>A callback interface on top of Common Test</modulesummary>
40
<warning><p>This feature is in alpha release right now. This means that the
41
interface may change in the future and that there may be bugs. We
42
encourage you to use this feature, but be prepared
43
that there might be bugs and that the interface might change
44
inbetween releases.</p></warning>
46
<p>The <em>Common Test Hook</em> (henceforth called CTH) framework allows
47
extensions of the default behaviour of Common Test by means of callbacks
48
before and after all test suite calls. It is meant for advanced users of
49
Common Test which want to abstract out behaviour which is common to
50
multiple test suites. </p>
52
<p>In brief, Common Test Hooks allows you to:</p>
55
<item>Manipulate the runtime config before each suite
56
configuration call</item>
57
<item>Manipulate the return of all suite configuration calls and in
58
extension the result of the test themselves.</item>
61
<p>The following sections describe the mandatory and optional CTH
62
functions Common Test will call during test execution. For more details
63
see <seealso marker="ct_hooks_chapter">Common Test Hooks</seealso> in
66
<p>For information about how to add a CTH to your suite see
67
<seealso marker="ct_hooks_chapter#installing">Installing a CTH
68
</seealso> in the User's Guide.</p>
71
<seealso marker="ct_hooks_chapter#example">Example CTH</seealso>
72
in the User's Guide for a minimal example of a CTH. </p></note>
77
<title>CALLBACK FUNCTIONS</title>
78
<p>The following functions define the callback interface
79
for a Common Test Hook.</p>
84
<name>Module:init(Id, Opts) -> State</name>
85
<fsummary>Initiates the Common Test Hook</fsummary>
87
<v>Id = reference() | term()</v>
95
<p>Always called before any other callback function.
96
Use this to initiate any common state.
97
It should return a state for this CTH.</p>
99
<p><c>Id</c> is the return value of
100
<seealso marker="#Module:id-1">id/1</seealso>, or a <c>reference</c>
102
<seealso marker="erts:erlang#make_ref-0">make_ref/0</seealso>)
103
if <seealso marker="#Module:id-1">id/1</seealso> is not implemented.
106
<p>For details about when init is called see
107
<seealso marker="ct_hooks_chapter#scope">scope</seealso>
108
in the User's Guide.</p>
114
<name>Module:pre_init_per_suite(SuiteName, Config, CTHState) ->
116
<fsummary>Called before init_per_suite</fsummary>
118
<v>SuiteName = atom()</v>
119
<v>Config = NewConfig = [{Key,Value}]</v>
120
<v>CTHState = NewCTHState = term()</v>
121
<v>Result = {Return, NewCTHState}</v>
122
<v>Return = NewConfig | SkipOrFail</v>
123
<v>SkipOrFail = {fail, Reason} | {skip, Reason}</v>
125
<v>Value = term()</v>
126
<v>Reason = term()</v>
132
<p>This function is called before
133
<seealso marker="common_test#Module:init_per_suite-1">
134
init_per_suite</seealso> if it exists.
135
It typically contains initialization/logging which needs to be done
136
before init_per_suite is called.
137
If <c>{skip,Reason}</c> or <c>{fail,Reason}</c> is returned,
138
init_per_suite and all test cases of the suite will be skipped and
139
Reason printed in the overview log of the suite.</p>
141
<p><c>SuiteName</c> is the name of the suite to be run.</p>
143
<p><c>Config</c> is the original config list of the test suite.</p>
145
<p><c>CTHState</c> is the current internal state of the CTH.</p>
147
<p><c>Return</c> is the result of the init_per_suite function.
148
If it is <c>{skip,Reason}</c> or <c>{fail,Reason}</c>
149
<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
150
</seealso> will never be called, instead the initiation is considered
151
to be skipped/failed respectively. If a <c>NewConfig</c> list
152
is returned, <seealso marker="common_test#Module:init_per_suite-1">
153
init_per_suite</seealso> will be called with that <c>NewConfig</c> list.
154
See <seealso marker="ct_hooks_chapter#pre">
155
Pre Hooks</seealso> in the User's Guide for more details.</p>
158
<p>Note that this function is only called if the CTH has been added
159
before init_per_suite is run, see
160
<seealso marker="ct_hooks_chapter#scope">CTH Scoping</seealso>
161
in the User's Guide for details.</p>
166
<name>Module:post_init_per_suite(SuiteName, Config, Return, CTHState) ->
168
<fsummary>Called after init_per_suite</fsummary>
170
<v>SuiteName = atom()</v>
171
<v>Config = [{Key,Value}]</v>
172
<v>Return = NewReturn = Config | SkipOrFail | term()</v>
173
<v>SkipOrFail = {fail, Reason} | {skip, Reason} | term()</v>
174
<v>CTHState = NewCTHState = term()</v>
175
<v>Result = {NewReturn, NewCTHState}</v>
177
<v>Value = term()</v>
178
<v>Reason = term()</v>
184
<p>This function is called after
185
<seealso marker="common_test#Module:init_per_suite-1">
186
init_per_suite</seealso> if it exists. It typically contains extra
187
checks to make sure that all the correct dependencies have
188
been started correctly.</p>
190
<p><c>Return</c> is what
191
<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
192
</seealso> returned, i.e. {fail,Reason}, {skip,Reason}, a <c>Config</c>
193
list or a term describing how
194
<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
195
</seealso> failed.</p>
197
<p><c>NewReturn</c> is the possibly modified return value of
198
<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
199
</seealso>. It is here possible to recover from a failure in
200
<seealso marker="common_test#Module:init_per_suite-1">init_per_suite
201
</seealso> by returning the <c>ConfigList</c> with the <c>tc_status</c>
202
element removed. See <seealso marker="ct_hooks_chapter#post">
203
Post Hooks</seealso> in the User's Guide for more details.</p>
205
<p><c>CTHState</c> is the current internal state of the CTH.</p>
207
<p>Note that this function is only called if the CTH has been added
208
before or in init_per_suite, see
209
<seealso marker="ct_hooks_chapter#scope">CTH Scoping</seealso>
210
in the User's Guide for details.</p>
215
<name>Module:pre_init_per_group(GroupName, Config, CTHState) ->
217
<fsummary>Called before init_per_group</fsummary>
219
<v>GroupName = atom()</v>
220
<v>Config = NewConfig = [{Key,Value}]</v>
221
<v>CTHState = NewCTHState = term()</v>
222
<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
223
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
225
<v>Value = term()</v>
226
<v>Reason = term()</v>
232
<p>This function is called before
233
<seealso marker="common_test#Module:init_per_group-2">
234
init_per_group</seealso> if it exists. It behaves the same way as
235
<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
236
pre_init_per_suite</seealso>, but for the
237
<seealso marker="common_test#Module:init_per_group-2">
238
init_per_group</seealso> instead.</p>
243
<name>Module:post_init_per_group(GroupName, Config, Return, CTHState) ->
245
<fsummary>Called after init_per_group</fsummary>
247
<v>GroupName = atom()</v>
248
<v>Config = [{Key,Value}]</v>
249
<v>Return = NewReturn = Config | SkipOrFail | term()</v>
250
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
251
<v>CTHState = NewCTHState = term()</v>
252
<v>Result = {NewReturn, NewCTHState}</v>
254
<v>Value = term()</v>
255
<v>Reason = term()</v>
261
<p>This function is called after
262
<seealso marker="common_test#Module:init_per_group-2">
263
init_per_group</seealso> if it exists. It behaves the same way as
264
<seealso marker="ct_hooks#Module:post_init_per_suite-4">
265
post_init_per_suite</seealso>, but for the
266
<seealso marker="common_test#Module:init_per_group-2">
267
init_per_group</seealso> instead.</p>
272
<name>Module:pre_init_per_testcase(TestcaseName, Config, CTHState) ->
274
<fsummary>Called before init_per_testcase</fsummary>
276
<v>TestcaseName = atom()</v>
277
<v>Config = NewConfig = [{Key,Value}]</v>
278
<v>CTHState = NewCTHState = term()</v>
279
<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
280
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
282
<v>Value = term()</v>
283
<v>Reason = term()</v>
289
<p>This function is called before
290
<seealso marker="common_test#Module:init_per_testcase-2">
291
init_per_testcase</seealso> if it exists. It behaves the same way as
292
<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
293
pre_init_per_suite</seealso>, but for the
294
<seealso marker="common_test#Module:init_per_testcase-2">
295
init_per_testcase</seealso> function instead.</p>
297
<p>Note that it is not possible to add CTH's here right now,
298
that feature might be added later,
299
but it would right now break backwards compatability.</p>
304
<name>Module:post_end_per_testcase(TestcaseName, Config, Return, CTHState)
306
<fsummary>Called after end_per_testcase</fsummary>
308
<v>TestcaseName = atom()</v>
309
<v>Config = [{Key,Value}]</v>
310
<v>Return = NewReturn = Config | SkipOrFail | term()</v>
311
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
312
<v>CTHState = NewCTHState = term()</v>
313
<v>Result = {NewReturn, NewCTHState}</v>
315
<v>Value = term()</v>
316
<v>Reason = term()</v>
322
<p>This function is called after
323
<seealso marker="common_test#Module:end_per_testcase-2">
324
end_per_testcase</seealso> if it exists. It behaves the same way as
325
<seealso marker="ct_hooks#Module:post_init_per_suite-4">
326
post_init_per_suite</seealso>, but for the
327
<seealso marker="common_test#Module:end_per_testcase-2">
328
end_per_testcase</seealso> function instead.</p>
333
<name>Module:pre_end_per_group(GroupName, Config, CTHState) ->
335
<fsummary>Called before end_per_group</fsummary>
337
<v>GroupName = atom()</v>
338
<v>Config = NewConfig = [{Key,Value}]</v>
339
<v>CTHState = NewCTHState = term()</v>
340
<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
341
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
343
<v>Value = term()</v>
344
<v>Reason = term()</v>
350
<p>This function is called before
351
<seealso marker="common_test#Module:end_per_group-2">
352
end_per_group</seealso> if it exists. It behaves the same way as
353
<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
354
pre_init_per_suite</seealso>, but for the
355
<seealso marker="common_test#Module:end_per_group-2">
356
end_per_group</seealso> function instead.</p>
361
<name>Module:post_end_per_group(GroupName, Config, Return, CTHState) ->
363
<fsummary>Called after end_per_group</fsummary>
365
<v>GroupName = atom()</v>
366
<v>Config = [{Key,Value}]</v>
367
<v>Return = NewReturn = Config | SkipOrFail | term()</v>
368
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
369
<v>CTHState = NewCTHState = term()</v>
370
<v>Result = {NewReturn, NewCTHState}</v>
372
<v>Value = term()</v>
373
<v>Reason = term()</v>
379
<p>This function is called after
380
<seealso marker="common_test#Module:end_per_group-2">
381
end_per_group</seealso> if it exists. It behaves the same way as
382
<seealso marker="ct_hooks#Module:post_init_per_suite-4">
383
post_init_per_suite</seealso>, but for the
384
<seealso marker="common_test#Module:end_per_group-2">
385
end_per_group</seealso> function instead.</p>
390
<name>Module:pre_end_per_suite(SuiteName, Config, CTHState) ->
392
<fsummary>Called before end_per_suite</fsummary>
394
<v>SuiteName = atom()</v>
395
<v>Config = NewConfig = [{Key,Value}]</v>
396
<v>CTHState = NewCTHState = term()</v>
397
<v>Result = {NewConfig | SkipOrFail, NewCTHState}</v>
398
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
400
<v>Value = term()</v>
401
<v>Reason = term()</v>
407
<p>This function is called before
408
<seealso marker="common_test#Module:end_per_suite-1">
409
end_per_suite</seealso> if it exists. It behaves the same way as
410
<seealso marker="ct_hooks#Module:pre_init_per_suite-3">
411
pre_init_per_suite</seealso>, but for the
412
<seealso marker="common_test#Module:end_per_suite-1">
413
end_per_suite</seealso> function instead.</p>
418
<name>Module:post_end_per_suite(SuiteName, Config, Return, CTHState) ->
420
<fsummary>Called after end_per_suite</fsummary>
422
<v>SuiteName = atom()</v>
423
<v>Config = [{Key,Value}]</v>
424
<v>Return = NewReturn = Config | SkipOrFail | term()</v>
425
<v>SkipOrFail = {fail,Reason} | {skip, Reason}</v>
426
<v>CTHState = NewCTHState = term()</v>
427
<v>Result = {NewReturn, NewCTHState}</v>
429
<v>Value = term()</v>
430
<v>Reason = term()</v>
436
<p>This function is called after
437
<seealso marker="common_test#Module:end_per_suite-1">
438
end_per_suite</seealso> if it exists. It behaves the same way as
439
<seealso marker="ct_hooks#Module:post_init_per_suite-4">
440
post_init_per_suite</seealso>, but for the
441
<seealso marker="common_test#Module:end_per_suite-1">
442
end_per_suite</seealso> function instead.</p>
447
<name>Module:on_tc_fail(TestcaseName, Reason, CTHState) ->
449
<fsummary>Called after the CTH scope ends</fsummary>
451
<v>TestcaseName = init_per_suite | end_per_suite |
452
init_per_group | end_per_group | atom()</v>
453
<v>Reason = term()</v>
454
<v>CTHState = NewCTHState = term()</v>
460
<p>This function is called whenever a testcase fails.
461
It is called after the post function has been called for
462
the testcase which failed. i.e.
463
if init_per_suite fails this function is called after
464
<seealso marker="#Module:post_init_per_suite-4">
465
post_init_per_suite</seealso>, and if a testcase fails it is called
466
after <seealso marker="#Module:post_end_per_testcase-4">
467
post_end_per_testcase</seealso>.</p>
469
<p>The data which comes with the Reason follows the same format as the
470
<seealso marker="event_handler_chapter#failreason">FailReason
471
</seealso> in the <seealso marker="event_handler_chapter#tc_done">tc_done</seealso> event.
472
See <seealso marker="event_handler_chapter#events">Event Handling
473
</seealso> in the User's Guide for details.</p>
478
<name>Module:on_tc_skip(TestcaseName, Reason, CTHState) ->
480
<fsummary>Called after the CTH scope ends</fsummary>
482
<v>TestcaseName = end_per_suite | init_per_group |
483
end_per_group | atom()</v>
484
<v>Reason = {tc_auto_skip | tc_user_skip, term()}</v>
485
<v>CTHState = NewCTHState = term()</v>
491
<p>This function is called whenever a testcase is skipped.
492
It is called after the post function has been called for the
493
testcase which was skipped.
494
i.e. if init_per_group is skipped this function is called after
495
<seealso marker="#Module:post_init_per_suite-4">post_init_per_group
496
</seealso>, and if a testcase is skipped it is called after
497
<seealso marker="#Module:post_end_per_testcase-4">post_end_per_testcase
500
<p>The data which comes with the Reason follows the same format as
501
<seealso marker="event_handler_chapter#tc_auto_skip">tc_auto_skip
502
</seealso> and <seealso marker="event_handler_chapter#tc_user_skip">
503
tc_user_skip</seealso> events.
504
See <seealso marker="event_handler_chapter#events">Event Handling
505
</seealso> in the User's Guide for details.</p>
510
<name>Module:terminate(CTHState)</name>
511
<fsummary>Called after the CTH scope ends</fsummary>
513
<v>CTHState = term()</v>
519
<p>This function is called at the end of a CTH's
520
<seealso marker="ct_hooks_chapter#scope">scope</seealso>.
526
<name>Module:id(Opts) -> Id</name>
527
<fsummary>Called before the init function of a CTH</fsummary>
536
<p>The <c>Id</c> is used to uniquely identify a CTH instance,
537
if two CTH's return the same <c>Id</c> the second CTH is ignored
538
and subsequent calls to the CTH will only be made to the first
539
instance. For more information see
540
<seealso marker="ct_hooks_chapter#installing">Installing a CTH
541
</seealso> in the User's Guide.
544
<p>This function should NOT have any side effects as it might
545
be called multiple times by Common Test.</p>
547
<p>If not implemented the CTH will act as if this function returned a
548
call to <c>make_ref/0</c>.</p>