1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
5
>A POSIX conforming test framework</TITLE
8
CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+
12
HREF="book1.html"><LINK
18
HREF="x107.html"><LINK
20
TITLE="Getting DejaGnu up and running"
21
HREF="c203.html"></HEAD
32
SUMMARY="Header navigation table"
41
>DejaGnu: The GNU Testing Framework</TH
51
><<< Previous</A
65
>Next >>></A
79
>A POSIX conforming test framework</H1
81
>DejaGnu conforms to the POSIX 1003.3 standard for test
82
frameworks. Rob Savoye was a member of that committee.</P
84
>The POSIX standard 1003.3 defines what a testing framework needs to
85
provide, in order to permit the creation of POSIX conformance test
86
suites. This standard is primarily oriented to running POSIX conformance
87
tests, but its requirements also support testing of features not related
88
to POSIX conformance. POSIX 1003.3 does not specify a particular testing
89
framework, but at this time there is only one other POSIX conforming test
90
framework: TET. TET was created by Unisoft for a consortium comprised of
91
X/Open, Unix International and the Open Software Foundation.</P
93
>The POSIX documentation refers to <I
97
An assertion is a description of behavior. For example, if a standard
98
says ``The sun shall shine'', a corresponding assertion might be ``The
99
sun is shining.'' A test based on this assertion would pass or fail
100
depending on whether it is day or night. It is important to note
101
that the standard being tested is never 1003.3; the standard being tested
102
is some other standard, for which the assertions were written.</P
104
>As there is no testsuite to test testing frameworks for POSIX
105
1003.3 conformance, verifying conformance to this standard is done by
106
repeatedly reading the standard and experimenting. One of the main
107
things 1003.3 does specify is the set of allowed output messages and
108
their definitions. Four messages are supported for a required feature of
109
POSIX conforming systems and a fifth for a conditional feature. DejaGnu
110
supports the use of all five output messages. In this sense a testsuite
111
that uses exactly these messages can be considered POSIX conforming.
112
These definitions specify the output of a test
123
>A test has succeeded. That is, it demonstrated that
124
the assertion is true.</P
130
>POSIX 1003.3 does not incorporate the notion of
131
expected failures, so <I
138
>, must also be returned for test cases
139
which were expected to fail and did not. This means that
143
> is in some sense more ambiguous than if
153
>A test has produced the bug it was intended to
154
capture. That is, it has demonstrated that the assertion is false.
158
> message is based on the test case only.
159
Other messages are used to indicate a failure of the framework. As
163
>, POSIX tests must return
171
if a failure was expected.</P
177
>A test produced indeterminate results. Usually, this
178
means the test executed in an unexpected fashion; this outcome
179
requires that a human being go over results, to determine if the test
180
should have passed or failed. This message is also used for any test
181
that requires human intervention because it is beyond the abilities
182
of the testing framework. Any unresolved test should resolved to
190
run can be considered finished.</P
192
>Note that for POSIX, each assertion must produce a test result
193
code. If the test isn't actually run, it must produce
197
> rather than just leaving that test
198
out of the output. This means that you have to be careful when
199
writing tests to not carelessly use Tcl commands like
203
>---if you alter the flow of control of the
204
Tcl code you must insure that every test still produces some result
207
>Here are some of the ways a test may wind up
219
STYLE="list-style-type: disc"
221
>A test's execution is
225
STYLE="list-style-type: disc"
227
>A test does not produce a clear
228
result. This is usually because there was an
232
> from DejaGnu while processing
233
the test, or because there were three or more
245
messages can invalidate the output of the test. This
246
usually requires a human being to examine the output to
247
determine what really happened---and to improve the test
251
STYLE="list-style-type: disc"
253
>A test depends on a previous test, which
257
STYLE="list-style-type: disc"
272
>A test was not run. This is a place-holder, used
273
when there is no real test case yet.</P
278
>The only remaining output message left is intended to test
279
features that are specified by the applicable POSIX standard as
290
>There is no support for the tested case. This may
291
mean that a conditional feature of an operating system, or of a
292
compiler, is not implemented. DejaGnu also uses this message when
293
a testing environment (often a ``bare board'' target) lacks basic
294
support for compiling or running the test case. For example, a
295
test for the system subroutine <I
299
would never work on a target board running only a boot
305
>DejaGnu uses the same output procedures to produce these messages
306
for all testsuites and these procedures are already known to conform
307
to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3,
308
you must avoid the <I
315
> section above and you must
316
be careful to return <I
320
as described in the <I
331
SUMMARY="Footer navigation table"
344
><<< Previous</A
362
>Next >>></A
384
>Getting DejaGnu up and running</TD
b'\\ No newline at end of file'
added
removed
doc/html/x134.html
