2
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
3
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
4
<!ENTITY legal SYSTEM "legal.xml">
5
<!ENTITY appversion "1.4.5">
6
<!ENTITY version "1.4.5">
7
<!ENTITY manrevision "1.0">
8
<!ENTITY date "Feb 2004">
9
<!ENTITY app "<application>DejaGnu</application>">
10
<!ENTITY appname "DejaGnu">
11
<!ENTITY dj "DejaGnu">
12
<!-- The reference material -->
13
<!ENTITY ref SYSTEM "ref.xml">
14
<!-- The user manual -->
15
<!ENTITY user SYSTEM "user.xml">
17
<!-- Begin Document Specific Declarations -->
21
<subtitle>The GNU Testing Framework</subtitle>
22
<date>January 2006</date>
25
<firstname>Rob</firstname>
26
<surname>Savoye</surname>
29
<firstname>Ben</firstname>
30
<surname>Elliston</surname>
36
<holder>Free Software Foundation, Inc.</holder>
40
<!-- [FIXME: must put legal notice here] -->
42
<!-- </legalnotice> -->
45
<revnumber>0.6.2</revnumber>
46
<date>2002-07-16</date>
47
<authorinitials>rob</authorinitials>
48
<revremark>Add new tutorial as a new sect1.</revremark>
51
<revnumber>0.6.1</revnumber>
52
<date>2001-02-16</date>
53
<authorinitials>rob</authorinitials>
54
<revremark>Add info on the new dejagnu.h file.</revremark>
57
<revnumber>0.6</revnumber>
58
<date>2001-02-16</date>
59
<authorinitials>rob</authorinitials>
60
<revremark>Updated for new release.</revremark>
63
<revnumber>0.5</revnumber>
64
<date>2000-01-24</date>
65
<authorinitials>rob</authorinitials>
66
<revremark>Initial version after conversion to DocBook.</revremark>
73
<title>Abstract</title>
75
<para>This document describes the functionality of &dj;, the
76
testing framework of the GNU project. &dj; is written in
77
<productname>Expect</productname>, which uses
78
<productname>Tcl</productname> as a command
79
language. <productname>Expect</productname> acts as a very
80
programmable shell. As with other Unix command shells, you can
81
run any program, but once the program is started, your test script
82
has programmable control over its input and output. This does not
83
just apply to the programs under test; <command>expect</command>
84
can also run any auxiliary program, such as
85
<command>diff</command> or <command>sh</command>, with full
86
control over its input and output.</para>
88
<para>&dj; itself is merely a framework for the creation of
89
testsuites. Testsuites are distributed with each
94
<sect1 id="overview" xreflabel="Overview">
95
<title>Overview</title>
96
<sect2 id="whatis" xreflabel="What is &dj; ?">
97
<title>What is &dj; ?</title>
99
<para><productname>&dj;</productname> is a framework for
100
testing other programs. Its purpose is to provide a single
101
front end for all tests. Think of it as a custom library of
102
Tcl procedures crafted to support writing a test harness. A
103
<emphasis>Test Harness</emphasis> is the testing
104
infrastructure that is created to support a specific program
105
or tool. Each program can have multiple testsuites, all
106
supported by a single test harness. &dj; is written in
107
<productname>Expect</productname>, which in turn uses
108
<productname>Tcl</productname> -- Tool command
109
language. There is more information on Tcl at the <ulink
110
url="http://www.scriptics.com">Scriptics</ulink> web site and the
111
Expect web site is at <ulink
112
url="http://expect.nist.gov">NIST</ulink>.</para>
114
<para>Julia Menapace first coined the term ``&dj;'' to describe
115
an earlier testing framework at Cygnus Support she had written
116
for <command>GDB</command>. When we replaced it with the
117
Expect-based framework, it was like &dj; all over again.
118
More importantly, it was also named after my daughter, <ulink
119
url="http://www.welcomehome.org/deja/">Deja Snow Savoye</ulink>
120
(now 14 years old as of Feb 2004), who was a toddler
121
during &dj;'s beginnings.</para>
123
<para>&dj; offers several advantages for testing:</para>
125
<itemizedlist mark="bullet" spacing="compact">
127
<listitem><para>The flexibility and consistency of the &dj;
128
framework make it easy to write tests for any program, with
129
either batch oriented, or interactive programs.</para>
132
<listitem><para>&dj; provides a layer of abstraction which
133
allows you to write tests that are portable to any host or
134
target where a program must be tested. For instance, a test
135
for <command>GDB</command> can run from any supported host
136
system on any supported target system. &dj; runs tests on
137
many single board computers, whose operating software ranges
138
from a simple boot monitor to a real-time OS.</para>
141
<listitem><para>All tests have the same output format. This
142
makes it easy to integrate testing into other software
143
development processes. &dj;'s output is designed to be
144
parsed by other filtering script and it is also human
148
<listitem><para>Using Tcl and Expect, it's easy to create wrappers
149
for existing testsuites. By incorporating existing tests under
150
&dj;, it's easier to have a single set of report analyse
156
<para>Running tests requires two things: the testing framework and
157
the testsuites themselves. Tests are usually written in
158
<productname>Expect</productname> using Tcl, but you can also use
159
a Tcl script to run a testsuite that is not based on
160
<productname>Expect</productname>. <productname>Expect</productname>
161
script filenames conventionally use <emphasis>.exp</emphasis> as a
162
suffix; for example, the main implementation of the &dj; test
163
driver is in the file
164
<productname>runtest.exp</productname>.)</para>
168
<sect2 id="new" xreflabel="Release Notes">
169
<title>New In This Release</title>
171
<para>This release has a number of substantial changes over version
172
1.3. The most visible change is that the version of Expect and Tcl
173
included in the release are up-to-date with the current stable net
174
releases. The biggest change is years of modifications to the
175
target configuration system, used for cross testing. While this
176
greatly improved cross testing, is has made that subsystem very
177
complicated. The goal is to have this entirely rewritten using
178
<productname>iTcl</productname> by the next release. Other changes
183
<para>More built-in support for building target binaries
184
with the correct linker flags. Currently this only works with
185
<productname>GCC</productname> as the cross compiler,
186
preferably with a target supported by
187
<xref linkend="libgloss"/>.
191
<listitem><para>Lots of little bug fixes from years of heavy
192
use at Cygnus Solutions.</para></listitem>
194
<listitem><para>&dj; now uses
195
<productname>Automake</productname> for Makefile
196
configuration.</para></listitem>
198
<listitem><para>Updated documentation, now in DocBook XML.
201
<listitem><para>Windows support. There is beta level support for
202
Windows that is still a work in progress. This requires the
203
<ulink url="http://www.cygwin.com/">Cygwin</ulink> POSIX
204
subsystem for Windows.</para></listitem>
208
<sect3 id="cygwin" xreflabel="Windows Support">
209
<title>Windows Support</title>
211
<para>To use &dj; on Windows, you need to first install the
212
<ulink url="http://www.cygwin.com/">Cygwin</ulink>
213
release. This works as of the B20.1 release. Cygwin is a POSIX
214
system for Windows. This covers both utility programs and a library
215
that adds POSIX system calls to Windows. Among them is pseudo tty
216
support for Windows that emulates the POSIX pty standard. The
217
latest Cygwin is always available from <ulink
218
url="http://www.cygwin.com/">this location</ulink>. This
219
works well enough to run <emphasis>"make check"</emphasis> of
220
the GNU development tree on Windows after a native build. But the
221
nature of ptys on Windows is still evolving. Your mileage may
228
<sect2 id="designgoals" xreflabel="Design Goals">
229
<title>Design Goals</title>
231
<para>&dj; grew out of the internal needs of Cygnus Solutions,
232
the company formerly known as Cygnus Support. Cygnus maintained
233
and enhanced a variety of free programs in many different
234
environments and we needed a testing tool that:</para>
236
<itemizedlist mark="bullet">
237
<listitem><para>was useful to developers while fixing
238
bugs;</para></listitem>
239
<listitem><para>automated running many tests during a software
240
release process;</para></listitem>
241
<listitem><para>was portable among a variety of host
242
computers;</para></listitem>
243
<listitem><para>supported cross-development
244
testing;</para></listitem>
245
<listitem><para>permitted testing interactive programs, like
246
<command>GDB</command>; and </para></listitem>
247
<listitem><para>permitted testing batch oriented programs, like
248
<command>GCC</command>.</para></listitem>
251
<para>Some of the requirements proved challenging. For example,
252
interactive programs do not lend themselves very well to automated testing.
253
But all the requirements are important: for instance, it is imperative to
254
make sure that <command>GDB</command> works as well when cross-debugging
255
as it does in a native configuration. </para>
257
<para>Probably the greatest challenge was testing in a
258
cross-development environment. Most cross-development
259
environments are customized by each developer. Even when buying
260
packaged boards from vendors there are many differences. The
261
communication interfaces vary from a serial line to Ethernet.
262
&dj; was designed with a modular communication setup, so that
263
each kind of communication can be added as required and supported
264
thereafter. Once a communication procedure is coded, any test can
265
use it. Currently &dj; can use <command>rsh</command>,
266
<command>rlogin</command>, <command>telnet</command>,
267
<command>tip</command>, <command>kermit</command> and
268
<command>mondfe</command> for remote communications.</para>
272
<sect2 id="posix" xreflabel="A POSIX Conforming Test Framework">
273
<title>A POSIX conforming test framework</title>
275
<para>&dj; conforms to the POSIX 1003.3 standard for test
276
frameworks. Rob Savoye was a member of that committee.</para>
278
<para>The POSIX standard 1003.3 defines what a testing framework needs to
279
provide, in order to permit the creation of POSIX conformance test
280
suites. This standard is primarily oriented to running POSIX conformance
281
tests, but its requirements also support testing of features not related
282
to POSIX conformance. POSIX 1003.3 does not specify a particular testing
283
framework, but at this time there is only one other POSIX conforming test
284
framework: TET. TET was created by Unisoft for a consortium comprised of
285
X/Open, Unix International and the Open Software Foundation.</para>
287
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
288
An assertion is a description of behavior. For example, if a standard
289
says ``The sun shall shine'', a corresponding assertion might be ``The
290
sun is shining.'' A test based on this assertion would pass or fail
291
depending on whether it is day or night. It is important to note
292
that the standard being tested is never 1003.3; the standard being tested
293
is some other standard, for which the assertions were written.</para>
295
<para>As there is no testsuite to test testing frameworks for POSIX
296
1003.3 conformance, verifying conformance to this standard is done by
297
repeatedly reading the standard and experimenting. One of the main
298
things 1003.3 does specify is the set of allowed output messages and
299
their definitions. Four messages are supported for a required feature of
300
POSIX conforming systems and a fifth for a conditional feature. &dj;
301
supports the use of all five output messages. In this sense a testsuite
302
that uses exactly these messages can be considered POSIX conforming.
303
These definitions specify the output of a test
309
<listitem><para>A test has succeeded. That is, it demonstrated that
310
the assertion is true.</para></listitem>
315
<listitem><para>POSIX 1003.3 does not incorporate the notion of
316
expected failures, so <emphasis>PASS</emphasis>, instead of
317
<emphasis>XPASS</emphasis>, must also be returned for test cases
318
which were expected to fail and did not. This means that
319
<emphasis>PASS</emphasis> is in some sense more ambiguous than if
320
<emphasis>XPASS</emphasis> is also used.</para></listitem>
325
<listitem><para>A test has produced the bug it was intended to
326
capture. That is, it has demonstrated that the assertion is false.
327
The <emphasis>FAIL</emphasis> message is based on the test case only.
328
Other messages are used to indicate a failure of the framework. As
329
with <emphasis>PASS</emphasis>, POSIX tests must return
330
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
331
if a failure was expected.</para></listitem>
335
<term>UNRESOLVED</term>
336
<listitem><para>A test produced indeterminate results. Usually, this
337
means the test executed in an unexpected fashion; this outcome
338
requires that a human being go over results, to determine if the test
339
should have passed or failed. This message is also used for any test
340
that requires human intervention because it is beyond the abilities
341
of the testing framework. Any unresolved test should resolved to
342
<emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
343
run can be considered finished.</para>
345
<para>Note that for POSIX, each assertion must produce a test result
346
code. If the test isn't actually run, it must produce
347
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test
348
out of the output. This means that you have to be careful when
349
writing tests to not carelessly use Tcl commands like
350
<emphasis>return</emphasis>---if you alter the flow of control of the
351
Tcl code you must insure that every test still produces some result
354
<para>Here are some of the ways a test may wind up
355
<emphasis>UNRESOLVED</emphasis>:</para></listitem>
361
<listitem><para>A test's execution is
362
interrupted.</para></listitem>
364
<listitem><para>A test does not produce a clear
365
result. This is usually because there was an
366
<emphasis>ERROR</emphasis> from &dj; while processing
367
the test, or because there were three or more
368
<emphasis>WARNING</emphasis> messages. Any
369
<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
370
messages can invalidate the output of the test. This
371
usually requires a human being to examine the output to
372
determine what really happened---and to improve the test
373
case.</para></listitem>
375
<listitem><para>A test depends on a previous test, which
376
fails.</para></listitem>
378
<listitem><para>The test was set up
379
incorrectly.</para></listitem>
384
<term>UNTESTED</term>
385
<listitem><para>A test was not run. This is a place-holder, used
386
when there is no real test case yet.</para></listitem>
390
<para>The only remaining output message left is intended to test
391
features that are specified by the applicable POSIX standard as
396
<term>UNSUPPORTED</term>
397
<listitem><para>There is no support for the tested case. This may
398
mean that a conditional feature of an operating system, or of a
399
compiler, is not implemented. &dj; also uses this message when
400
a testing environment (often a ``bare board'' target) lacks basic
401
support for compiling or running the test case. For example, a
402
test for the system subroutine <emphasis>gethostname</emphasis>
403
would never work on a target board running only a boot
404
monitor.</para></listitem>
408
<para>&dj; uses the same output procedures to produce these messages
409
for all testsuites and these procedures are already known to conform
410
to POSIX 1003.3. For a &dj; testsuite to conform to POSIX 1003.3,
411
you must avoid the <emphasis>setup</emphasis>xfail} procedure as
412
described in the <emphasis>PASS</emphasis> section above and you must
413
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
414
as described in the <emphasis>UNRESOLVED</emphasis> section
419
<!-- include the user manual -->
422
<!-- include the reference manual -->
427
<!-- Keep this comment at the end of the file
432
sgml-namecase-general:t
433
sgml-general-insert-case:lower
434
sgml-minimize-attributes:nil
435
sgml-always-quote-attributes:t
438
sgml-parent-document:nil
439
sgml-exposed-tags:nil
440
sgml-local-catalogs:nil
441
sgml-local-ecat-files:nil
added
removed
doc/dejagnu.xml
