1
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [
3
<!-- Begin Document Specific Declarations -->
7
<!ENTITY version "1.4.4">
10
<!ENTITY dejagnu-copyright "
12
<HOLDER>Free Software Foundation, Inc.</HOLDER>">
14
<!ENTITY dejagnu-code-copyright "
15
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
16
This file documents the GNU Testing Framework ``DejaGnu''
18
Copyright (C) 92 - 2001, 2002 Free Software
21
This text may be freely distributed under the terms of the GNU
22
General Public License.
23
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
26
<!ENTITY dejagnu-copyright "
27
Copyright 92 - 2001, 2002 Free Software
30
Permission is granted to make and distribute verbatim copies of
31
this manual provided the copyright notice and this permission notice
32
are preserved on all copies.
34
Permission is granted to copy and distribute modified versions of this
35
manual under the conditions for verbatim copying, provided also that
36
the entire resulting derived work is distributed under the terms of a
37
permission notice identical to this one.
39
Permission is granted to copy and distribute translations of this manual
40
into another language, under the above conditions for modified versions.
43
<!-- The reference material -->
44
<!entity ref SYSTEM "ref.sgml">
46
<!-- The user manual -->
47
<!entity user SYSTEM "user.sgml">
49
<!-- End Document Specific Declarations -->
56
<subtitle>The GNU Testing Framework</subtitle>
57
<date>2002 Sept 04</date>
58
<edition> &version</edition>
59
<releaseinfo> New release</releaseinfo>
62
<firstname>Rob Savoye</firstname>
64
<orgname>Free Software Foundation</orgname></affiliation>
66
<title>Rob Savoye</title>
68
His home page is at <ulink>
69
URL="http://www.welcomehome.org/rob.html">this
77
<email>rob@welcomehome.org</email>
79
<!-- &cygnus-street-address; -->
80
<copyright>&dejagnu-copyright;</copyright>
83
<!-- [FIXME: must put legal notice here] -->
85
<!-- &cygnus-legal-notice; -->
86
<!-- </legalnotice> -->
89
<revnumber>0.6.2</revnumber>
90
<date>2002-7-16</date>
91
<authorinitials>rob@welcomehome.org</authorinitials>
92
<revremark>Add new tutorial as a new chapter.</revremark>
95
<revnumber>0.6.1</revnumber>
96
<date>2001-2-16</date>
97
<authorinitials>rob@welcomehome.org</authorinitials>
98
<revremark>Add info on the new dejagnu.h file.</revremark>
101
<revnumber>0.6</revnumber>
102
<date>2001-2-16</date>
103
<authorinitials>rob@welcomehome.org</authorinitials>
104
<revremark>Updated for new release.</revremark>
107
<revnumber>0.5</revnumber>
108
<date>2000-1-24</date>
109
<authorinitials>rob@welcomehome.org</authorinitials>
110
<revremark>Initial version after conversion to DocBook.</revremark>
119
<title>Abstract</title>
121
<para>This document describes the functionality of DejaGnu, the
122
testing framework of the GNU project. DejaGnu is written in
123
<productname>Expect</productname>, which uses
124
<productname>Tcl</productname> as a command
125
language. <productname>Expect</productname> acts as a very
126
programmable shell. As with other Unix command shells, you can
127
run any program, but once the program is started, your test script
128
has programmable control over its input and output. This does not
129
just apply to the programs under test; <command>expect</command>
130
can also run any auxiliary program, such as
131
<command>diff</command> or <command>sh</command>, with full
132
control over its input and output.</para>
134
<para>DejaGnu itself is merely a framework for the creation of
135
testsuites. Testsuites are distributed with each
140
<chapter id=overview xreflabel=Overview>
141
<title>Overview</title>
143
<sect1 id=whatis xreflabel="What is &dj; ?">
144
<title>What is &dj; ?</title>
146
<para><productname>DejaGnu</productname> is a framework for
147
testing other programs. Its purpose is to provide a single
148
front end for all tests. Think of it as a custom library of
149
Tcl procedures crafted to support writing a test harness. A
150
<emphasis>Test Harness</emphasis> is the testing
151
infrastructure that is created to support a specific program
152
or tool. Each program can have multiple testsuites, all
153
supported by a single test harness. DejaGnu is written in
154
<productname>Expect</productname>, which in turn uses
155
<productname>Tcl</productname> -- Tool command
156
language. There is more information on Tcl at the <ulink
157
URL="http://www.scriptics.com">Scriptics</ulink> web site and the
158
Expect web site is at <ulink
159
URL="http://expect.nist.gov">NIST</ulink>.</para>
161
<para>Julia Menapace first coined the term ``DejaGnu'' to describe
162
an earlier testing framework at Cygnus Support she had written
163
for <command>GDB</command>. When we replaced it with the
164
Expect-based framework, it was like DejaGnu all over again.
165
More importantly, it was also named after my daughter, <ulink
166
URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink>
167
(now 13 years old as of September 2003), who was a toddler
168
during DejaGnu's beginnings.</para>
170
<para>DejaGnu offers several advantages for testing:</para>
172
<itemizedlist mark="bullet" spacing="compact">
174
<listitem><para>The flexibility and consistency of the DejaGnu
175
framework make it easy to write tests for any program, with
176
either batch oriented, or interactive programs.</para>
179
<listitem><para>DejaGnu provides a layer of abstraction which
180
allows you to write tests that are portable to any host or
181
target where a program must be tested. For instance, a test
182
for <command>GDB</command> can run from any supported host
183
system on any supported target system. DejaGnu runs tests on
184
many single board computers, whose operating software ranges
185
from a simple boot monitor to a real-time OS.</para>
188
<listitem><para>All tests have the same output format. This
189
makes it easy to integrate testing into other software
190
development processes. DejaGnu's output is designed to be
191
parsed by other filtering script and it is also human
195
<listitem><para>Using Tcl and Expect, it's easy to create wrappers
196
for existing testsuites. By incorporating existing tests under
197
DejaGnu, it's easier to have a single set of report analyse
203
<para>Running tests requires two things: the testing framework and
204
the testsuites themselves. Tests are usually written in
205
<productname>Expect</productname> using Tcl, but you can also use
206
a Tcl script to run a testsuite that is not based on
207
<productname>Expect</productname>. <productname>Expect</productname>
208
script filenames conventionally use <emphasis>.exp</emphasis> as a
209
suffix; for example, the main implementation of the DejaGnu test
210
driver is in the file
211
<productname>runtest.exp</productname>.)</para>
215
<sect1 id=new xreflabel="Release Notes">
216
<title>What's New In This Release</title>
218
<para>This release has a number of substantial changes over version
219
1.3. The most visible change is that the version of Expect and Tcl
220
included in the release are up-to-date with the current stable net
221
releases. The biggest change is years of modifications to the
222
target configuration system, used for cross testing. While this
223
greatly improved cross testing, is has made that subsystem very
224
complicated. The goal is to have this entirely rewritten using
225
<productname>iTcl</productname> by the next release. Other changes
229
<listitem><para>More built-in support for building target binaries
230
with the correct linker flags. Currently this only works with
231
<productname>GCC</productname> as the cross compiler,
232
preferably with a target supported by
233
<xref linkend=libgloss>.</para></listitem>
235
<listitem><para>Lots of little bug fixes from years of heavy
236
use at Cygnus Solutions.</para></listitem>
238
<listitem><para>DejaGnu now uses
239
<productname>Automake</productname> for Makefile
240
configuration.</para></listitem>
242
<listitem><para>Updated documentation, now in SGML
244
URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free
245
GNU DocBook tools</ulink>) format.</para></listitem>
247
<listitem><para>Windows support. There is beta level support for
248
Windows that is still a work in progress. This requires the
249
<ulink URL="http://www.cygwin.com/">Cygwin</ulink> POSIX
250
subsystem for Windows.</para></listitem>
254
<sect2 id=cygwin xreflabel="Windows Support">
255
<title>Windows Support</title>
257
<para>To use DejaGnu on Windows, you need to first install the
258
<ulink URL="http://www.cygwin.com/">Cygwin</ulink>
259
release. This works as of the B20.1 release. Cygwin is a POSIX
260
system for Windows. This covers both utility programs and a library
261
that adds POSIX system calls to Windows. Among them is pseudo tty
262
support for Windows that emulates the POSIX pty standard. The
263
latest Cygwin is always available from <ulink
264
URL="http://www.cygwin.com/">this location</ulink>. This
265
works well enough to run <emphasis>"make check"</emphasis> of
266
the GNU development tree on Windows after a native build. But the
267
nature of ptys on Windows is still evolving. Your mileage may
274
<sect1 id=designgoals xreflabel="Design Goals">
275
<title>Design Goals</title>
277
<para>DejaGnu grew out of the internal needs of Cygnus Solutions,
278
the company formerly known as Cygnus Support. Cygnus maintained
279
and enhanced a variety of free programs in many different
280
environments and we needed a testing tool that:</para>
282
<itemizedlist mark="bullet">
283
<listitem><para>was useful to developers while fixing
284
bugs;</para></listitem>
285
<listitem><para>automated running many tests during a software
286
release process;</para></listitem>
287
<listitem><para>was portable among a variety of host
288
computers;</para></listitem>
289
<listitem><para>supported cross-development
290
testing;</para></listitem>
291
<listitem><para>permitted testing interactive programs, like
292
<command>GDB</command>; and </para></listitem>
293
<listitem><para>permitted testing batch oriented programs, like
294
<command>GCC</command>.</para></listitem>
297
<para>Some of the requirements proved challenging. For example,
298
interactive programs do not lend themselves very well to automated testing.
299
But all the requirements are important: for instance, it is imperative to
300
make sure that <command>GDB</command> works as well when cross-debugging
301
as it does in a native configuration. </para>
303
<para>Probably the greatest challenge was testing in a
304
cross-development environment. Most cross-development
305
environments are customized by each developer. Even when buying
306
packaged boards from vendors there are many differences. The
307
communication interfaces vary from a serial line to Ethernet.
308
DejaGnu was designed with a modular communication setup, so that
309
each kind of communication can be added as required and supported
310
thereafter. Once a communication procedure is coded, any test can
311
use it. Currently DejaGnu can use <command>rsh</command>,
312
<command>rlogin</command>, <command>telnet</command>,
313
<command>tip</command>, <command>kermit</command> and
314
<command>mondfe</command> for remote communications.</para>
318
<sect1 id=posix xreflabel="A POSIX Conforming Test Framework">
319
<title>A POSIX conforming test framework</title>
321
<para>DejaGnu conforms to the POSIX 1003.3 standard for test
322
frameworks. Rob Savoye was a member of that committee.</para>
324
<para>The POSIX standard 1003.3 defines what a testing framework needs to
325
provide, in order to permit the creation of POSIX conformance test
326
suites. This standard is primarily oriented to running POSIX conformance
327
tests, but its requirements also support testing of features not related
328
to POSIX conformance. POSIX 1003.3 does not specify a particular testing
329
framework, but at this time there is only one other POSIX conforming test
330
framework: TET. TET was created by Unisoft for a consortium comprised of
331
X/Open, Unix International and the Open Software Foundation.</para>
333
<para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
334
An assertion is a description of behavior. For example, if a standard
335
says ``The sun shall shine'', a corresponding assertion might be ``The
336
sun is shining.'' A test based on this assertion would pass or fail
337
depending on whether it is day or night. It is important to note
338
that the standard being tested is never 1003.3; the standard being tested
339
is some other standard, for which the assertions were written.</para>
341
<para>As there is no testsuite to test testing frameworks for POSIX
342
1003.3 conformance, verifying conformance to this standard is done by
343
repeatedly reading the standard and experimenting. One of the main
344
things 1003.3 does specify is the set of allowed output messages and
345
their definitions. Four messages are supported for a required feature of
346
POSIX conforming systems and a fifth for a conditional feature. DejaGnu
347
supports the use of all five output messages. In this sense a testsuite
348
that uses exactly these messages can be considered POSIX conforming.
349
These definitions specify the output of a test
355
<listitem><para>A test has succeeded. That is, it demonstrated that
356
the assertion is true.</para></listitem>
361
<listitem><para>POSIX 1003.3 does not incorporate the notion of
362
expected failures, so <emphasis>PASS</emphasis>, instead of
363
<emphasis>XPASS</emphasis>, must also be returned for test cases
364
which were expected to fail and did not. This means that
365
<emphasis>PASS</emphasis> is in some sense more ambiguous than if
366
<emphasis>XPASS</emphasis> is also used.</para></listitem>
371
<listitem><para>A test has produced the bug it was intended to
372
capture. That is, it has demonstrated that the assertion is false.
373
The <emphasis>FAIL</emphasis> message is based on the test case only.
374
Other messages are used to indicate a failure of the framework. As
375
with <emphasis>PASS</emphasis>, POSIX tests must return
376
<emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
377
if a failure was expected.</para></listitem>
381
<term>UNRESOLVED</term>
382
<listitem><para>A test produced indeterminate results. Usually, this
383
means the test executed in an unexpected fashion; this outcome
384
requires that a human being go over results, to determine if the test
385
should have passed or failed. This message is also used for any test
386
that requires human intervention because it is beyond the abilities
387
of the testing framework. Any unresolved test should resolved to
388
<emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
389
run can be considered finished.</para>
391
<para>Note that for POSIX, each assertion must produce a test result
392
code. If the test isn't actually run, it must produce
393
<emphasis>UNRESOLVED</emphasis> rather than just leaving that test
394
out of the output. This means that you have to be careful when
395
writing tests to not carelessly use Tcl commands like
396
<emphasis>return</emphasis>---if you alter the flow of control of the
397
Tcl code you must insure that every test still produces some result
400
<para>Here are some of the ways a test may wind up
401
<emphasis>UNRESOLVED</emphasis>:</para></listitem>
406
<itemizedlist mark=bullet>
407
<listitem><para>A test's execution is
408
interrupted.</para></listitem>
410
<listitem><para>A test does not produce a clear
411
result. This is usually because there was an
412
<emphasis>ERROR</emphasis> from DejaGnu while processing
413
the test, or because there were three or more
414
<emphasis>WARNING</emphasis> messages. Any
415
<emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
416
messages can invalidate the output of the test. This
417
usually requires a human being to examine the output to
418
determine what really happened---and to improve the test
419
case.</para></listitem>
421
<listitem><para>A test depends on a previous test, which
422
fails.</para></listitem>
424
<listitem><para>The test was set up
425
incorrectly.</para></listitem>
430
<term>UNTESTED</term>
431
<listitem><para>A test was not run. This is a place-holder, used
432
when there is no real test case yet.</para></listitem>
436
<para>The only remaining output message left is intended to test
437
features that are specified by the applicable POSIX standard as
442
<term>UNSUPPORTED</term>
443
<listitem><para>There is no support for the tested case. This may
444
mean that a conditional feature of an operating system, or of a
445
compiler, is not implemented. DejaGnu also uses this message when
446
a testing environment (often a ``bare board'' target) lacks basic
447
support for compiling or running the test case. For example, a
448
test for the system subroutine <emphasis>gethostname</emphasis>
449
would never work on a target board running only a boot
450
monitor.</para></listitem>
454
<para>DejaGnu uses the same output procedures to produce these messages
455
for all testsuites and these procedures are already known to conform
456
to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3,
457
you must avoid the <emphasis>setup</emphasis>xfail} procedure as
458
described in the <emphasis>PASS</emphasis> section above and you must
459
be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
460
as described in the <emphasis>UNRESOLVED</emphasis> section
466
<!-- include the user manual -->
469
<!-- include the reference manual -->
474
<!-- Keep this comment at the end of the file
479
sgml-namecase-general:t
480
sgml-general-insert-case:lower
481
sgml-minimize-attributes:nil
482
sgml-always-quote-attributes:t
485
sgml-parent-document:nil
486
sgml-exposed-tags:nil
487
sgml-local-catalogs:nil
488
sgml-local-ecat-files:nil
added
removed
doc/overview.sgml
