« back to all changes in this revision
Viewing changes to doc/dejagnu.texi
- browse files at revision 1
- compare with another revision
- download diff
- download tarball
- view history from revision 1
- Committer: Bazaar Package Importer
- Author(s): Daniel Jacobowitz
- Date: 2004-02-09 15:07:58 UTC
- Revision ID: james.westby@ubuntu.com-20040209150758-oaj7r5zrop60v8sb
Tags: upstream-1.4.4
Import upstream version 1.4.4
- files added:
- baseboards
- config
- doc
- doc/html
- example
- example/calc
- example/calc/testsuite
- example/calc/testsuite/calc.test
- example/calc/testsuite/config
- example/hello
- example/hello/lib
- example/hello/testsuite
- example/hello/testsuite/config
- example/hello/testsuite/helloworld.test1
- example/hello/testsuite/helloworld.test1/helloworld.test1-1
- example/hello/testsuite/helloworld.test2
- example/hello/testsuite/helloworld.test3
- example/hello/testsuite/helloworld.test3/config
- example/hello/testsuite/helloworld.test3/lib
- example/hello/testsuite/helloworld.test3/non_compliant_dir_name
- example/hello/testsuite/lib
- lib
- packaging
- packaging/deb
- packaging/depot
- packaging/pkg
- packaging/rpm
- testsuite
- testsuite/config
- testsuite/lib
- testsuite/libdejagnu
- testsuite/runtest.all
- testsuite/runtest.all/topdir
- testsuite/runtest.all/topdir/subdir1
- testsuite/runtest.all/topdir/subdir1/subsubdir1
- testsuite/runtest.all/topdir/subdir2
added
removed
doc/dejagnu.texi
1
\input texinfo @c -*- Texinfo -*-
2
@finalout
3
@setfilename dejagnu.info
4
@c
5
@c This file documents the GNU Testing Framework ``DejaGnu''
6
@c
7
@c Copyright (C) 92 - 2000, 2001 Free Software Foundation, Inc.
8
@c
9
@c This text may be freely distributed under the terms of the GNU
10
@c General Public License.
11
@c
12
13
@c FIXME---MAIN TODO LIST!
14
@c
15
@c * Revisit organization.
16
@c
17
@c * discuss Tcl/expect basics---enough to get started (see seminar notes).
18
@c Maybe this would permit abbreviating appendices.
19
20
@ifinfo
21
@format
22
START-INFO-DIR-ENTRY
23
* DejaGnu: (dejagnu). The GNU testing framework.
24
END-INFO-DIR-ENTRY
25
@end format
26
@end ifinfo
27
28
@syncodeindex ky cp
29
@syncodeindex fn cp
30
31
@setchapternewpage odd
32
@settitle DejaGnu Testing Framework
33
@titlepage
34
@title The DejaGnu Testing Framework
35
@subtitle for DejaGnu Version 1.3
36
@sp 1
37
@subtitle Jan 1996
38
@author Rob Savoye
39
@page
40
41
@tex
42
{\parskip=0pt \hfill Cygnus Support}
43
@end tex
44
45
@vskip 0pt plus 1filll
46
Copyright @copyright{} 92, 93, 94, 95, 96, 97, 98, 99, 2000 Free Software Foundation, Inc.
47
48
Permission is granted to make and distribute verbatim copies of
49
this manual provided the copyright notice and this permission notice
50
are preserved on all copies.
51
52
@noindent
53
Permission is granted to copy and distribute modified versions of this
54
manual under the conditions for verbatim copying, provided also that
55
the entire resulting derived work is distributed under the terms of a
56
permission notice identical to this one.
57
58
Permission is granted to copy and distribute translations of this manual
59
into another language, under the above conditions for modified versions.
60
@end titlepage
61
62
@ifinfo
63
Copyright @copyright{} 92, 93, 94, 95, 96, 97, 98, 99, 2000 Free Software Foundation, Inc.
64
65
Permission is granted to make and distribute verbatim copies of
66
this manual provided the copyright notice and this permission notice
67
are preserved on all copies.
68
69
@ignore
70
Permission is granted to process this file through TeX and print the
71
results, provided the printed document carries a copying permission
72
notice identical to this one except for the removal of this paragraph
73
(this paragraph not being relevant to the printed manual).
74
@end ignore
75
76
Permission is granted to copy and distribute modified versions of this
77
manual under the conditions for verbatim copying, provided also that
78
the entire resulting derived work is distributed under the terms of a
79
permission notice identical to this one.
80
81
Permission is granted to copy and distribute translations of this manual
82
into another language, under the above conditions for modified versions.
83
84
@node Top
85
86
@top DejaGnu
87
88
DejaGnu is a framework for running test suites on software tools.
89
90
This file describes version 1.3 of DejaGnu.
91
92
@menu
93
* Overview:: What is DejaGnu?
94
* What is New:: What is new in this release.
95
* Invoking runtest:: Using `runtest', the main test driver
96
* Customizing:: Setting `runtest' defaults
97
* Internals:: The DejaGnu implementation
98
* Tests:: How to write a test case
99
* Extending:: New tools, new targets, and new hosts
100
* Installation:: Configuring and Installing DejaGnu
101
* Index:: Index
102
@end menu
103
@end ifinfo
104
105
@iftex
106
@raggedbottom
107
@end iftex
108
109
@node Overview
110
@chapter What is DejaGnu?
111
@cindex overview
112
113
DejaGnu is a framework for testing other programs. Its purpose is to
114
provide a single front end for all tests. Beyond this, DejaGnu offers
115
several advantages for testing:
116
117
@enumerate
118
@item
119
The flexibility and consistency of the DejaGnu framework make it easy
120
to write tests for any program.
121
122
@item
123
DejaGnu provides a layer of abstraction which allows you to write tests
124
that are portable to any host or target where a program must be tested.
125
For instance, a test for GDB can run (from any Unix based host) on any
126
target architecture that DejaGnu supports. Currently DejaGnu runs tests
127
on several single board computers, whose operating software ranges from
128
just a boot monitor to a full-fledged, Unix-like realtime OS.
129
130
@item
131
All tests have the same output format. This makes it easy to integrate
132
testing into other software development processes. DejaGnu's output is
133
designed to be parsed by other filtering script, and it is also human
134
readable.
135
@end enumerate
136
137
DejaGnu is written in @code{expect}, which in turn uses @dfn{Tcl}---Tool
138
command language.
139
140
@cindex @code{expect} script names
141
@kindex .exp
142
@cindex suffix, @code{expect} scripts
143
Running tests requires two things: the testing framework, and the test
144
suites themselves. Tests are usually written in @code{expect} using
145
Tcl, but you can also use a Tcl script to run a test suite that is not
146
based on @code{expect}. (@code{expect} script filenames conventionally
147
use @samp{.exp} as a suffix; for example, the main implementation of the
148
DejaGnu test driver is in the file @file{runtest.exp}.)
149
150
151
@menu
152
* Running Tests:: A first look at running DejaGnu tests
153
* Sample Test:: What does a DejaGnu test case look like?
154
* Design Goals:: Goals behind DejaGnu
155
* Posix:: DejaGnu conforms to POSIX 1003.3
156
* Future Directions:: Where is DejaGnu going?
157
* Tcl and Expect:: Reading more about Tcl and Expect
158
@end menu
159
160
@node What is New
161
@chapter What is new in this release ?
162
@cindex What is New
163
164
This release has a number of substantial changes over version 1.2. The
165
most visible change is that the version of expect and Tcl included in
166
the release are up-to-date with the current stable net releases. Other
167
changes are:
168
169
@enumerate
170
@item
171
@c FIXME: add a link to the config section
172
The config sub-system in DejaGnu has been completely redesigned. It now
173
supports testing on remote hosts as well as remote targets.
174
175
@item
176
More builtin support for building target binaries with the correct
177
linker flags. Currently this only works with GCC, preferably with a
178
target support by @code{libgloss}.
179
180
@item
181
Lots of little bug fixes from a year of heavy use here at Cygnus
182
Support.
183
184
@item
185
DejaGnu now uses @code{autoconf} for configuration.
186
187
@item
188
New test cases for DejaGnu have been added for the new features, plus
189
the "--tool" option bug in the 1.2 testsuite has been fixed.
190
191
@item
192
The @code{--tool} option is now optional.
193
194
@item
195
@code{runtest} when searching for test drivers ignores all directories
196
named SCCS, RCS, and CVS.
197
198
@item
199
There is now a generic keyword based test harness that uses comments in
200
source code to control how each test case gets built and run.
201
202
@item
203
There is now some support for running a testsuite with multiple passes
204
and multiple targets.
205
206
@end enumerate
207
208
@node Running Tests
209
@section Running existing tests
210
@cindex existing tests, running
211
@cindex running tests
212
@cindex tests, running
213
214
@kindex make check
215
To run tests from an existing collection, first use @code{configure} as
216
usual to set up the source directory containing the tests. Then try
217
running
218
219
@example
220
make check
221
@end example
222
223
@cindex @code{check} makefile target
224
If the @code{check} target exists, it usually saves you some
225
trouble---for instance, it can set up any auxiliary programs or other
226
files needed by the tests.
227
228
@cindex auxiliary files, building
229
Once you have run @samp{make check} to build any auxiliary files, you
230
might want to call the test driver @code{runtest} directly to repeat the
231
tests. You may also have to call @code{runtest} directly for test
232
collections with no @code{check} target in the @file{Makefile}.
233
234
@c force page break to avoid losing footnote to another page
235
@page
236
@cindex @code{runtest}, most common options
237
@cindex options for @code{runtest}, common
238
Typically, you must use two command-line options: @samp{--tool}, to
239
specify which set of tests to run@footnote{@samp{--tool} selects a
240
particular suite of tests, @emph{not} the name of the executable program
241
to run. @xref{Config Values,,Configuration dependent values}, for
242
information on the variables that you can use to specify the names of
243
programs to run.}, and @samp{--srcdir}, to specify where to find test
244
directories.
245
246
For example, if the directory @file{gdb/testsuite} contains a collection
247
of DejaGnu tests for @sc{gdb}, you can run them like this:
248
249
@example
250
eg$ cd gdb/testsuite
251
eg$ runtest --tool gdb
252
@exdent @emph{Test output follows, ending with:}
253
254
=== gdb Summary ===
255
256
# of expected passes 508
257
# of expected failures 103
258
/usr/latest/bin/gdb version 4.14.4 -nx
259
@end example
260
261
You can use the option @samp{--srcdir} to point to some other directory
262
containing a collection of tests:
263
264
@smallexample
265
eg$ runtest --tool gdb --srcdir /devo/gdb/testsuite
266
@end smallexample
267
268
@cindex native configuration
269
@cindex cross configuration
270
These examples assume a @dfn{native} configuration, where the same
271
computer runs both @code{runtest} and the tests themselves. When you
272
have a @dfn{cross} configuration, the tests run on a different computer,
273
controlled by the host running @code{runtest}. In this situation, you
274
need the option @samp{--name} to specify the network address for the
275
other computer:
276
277
@smallexample
278
eg$ runtest --tool gdb --name vx9.munist.com
279
@end smallexample
280
281
If you always use the same option values, you can record them in a file
282
called @file{site.exp}, rather than typing them each time. @xref{Config
283
Values,,Setting defaults for @code{runtest} options}.
284
285
By default, @code{runtest} prints only the names of the tests it runs,
286
output from any tests that have unexpected results, and a summary
287
showing how many tests passed and how many failed. To display output
288
from all tests (whether or not they behave as expected), use the
289
@samp{--all} option. For more verbose output about processes being run,
290
communication, and so on, use @samp{--verbose}. To see even more output,
291
use multiple @samp{--verbose} options. @xref{Invoking runtest,,Using
292
@code{runtest}}, for a more detailed explanation of each @code{runtest}
293
option.
294
295
Test output goes into two files in your current directory: summary
296
output in @file{@var{tool}.sum}, and detailed output in
297
@file{@var{tool}.log}. (@var{tool} refers to the collection of tests;
298
for example, after a run with @samp{--tool gdb}, look for output files
299
@file{gdb.sum} and @file{gdb.log}.) @xref{Output Files,,The files
300
DejaGnu writes}.
301
302
@node Sample Test
303
@section What does a DejaGnu test look like?
304
305
@cindex example
306
Each DejaGnu test is an @code{expect} script; the tests vary widely in
307
complexity, depending on the nature of the tool and the feature tested.
308
309
@kindex gdb.t00/echo.exp
310
@kindex echo.exp
311
Here is a very simple @sc{gdb} test---one of the simplest tests shipped
312
with DejaGnu (extracted from @file{gdb.t00/echo.exp}):@footnote{More
313
recent @sc{gdb} tests use the @samp{gdb_test} procedure.
314
An equivalent test using that procedure is @samp{ gdb_test "echo Hello
315
world!" "Hello world!" }}
316
@c FIXME! include xref in footnote, when gdb_test documented in some manual.
317
@c @xref{}.
318
@c Extra spaces in @samp above avoid running end ' against " inside.
319
320
@cartouche
321
@smallexample
322
# send a string to the GDB stdin:
323
send "echo Hello world!\n"
324
325
# inspect the GDB stdout for the correct reply,
326
# and determine whether the test passes or fails:
327
expect @{
328
-re "Hello world.*$prompt $" @{ pass "Echo test" @}
329
-re "$prompt $" @{ fail "Echo test" @}
330
timeout @{ fail "(timeout) Echo test" @}
331
@}
332
@end smallexample
333
@end cartouche
334
335
Though brief, this example is a complete test. It illustrates some of
336
the main features of DejaGnu test scripts:
337
338
@itemize @bullet
339
@item
340
The test case does not start the tested program (@sc{gdb} in this case);
341
all test scripts for interactive tools can assume the corresponding tool
342
is running.
343
344
@item
345
Comments start with @samp{#}.
346
347
@item
348
The main commands you use to control a tested program are @code{send}
349
(to give it commands) and @code{expect} (to analyze its responses).
350
351
@item
352
The @code{expect} command uses a list of pairs; a pattern (regular
353
expression if @samp{-re} specified), followed by an action to run if the
354
pattern matches output from the program. Only the action for the
355
@emph{first} matching pattern will execute.
356
357
@item
358
Test cases use the commands @code{pass} and @code{fail} to record the
359
test outcome.
360
@end itemize
361
362
@node Design Goals
363
@section Design goals
364
@cindex design goals
365
366
DejaGnu grew out of the internal needs of Cygnus Support. Cygnus
367
maintains and enhances a variety of free programs in many different
368
environments, and we needed a testing tool that:
369
370
@itemize @bullet
371
@item
372
is useful to developers while fixing bugs;
373
374
@item
375
automates running many tests during a software release process;
376
377
@item
378
is portable among a variety of host computers;
379
380
@item
381
supports cross-development testing;
382
383
@item
384
permits testing interactive programs, like @sc{gdb}; and
385
386
@item
387
permits testing batch oriented programs, like @sc{gcc}.
388
@end itemize
389
390
Some of the requirements proved challenging. For example, interactive
391
programs do not lend themselves very well to automated testing. But all
392
the requirements are important: for instance, it is imperative to make
393
sure that @sc{gdb} works as well when cross-debugging as it does in a
394
native configuration.
395
396
Probably the greatest challenge was testing in a cross-development
397
environment (which can be a real nightmare). Most cross-development
398
environments are customized by each developer. Even when buying
399
packaged boards from vendors there are many differences. The
400
communication interfaces vary from a serial line to ethernet. DejaGnu
401
was designed with a modular communication setup, so that each kind of
402
communication can be added as required, and supported thereafter. Once
403
a communication procedure is coded, any test can use it. Currently
404
DejaGnu can use @code{rsh}, @code{rlogin}, @code{telnet}, @code{tip},
405
@code{kermit}, and @code{mondfe} for remote communications.
406
407
@cindex name ``DejaGnu''
408
@cindex DejaGnu, the name
409
@cindex Menapace, Julia
410
Julia Menapace first coined the term ``DejaGnu'' to describe an earlier
411
testing framework at Cygnus Support. When we replaced it with the
412
Expect-based framework, it was like DejaGnu all over again@dots{}
413
414
@node Posix
415
@section A POSIX conforming test framework
416
417
@cindex POSIX conformance
418
@cindex standard conformance: POSIX 1003.3
419
DejaGnu conforms to the @sc{posix} standard for test frameworks.
420
421
@cindex TET
422
@sc{posix} standard 1003.3 defines what a testing framework needs to
423
provide, in order to permit the creation of @sc{posix} conformance
424
test suites. This standard is primarily oriented to running @sc{posix}
425
conformance tests, but its requirements also support testing of features
426
not related to @sc{posix} conformance. @sc{posix} 1003.3 does not
427
specify a particular testing framework, but at this time there is only
428
one other @sc{posix} conforming test framework:
429
@sc{tet}.@footnote{@sc{tet} was created by Unisoft for a consortium
430
comprised of X/Open, Unix International, and the Open Software
431
Foundation.}
432
433
The @sc{posix} documentation refers to @dfn{assertions}. An assertion
434
is a description of behavior. For example, if a standard says ``The sun
435
shall shine'', a corresponding assertion might be ``The sun is
436
shining.'' A test based on this assertion would pass or fail depending
437
on whether it is daytime or nighttime. It is important to note that the
438
standard being tested is never 1003.3; the standard being tested is some
439
other standard, for which the assertions were written.
440
441
As there is no test suite to test @emph{testing frameworks} for
442
@sc{posix} 1003.3 conformance, verifying conformance to this standard is
443
done by repeatedly reading the standard and experimenting. One of the
444
main things 1003.3 does specify is the set of allowed output messages,
445
and their definitions. Four messages are supported for a required
446
feature of @sc{posix} conforming systems, and a fifth for a conditional
447
feature. DejaGnu supports the use of all five output messages; in this
448
sense a test suite that uses exactly these messages can be considered
449
@sc{posix} conforming. These definitions specify the output of a test
450
case:
451
452
@ftable @code
453
@cindex success, POSIX definition
454
@item PASS
455
A test has succeeded. That is, it demonstrated that the assertion is true.
456
457
@item FAIL
458
@cindex failure, POSIX definition
459
A test @emph{has} produced the bug it was intended to capture. That is,
460
it has demonstrated that the assertion is false. The @code{FAIL}
461
message is based on the test case only. Other messages are used to
462
indicate a failure of the framework.
463
464
@item UNRESOLVED
465
@cindex ambiguity, required for POSIX
466
A test produced indeterminate results. Usually, this means the test
467
executed in an unexpected fashion; this outcome requires that a human
468
being go over results, to determine if the test should have passed or
469
failed. This message is also used for any test that requires human
470
intervention because it is beyond the abilities of the testing
471
framework. Any unresolved test should resolved to @code{PASS} or
472
@code{FAIL} before a test run can be considered finished.
473
474
Note that for @sc{posix}, each assertion must produce a test result
475
code. If the test isn't actually run, it must produce @code{UNRESOLVED}
476
rather than just leaving that test out of the output. This means that
477
you have to be careful when writing tests, to not carelessly use tcl
478
statements like @code{return}---if you alter the flow of control of the
479
tcl code you must insure that every test still produces some result code.
480
481
Here are some of the ways a test may wind up @code{UNRESOLVED}:
482
483
@itemize @bullet
484
@item
485
A test's execution is interrupted.
486
487
@item
488
A test does not produce a clear result. This is usually because there
489
was an @code{ERROR} from DejaGnu while processing the test, or because there
490
were three or more @code{WARNING} messages. Any @code{WARNING} or
491
@code{ERROR} messages can invalidate the output of the test. This
492
usually requires a human being to examine the output to
493
determine what really happened---and to improve the test case.
494
495
@item
496
A test depends on a previous test, which fails.
497
498
@item
499
The test was set up incorrectly.
500
@end itemize
501
502
@item UNTESTED
503
A test was not run. This is a placeholder, used when there is no
504
real test case yet.
505
@end ftable
506
507
@noindent
508
The only remaining @sc{posix} output message left is intended to test
509
features that are specified by the applicable @sc{posix} standard as
510
conditional:
511
512
@ftable @code
513
@item UNSUPPORTED
514
There is no support for the tested case. This may mean that a
515
conditional feature of an operating system, or of a compiler, is not
516
implemented. DejaGnu also uses this message when a testing environment
517
(often a ``bare board'' target) lacks basic support for compiling or
518
running the test case. For example, a test for the system subroutine
519
@code{gethostname} would never work on a target board running only a
520
boot monitor.
521
@end ftable
522
523
DejaGnu uses the same output procedures to produce these messages for
524
all test suites, and these procedures are already known to conform to
525
@sc{posix} 1003.3. For a DejaGnu test suite to conform to @sc{posix}
526
1003.3, you must avoid the @code{setup_xfail} and @code{setup_kfail}
527
procedures (see below), and you must be careful to return
528
@code{UNRESOLVED} where appropriate, as described in the
529
@code{UNRESOLVED} section above.
530
531
Besides the @sc{posix} messages, DejaGnu provides for variations of the
532
PASS and FAIL messages that can be helpful for the tool maintainers.
533
It must be noted, however, that this feature is not @sc{posix} 1003.3
534
compliant, so its use should be avoided if compliance is necessary.
535
536
The additional messages are:
537
538
@ftable @code
539
540
@item XFAIL
541
A test is expected to fail in some environment(s) due to some bug
542
in the environment that we hope is fixed someday (but we can't do
543
nothing about as it is not a bug in the tool that we are testing).
544
The procedure @code{setup_xfail} is used to indicate that a failure
545
is expected.
546
547
@cindex XFAIL, avoiding for POSIX
548
@sc{posix} 1003.3 does not incorporate the notion of expected failures,
549
so @sc{posix} tests must return @code{FAIL} rather
550
than @code{XFAIL} even if a failure was expected.
551
552
@item KFAIL
553
A test is known to fail in some environment(s) due to a known bug
554
in the tool being tested (identified by a bug id string). This
555
exists so that, after a bug is identified and properly registered
556
in a bug tracking database (Gnats, for instance), the count of
557
failures can be kept as zero. Having zero as a baseline in all
558
platforms allow the tool developers to immediately detect regressions
559
caused by changes (which may affect some platforms and not others).
560
The connection with a bug tracking database allows for automatic
561
generation of the BUGS section of man pages or Release Notes, as
562
well as a "Bugs Fixed this Release" section (by comparing to a
563
previous release set of known failures).
564
The procedure @code{setup_kfail} is used to indicate a failure is
565
known to exist.
566
567
@cindex KFAIL, avoiding for POSIX
568
As with @code{XFAIL}, @sc{posix} tests must return @code{FAIL} rather
569
than @code{KFAIL} even if a failure was due to a known bug.
570
571
572
@item XPASS
573
A test was expected to fail with @code{XFAIL} but passed instead.
574
Whatever problem that used to exist in the environment was corrected
575
The test may also be failing to detect the failure due to some
576
environment or output changes, so this possibility must be investigated
577
as well.
578
579
@item KPASS
580
A test was expected to fail with @code{KFAIL} but passed instead.
581
Someone may have fixed the bug and failed to unmark the test.
582
As for XPASS, the test may also be failing to detect the
583
failure due to some environment or output changes, so this possibility
584
must also be checked.
585
586
@code{PASS}, instead of @code{XPASS} or @code{KPASS}, must also be
587
returned for test cases which were expected to fail and did not,
588
if @sc{posix} 1003.3 compliance is required.
589
This means that @code{PASS} is in some sense more ambiguous than if
590
@code{XPASS} and @code{KPASS} are also used.
591
592
@end ftable
593
594
See also @ref{Invoking runtest,,Using @code{runtest}}.
595
For information on how to mark tests as expected/known to fail by using
596
@code{setup_xfail} and @code{setup_kfail}, see
597
@ref{framework.exp,,Core Internal Procedures}.
598
599
600
@node Future Directions
601
@section Future directions
602
@cindex future directions
603
604
In the near future, there are two parallel directions for DejaGnu
605
development. The first is to add support for more hosts and targets.
606
607
The second would permit testing programs with a more complex interface,
608
whether text based or GUI based. Two components already exist: a Tcl
609
based X window toolkit, and a terminal package for @code{expect}. Both
610
of these could be merged into DejaGnu in a way that permits testing
611
programs that run in each environment.
612
613
Meanwhile, we hope DejaGnu enables the creation of test suites for
614
conformance to @sc{ansi} C and C++, to @sc{posix}, and to other
615
standards. We encourage you to make any test suites you create freely
616
available, under the same terms as DejaGnu itself.
617
618
@node Tcl and Expect
619
@section Tcl and Expect
620
@cindex tool command language
621
@cindex tcl
622
@cindex Ousterhout, John K.
623
Tcl was introduced in a paper by John K. Ousterhout at the 1990 Winter
624
Usenix conference, @cite{Tcl: An Embeddable Command Language}. That
625
paper is included in PostScript form in the @file{doc} subdirectory of
626
the Tcl distribution. The version of Tcl included in DejaGnu at this time is
627
Tcl 7.4p3.
628
629
@cindex @code{expect} scripting language
630
@cindex Libes, Don
631
Don Libes introduced @code{expect} in his paper @cite{expect: Curing
632
Those Uncontrollable Fits of Interaction} at the 1990 Summer Usenix
633
conference. The paper is included in PostScript form in the
634
@code{expect} distribution (as are several other papers about
635
@code{expect}). The version of expect included in DejaGnu at this time
636
is expect 5.18.0.
637
638
@node Invoking runtest
639
@chapter Using @code{runtest}
640
@cindex invoking
641
@cindex running
642
@cindex command line options
643
@cindex options
644
645
@cindex @code{runtest} description
646
@cindex DejaGnu test driver
647
@code{runtest} is the executable test driver for DejaGnu. You can
648
specify two kinds of things on the @code{runtest} command line: command
649
line options, and Tcl variables for the test scripts. The options are
650
listed alphabetically below.
651
652
@cindex exit code from @code{runtest}
653
@cindex @code{runtest} exit code
654
@code{runtest} returns an exit code of @code{1} if any test
655
has an unexpected result; otherwise (if all tests pass or fail as
656
expected) it returns @code{0} as the exit code.
657
658
@code{runtest} flags the outcome of each test as one of these cases.
659
(@xref{Posix,,A POSIX conforming test framework}, for a discussion of
660
how @sc{posix} specifies the meanings of these cases.)
661
662
@table @code
663
@item PASS
664
@kindex PASS
665
@cindex successful test
666
@cindex test, successful
667
The most desirable outcome: the test succeeded, and was expected to
668
succeed.
669
670
@item XPASS
671
@kindex XPASS
672
@cindex successful test, unexpected
673
@cindex unexpected success
674
A pleasant kind of failure: a test was expected to fail, but succeeded.
675
This may indicate progress; inspect the test case to determine whether
676
you should amend it to stop expecting failure.
677
678
@item KPASS
679
@kindex KPASS
680
@cindex successful test, unexpected
681
@cindex unexpected success
682
A pleasant kind of failure: a test was known to fail, but succeeded.
683
This may indicate progress; inspect the test case to determine whether
684
you should amend it to stop expecting failure.
685
686
@item FAIL
687
@kindex FAIL
688
@cindex failing test, unexpected
689
@cindex test, failing
690
A test failed, although it was expected to succeed. This may indicate
691
regress; inspect the test case and the failing software to locate the bug.
692
693
@item XFAIL
694
@kindex XFAIL
695
@cindex expected failure
696
@cindex failing test, expected
697
A test failed, but it was expected to fail. This result indicates no
698
change in a known environment bug. If a test fails because the operating
699
system where the test runs lacks some facility required by the test
700
(i.e. failure is due to the lack of a feature, not the existence of a bug),
701
the outcome is @code{UNSUPPORTED} instead.
702
703
@item KFAIL
704
@kindex KFAIL
705
@cindex known failure
706
@cindex failing test, known
707
A test failed, but it was known to fail. This result indicates no
708
change in a known bug. If a test fails because of a problem in the
709
environment, not in the tool being tested, that is expected to be
710
fixed one day, the outcome is @code{XFAIL} instead.
711
712
@item UNRESOLVED
713
@kindex UNRESOLVED
714
@cindex test, unresolved outcome
715
Output from a test requires manual inspection; the test suite could not
716
automatically determine the outcome. For example, your tests can report
717
this outcome is when a test does not complete as expected.
718
719
@item UNTESTED
720
@kindex UNTESTED
721
@cindex untested properties
722
A test case is not yet complete, and in particular cannot yet produce a
723
@code{PASS} or @code{FAIL}. You can also use this outcome in dummy
724
``tests'' that note explicitly the absence of a real test case
725
for a particular property.
726
727
@item UNSUPPORTED
728
@kindex UNSUPPORTED
729
@cindex unsupported test
730
@cindex test, unsupported
731
A test depends on a conditionally available feature that does not exist
732
(in the configured testing environment). For example, you can use this
733
outcome to report on a test case that does not work on a particular
734
target because its operating system support does not include a required
735
subroutine.
736
@end table
737
738
@code{runtest} may also display the following messages:
739
740
@table @code
741
@item ERROR
742
@kindex ERROR
743
@cindex problem, detected by test case
744
@cindex test case cannot run
745
Indicates a major problem (detected by the test case itself) in running
746
the test. This is usually an unrecoverable error, such as a missing file
747
or loss of communication to the target. (@sc{posix} test suites should
748
not emit this message; use @code{UNSUPPORTED}, @code{UNTESTED}, or
749
@code{UNRESOLVED} instead, as appropriate.)
750
751
@item WARNING
752
@kindex WARNING
753
@cindex test case warnings
754
Indicates a possible problem in running the test. Usually warnings
755
correspond to recoverable errors, or display an important message about
756
the following tests.
757
758
@item NOTE
759
@kindex NOTE
760
@cindex test case messages
761
An informational message about the test case.
762
@end table
763
764
This is the full set of command line options that @code{runtest}
765
recognizes. Arguments may be abbreviated to the shortest unique string.
766
767
@cindex @code{runtest} option list
768
@cindex option list, @code{runtest}
769
@smallexample
770
runtest --tool @var{tool} [ @var{testsuite}.exp @dots{} ]
771
[ @var{testsuite}.exp="testfile1 @dots{}" ]
772
[ @var{tclvar}=@var{value}@dots{} ]
773
[ --all ] [ --baud @var{baud-rate} ] [ --connect @var{type} ]
774
[ --debug ] [ --help ] [ --host @var{string} ]
775
[ --mail "@var{name} @dots{}" ] [ --name @var{string} ]
776
[ --name @var{name} ] [ --outdir @var{path} ]
777
[ --objdir @var{path} ] [ --reboot ]
778
[ --srcdir @var{path} ] [ --strace @var{n} ]
779
[ --target @var{string} --build @var{string} ]
780
[ -v | --verbose ] [ -V | --version ] [ --D@var{n} ]
781
@end smallexample
782
783
@table @code
784
@item --tool @var{tool}
785
@cindex selecting tests for a tool
786
@cindex @code{--tool} (@code{runtest} option)
787
@var{tool} specifies what set of tests to run, and what initialization
788
module to use. @var{tool} is used @emph{only} for these two purposes:
789
it is @emph{not} used to name the executable program to test.
790
Executable tool names (and paths) are recorded in @file{site.exp}
791
(@pxref{Config Values,,Configuration dependent values}), and you can
792
override them by specifying Tcl variables on the command line.
793
794
For example, including @samp{--tool gcc} on the @code{runtest} command
795
line runs tests from all test subdirectories whose names match
796
@file{gcc.*}, and uses one of the initialization modules named
797
@file{config/*-gcc.exp}. To specify the name of the compiler (perhaps
798
as an alternative path to what @code{runtest} would use by default), use
799
@samp{GCC=@var{binname}} on the @code{runtest} command line.
800
801
@item @var{testsuite}.exp @dots{}
802
@cindex selecting a range of tests
803
@cindex tests, running specifically
804
@cindex naming tests to run
805
Specify the names of testsuites to run.
806
By default, @code{runtest} runs all tests for the tool, but you can
807
restrict it to particular testsuites by giving the names of the @samp{.exp}
808
@code{expect} scripts that control them.
809
810
@var{testsuite}.exp may not include path information; use plain filenames.
811
812
@item @var{testfile}.exp="testfile1 @dots{}"
813
@cindex selecting a range of tests
814
@cindex tests, running specifically
815
@cindex naming tests to run
816
Specify a subset of tests in a suite to run.
817
For compiler or assembler tests, which often use a single @samp{.exp}
818
script covering many different source files, this option allows you to
819
further restrict the tests by listing particular source files to compile.
820
Some tools even support wildcards here. The wildcards supported depend
821
upon the tool, but typically they are @code{?}, @code{*}, and @code{[chars]}.
822
823
@item @var{tclvar}=@var{value}
824
@kindex @var{tclvar}=@var{value}
825
@cindex Tcl variables, defining for @code{runtest}
826
@cindex command line Tcl variable definition
827
@cindex @code{runtest}, variable defns on cmdline
828
You can define Tcl variables for use by your test scripts in the same
829
style used with @code{make} for environment variables. For example,
830
@samp{runtest GDB=gdb.old} defines a variable called @samp{GDB}; when
831
your scripts refer to @samp{$GDB} in this run, they use the value
832
@samp{gdb.old}.
833
834
The default Tcl variables used for most tools are defined in the main
835
DejaGnu @code{Makefile}; their values are captured in the
836
@file{site.exp} file. @xref{Config Values,,Configuration dependent
837
values}.
838
839
@item --all
840
@cindex @code{--all} (@code{runtest} option)
841
@cindex test output, displaying all
842
Display all test output. By default, @code{runtest} shows only the
843
output of tests that produce unexpected results; that is, tests with
844
status @samp{FAIL} (unexpected failure), @samp{XPASS} or @samp{KPASS}
845
(unexpected success), or @samp{ERROR} (a severe error in the test case itself).
846
Specify @samp{--all} to see output for tests with status @samp{PASS}
847
(success, as expected) @samp{XFAIL} (failure, as expected), or
848
@samp{WARNING} (minor error in the test case itself).
849
850
@item --baud @var{baud-rate}
851
@itemx -b @var{baud-rate}
852
@cindex baud rate, specifying
853
@cindex bps, specifying
854
@cindex @code{--baud} (@code{runtest} option)
855
@cindex @code{-b} (@code{runtest} option)
856
Set the default baud rate to something other than 9600. (Some serial
857
interface programs, like @code{tip}, use a separate initialization file
858
instead of this value.)
859
860
@item --connect @var{type}
861
@cindex connecting to target
862
@cindex @code{--connect} (@code{runtest} option)
863
@cindex remote testbed, connecting to
864
@cindex @code{rlogin}, remote testing via
865
@cindex @code{telnet}, remote testing via
866
@cindex @code{rsh}, remote testing via
867
@cindex @code{tip}, remote testing via
868
@cindex @code{kermit}, remote testing via
869
@cindex @code{mondfe}, remote testing via
870
@cindex remote testing via @code{rlogin}
871
@cindex remote testing via @code{telnet}
872
@cindex remote testing via @code{rsh}
873
@cindex remote testing via @code{tip}
874
@cindex remote testing via @code{kermit}
875
@cindex remote testing via @code{mondfe}
876
Connect to a target testing environment as specified by @var{type}, if
877
the target is not the computer running @code{runtest}. For example, use
878
@samp{--connect} to change the program used to connect to a ``bare
879
board'' boot monitor. The choices for @var{type} in the DejaGnu 1.0
880
distribution are @samp{rlogin}, @samp{telnet}, @samp{rsh}, @samp{tip},
881
@samp{kermit}, and @samp{mondfe}.
882
883
@noindent
884
The default for this option depends on the configuration (@pxref{Cross
885
Targets,,Remote targets supported}). The default is chosen to be the
886
most convenient communication method available, but often other
887
alternatives work as well; you may find it useful to try alternative
888
connect methods if you suspect a communication problem with your testing
889
target.
890
891
@item --debug
892
@cindex @code{--debug} (@code{runtest} option)
893
@cindex debug log for test cases
894
@cindex test cases, debug log
895
@cindex @code{dbg.log} file
896
Turns on the @code{expect} internal debugging output. Debugging output
897
is displayed as part of the @code{runtest} output, and logged to a file
898
called @file{dbg.log}. The extra debugging output does @emph{not}
899
appear on standard output, unless the verbose level is greater than 2
900
(for instance, to see debug output immediately, specify @samp{--debug -v
901
-v}). The debugging output shows all attempts at matching the test
902
output of the tool with the scripted patterns describing expected
903
output. The output generated with @samp{--strace} also goes into
904
@file{dbg.log}.
905
906
@item --help
907
@itemx -he
908
@cindex @code{--help} (@code{runtest} option)
909
@cindex help with @code{runtest}
910
@cindex @code{runtest}, listing options
911
Prints out a short summary of the @code{runtest} options, then exits
912
(even if you also specify other options).
913
914
@item --host @var{string}
915
@cindex @code{--host} (@code{runtest} option)
916
@cindex specifying the host config name
917
@cindex host config name, changing
918
@var{string} is a full configuration ``triple'' name as used by
919
@code{configure}. Use this option to override the default string
920
recorded by your configuration's choice of host. This choice does not
921
change how anything is actually configured unless --build is also
922
specified; it affects @emph{only} DejaGnu procedures that compare the
923
host string with particular values. The procedures @code{ishost},
924
@code{istarget}, @code{isnative}, @code{setup_xfail} and
925
@code{setup_kfail} are affected by @samp{--host}.
926
In this usage, @code{host} refers to the machine that the
927
tests are to be run on, which may not be the same as the @code{build}
928
machine. If @code{--build} is also specified, then @code{--host} refers
929
to the machine that the tests wil, be run on, not the machine DejaGnu is
930
run on.
931
932
@item --build @var{string}
933
@cindex @code{--build} (@code{runtest} option)
934
@cindex specifying the build config name
935
@cindex build config name, changing
936
@var{string} is a full configuration ``triple'' name as used by
937
@code{configure}. This is the type of machine DejaGnu and the tools to
938
be tested are built on. For a normal cross this is the same as the host,
939
but for a canadian cross, they are seperate.
940
941
@item --name @var{name}
942
@cindex specifying target name
943
@cindex target machine name
944
@cindex @code{--name} (@code{runtest} option)
945
@var{name} is a name for the particular testing target machine (for
946
cross testing). If the testing target has IP network support (for
947
example, @code{RPC} or @code{NFS}), this is the network name for the
948
target itself. (@var{name} is @emph{not the configuration string} you
949
specify as a target with @code{configure}; the @samp{--name} option
950
names a particular target, rather than describing a class of targets.)
951
For targets that connect in other ways, the meaning of the @var{name}
952
string depends on the connection method. @xref{Cross Targets,,Remote
953
targets supported}.
954
955
@item --name @var{string}
956
@cindex remote test machine name
957
@cindex name for remote test machine
958
Specify a network name of testing target or its host. The particular
959
names that are meaningful with @samp{--name} will depend on your site
960
configuration, and on the connection protocol: for example, @code{tip}
961
connections require names from a serial line configuration file (usually
962
called @file{/etc/remote}), while @code{telnet} connections use IP
963
hostnames.
964
965
@item --objdir @var{path}
966
@cindex @code{--objdir} (@code{runtest} option)
967
@cindex object directory
968
@cindex test programs, auxiliary
969
@cindex auxiliary test programs
970
Use @var{path} as the top directory containing any auxiliary compiled
971
test code. This defaults to @file{.}. Use this option to locate
972
pre-compiled test code. You can normally prepare any auxiliary files
973
needed with @code{make}.
974
975
@item --outdir @var{path}
976
@cindex output directory
977
@cindex @code{--outdir} (@code{runtest} option)
978
@cindex log files, where to write
979
Write output logs in directory @var{path}. The default is @samp{.}, the
980
directory where you start @code{runtest}. This option affects only the
981
summary and the detailed log files @file{@var{tool}.sum} and
982
@file{@var{tool}.log}. The DejaGnu debug log @file{dbg.log} always
983
appears (when requested) in the local directory.
984
985
@item --reboot
986
@cindex rebooting remote targets
987
@cindex @code{--reboot} (@code{runtest} option)
988
Reboot the target board when @code{runtest} initializes.
989
Usually, when running tests on a separate target board, it is safer to
990
reboot the target to be certain of its state. However, when developing
991
test scripts, rebooting takes a lot of time.
992
993
@item --srcdir @var{path}
994
@cindex source directory
995
@cindex @code{--srcdir} (@code{runtest} option)
996
Use @var{path} as the top directory for test scripts to run.
997
@code{runtest} looks in this directory for any subdirectory whose name
998
begins with the toolname (specified with @samp{--tool}). For instance,
999
with @samp{--tool gdb}, @code{runtest} uses tests in subdirectories
1000
@file{gdb.*} (with the usual shell-like filename expansion). If you do
1001
not use @samp{--srcdir}, @code{runtest} looks for test directories under
1002
the current working directory.
1003
1004
@item --strace @var{n}
1005
@cindex @code{--strace} (@code{runtest} option)
1006
@cindex tracing Tcl commands
1007
@cindex @code{expect} internal tracing
1008
Turn on internal tracing for @code{expect}, to @var{n} levels deep. By
1009
adjusting the level, you can control the extent to which your output
1010
expands multi-level Tcl statements. This allows you to ignore some
1011
levels of @code{case} or @code{if} statements. Each procedure call or
1012
control structure counts as one ``level''.
1013
1014
The output is recorded in the same file, @file{dbg.log}, used for output
1015
from @samp{--debug}.
1016
1017
@item --target @var{string}
1018
@cindex @code{--target} (@code{runtest} option)
1019
@cindex specifying the target configuration
1020
@cindex target configuration, specifying
1021
Use this option to override the default setting (running native tests).
1022
@var{string} is a full configuration ``triple''
1023
name@footnote{Configuration triples have the form
1024
@samp{@var{cpu}-@var{vendor}-@var{os}}.} as used by @code{configure}.
1025
This option changes the configuration @code{runtest} uses for the
1026
default tool names, and other setup information. @xref{Using
1027
configure,,Using @code{configure}, configure.info, Cygnus configure},
1028
for details about @code{configure} names.
1029
1030
@item --verbose
1031
@itemx -v
1032
@cindex @code{--verbose} (@code{runtest} option)
1033
@cindex @code{-v} (@code{runtest} option)
1034
@cindex turning on output
1035
@cindex output, additional
1036
Turns on more output. Repeating this option increases the amount of
1037
output displayed. Level one (@samp{-v}) is simply test output. Level
1038
two (@samp{-v -v}) shows messages on options, configuration, and process
1039
control. Verbose messages appear in the detailed (@file{*.log}) log
1040
file, but not in the summary (@file{*.sum}) log file.
1041
1042
@item --version
1043
@itemx -V
1044
@cindex @code{-V} (@code{runtest} option)
1045
@cindex @code{--version} (@code{runtest} option)
1046
@cindex version numbers
1047
Prints out the version numbers of DejaGnu, @code{expect} and Tcl, and
1048
exits without running any tests.
1049
1050
@item -D0
1051
@itemx -D1
1052
@cindex starting the tcl debugger
1053
@cindex tcl debugger
1054
@c FIXME!!! we should say a *lot* more about this debugger
1055
Start the internal Tcl debugger. The Tcl debugger supports breakpoints,
1056
single stepping, and other common debugging activities. (See @cite{A
1057
Debugger for Tcl Applications} by Don Libes. @footnote{Distributed in
1058
PostScript form with @code{expect} as the file@*
1059
@file{expect/tcl-debug.ps}.})
1060
1061
If you specify @samp{-D1}, the @code{expect} shell stops at a breakpoint
1062
as soon as DejaGnu invokes it.
1063
1064
If you specify @samp{-D0}, DejaGnu starts as usual, but you can enter
1065
the debugger by sending an interrupt (e.g. by typing @key{C-c}).
1066
@end table
1067
1068
@node Customizing
1069
@chapter Setting @code{runtest} defaults
1070
1071
@kindex site.exp
1072
@cindex variables of DejaGnu, defaults
1073
The site configuration file, @file{site.exp}, captures
1074
configuration-dependent values and propagates them to the DejaGnu test
1075
environment using Tcl variables. This ties the DejaGnu test scripts
1076
into the @code{configure} and @code{make} programs.
1077
1078
@cindex @file{site.exp}, multiple
1079
@cindex overriding @file{site.exp}
1080
DejaGnu supports more than one @file{site.exp} file. The multiple
1081
instances of @file{site.exp} are loaded in a fixed order built into
1082
DejaGnu (the more local last). The first file loaded is the optional
1083
@code{~/.dejagnurc}, then the local files, and finally the global file.
1084
1085
@enumerate
1086
@item
1087
There is am optional ``master'' @file{site.exp}, capturing configuration values
1088
that apply to DejaGnu across the board, in each configuration-specific
1089
subdirectory of the DejaGnu library directory. @code{runtest} loads
1090
these values first. @xref{Installation,,Configuring and Installing
1091
DejaGnu}. The master @file{site.exp} contains the default values for
1092
all targets and hosts supported by DejaGnu. This master file is
1093
identified by setting the environment variable @code{DEJAGNU} to the
1094
name of the file. This is also refered to as the ``global'' config file.
1095
1096
@item
1097
Any directory containing a configured test suite also has a
1098
@file{site.exp}, capturing configuration values specific to the tool
1099
under test. Since @code{runtest} loads these values last, the
1100
individual test configuration can either rely on and use, or override,
1101
any of the global values from the ``master'' @file{site.exp}.
1102
1103
You can usually generate or update the testsuite @file{site.exp} by
1104
typing @samp{make site.exp} in the test suite directory, after the test
1105
suite is configured.
1106
1107
@item
1108
You can also have a file in your home directory called
1109
@code{.dejagnurc}. This gets loaded first before the other config
1110
files. Usually this is used for personal stuff, like setting
1111
@code{all_flag} so all the output gets printed, or verbosity levels.
1112
@end enumerate
1113
1114
You can further override the default values in a user-editable section
1115
of any @file{site.exp}, or by setting variables on the @code{runtest}
1116
command line.
1117
1118
@menu
1119
* Config Values:: Variables used in the configuration file.
1120
* Master Config File:: The master configuration file.
1121
* Local Config File:: The local configuration file.
1122
* Personal Config File:: The personal configuration file.
1123
@end menu
1124
1125
@node Config Values, Master Config File, , Customizing
1126
@subsection Config Variables
1127
@cindex configuration dependent defaults
1128
@cindex setting defaults for DejaGnu variables
1129
1130
@c NOTE: default values are given via @code{"fubar"} rather than the
1131
@c more conventional @samp{fubar} to permit a consistent and clear
1132
@c notation for the empty string (@code{""}), which will work exactly as
1133
@c typed.
1134
1135
DejaGnu uses a named array in Tcl to hold all the info for each
1136
machine. In the case of a canadian cross, this means host information as
1137
well as target information. The named array is called
1138
@code{target_info}, and it has two indices. The following fields are
1139
part of the array.
1140
1141
@table @code
1142
@item name
1143
The name of the target. (mostly for error messages) This
1144
should also be the string used for this target's array.
1145
It should also be the same as the linker script so we
1146
can find them dynamically. This should be the same as the argument used
1147
for @code{push_target@{@}}.
1148
1149
@item ldflags
1150
This is the linker flags required to produce a fully linked
1151
executable. For @code{libgloss} supported targets this is usually just
1152
the name of the linker script.
1153
1154
@item config
1155
The target canonical for this target. This is used by some init files to
1156
make sure the target is supported.
1157
1158
@item cflags
1159
The flags required to produce an object file from a source file.
1160
1161
@item connect
1162
This is the connectmode for this target. This is for both IP and
1163
serial connections. Typically this is either @code{telnet},
1164
@code{rlogin}, or @code{rsh}.
1165
1166
@item target
1167
This is the hostname of the target. This is for TCP/IP based connections,
1168
and is also used for version of tip that use /etc/remote.
1169
1170
@item serial
1171
This is the serial port. This is typically /dev/tty? or com?:.
1172
1173
@item netport
1174
This is the IP port. This is commonly used for telneting to target
1175
boards that are connected to a terminal server. In that case the IP port
1176
specifies the which serial port to use.
1177
1178
@item baud
1179
This is the baud rate for a serial port connection.
1180
1181
@item x10
1182
This is the parameters for an x10 controller. These are simple devices
1183
that let us power cycle or reset a target board remotely.
1184
1185
@item fileid
1186
This is the fileid or spawn id of of the connection.
1187
1188
@item prompt
1189
a glob style pattern to recognize the prompt.
1190
1191
@item abbrev
1192
abbreviation for tool init files.
1193
1194
@item ioport
1195
This is the port for I/O on dual port systems. In this configuration,
1196
the main serial port @code{0} is usually used for stdin and stdout,
1197
which the second serial port can be used for debugging.
1198
@end table
1199
1200
The first index into the array is the same value as used in the
1201
@code{name} field. This is usually a short version of the name of the
1202
target board. For an example, here's the settings I use for my
1203
@code{Motorola's} @code{IDP} board and my @code{Motorola} 6U VME
1204
@code{MVME135-1} board. (both m68k targets)
1205
1206
@cartouche
1207
@smallexample
1208
# IDP board
1209
set target_info(idp,name) "idp"
1210
set target_info(idp,ldflags) "-Tidp.ld"
1211
set target_info(idp,config) m68k-unknown-aout
1212
set target_info(idp,cflags) ""
1213
set target_info(idp,connect) telnet
1214
set target_info(idp,target) "s7"
1215
set target_info(idp,serial) "tstty7"
1216
set target_info(idp,netport) "wharfrat:1007"
1217
set target_info(idp,baud) "9600"
1218
# MVME 135 board
1219
set target_info(idp,name) "mvme"
1220
set target_info(idp,ldflags) "-Tmvme.ld"
1221
set target_info(idp,config) m68k-unknown-aout
1222
set target_info(idp,cflags) ""
1223
set target_info(idp,connect) telnet
1224
set target_info(idp,target) "s8"
1225
set target_info(idp,serial) "tstty8"
1226
set target_info(idp,netport) "wharfrat:1008"
1227
set target_info(idp,baud) "9600"
1228
@end smallexample
1229
@end cartouche
1230
1231
DejaGnu can use this information to switch between multiple targets in
1232
one test run. This is done through the use of the @code{push_target}
1233
procedure, which is discussed elsewhere.
1234
@c FIXME: write that section and put an xref here
1235
1236
This array can also hold information for a remote host, which is used
1237
when testing a candain cross. In this case, the only thing different is
1238
the index is just @code{host}. Here's the settings I use to run tests
1239
on my NT machine while running DejaGnu on a Unix machine. (in this case
1240
a Linux box)
1241
1242
@cartouche
1243
@smallexample
1244
set target_info(host,name) "nt-host"
1245
set target_info(host,config) "386-unknown-winnt"
1246
set target_info(host,connect) "telnet"
1247
set target_info(host,target) "ripple"
1248
@end smallexample
1249
@end cartouche
1250
1251
There is more info on how to use these variables in the sections on the
1252
config files. @xref{Master Config File,,Configuration Files}.
1253
1254
@cindex option defaults
1255
@cindex @code{runtest} option defaults
1256
@cindex variables for option defaults
1257
@cindex defaults, option
1258
In the user editable second section of @file{site.exp}, you can not only
1259
override the configuration variables captured in the first section, but
1260
also specify default values for all the @code{runtest} command line
1261
options. Save for @samp{--debug}, @samp{--help}, and @samp{--version},
1262
each command line option has an associated Tcl variable. Use the Tcl
1263
@code{set} command to specify a new default value (as for the
1264
configuration variables). The following table describes the
1265
correspondence between command line options and variables you can set in
1266
@file{site.exp}. @xref{Invoking runtest,,Running the Tests}, for
1267
explanations of the command-line options.
1268
1269
@kindex all_flag
1270
@kindex baud
1271
@kindex reboot
1272
@kindex outdir
1273
@kindex objdir
1274
@kindex runtests
1275
@kindex ignoretests
1276
@kindex srcdir
1277
@kindex tracelevel
1278
@kindex targetname
1279
@kindex connectmode
1280
@kindex tool
1281
@kindex target_triplet
1282
@kindex host_triplet
1283
@kindex build_triplet
1284
@kindex verbose
1285
1286
@cindex command line option variables
1287
@cindex Tcl variables for option defaults
1288
@cindex default options, controlling
1289
@cindex options, Tcl variables for defaults
1290
1291
@ifinfo
1292
@display
1293
runtest Tcl
1294
option variable description
1295
__________ ________ ___________________________________________
1296
1297
--all all_flag display all test results if set
1298
1299
--baud baud set the default baud rate to something other
1300
than 9600.
1301
--connect connectmode @samp{rlogin}, @samp{telnet}, @samp{rsh},
1302
@samp{kermit}, @samp{tip}, or @samp{mondfe}
1303
1304
--outdir outdir directory for @file{@var{tool}.sum} and @file{@var{tool}.log}
1305
1306
--objdir objdir directory for pre-compiled binaries
1307
1308
--reboot reboot reboot the target if set to @code{"1"};
1309
do not reboot if set to @code{"0"} (the default)
1310
1311
--srcdir srcdir directory of test subdirectories
1312
1313
--strace tracelevel a number: Tcl trace depth
1314
1315
--tool tool name of tool to test; identifies init, test subdir
1316
1317
--verbose verbose verbosity level. As option, use multiple times;
1318
as variable, set a number, 0 or greater
1319
--target target_triplet The canonical configuration string for the target.
1320
--host host_triplet The canonical configuration string for the host.
1321
--build build_triplet The canonical configuration string for the
1322
build host.
1323
1324
@end display
1325
@end ifinfo
1326
1327
@tex
1328
\vbox{\halign{\hfil \tt #\quad &\quad\tt #\hfil &\hbox{\vtop{{\raggedright\parindent=0pt\parskip=5pt\hsize=2.75in\rm#\strut\par}}}\hfill\cr
1329
\cr
1330
{\it runtest}&{\it Tcl}\cr
1331
{\it option}&{\it variable}&{\it description}\cr
1332
\noalign{\hrule width\hsize}\cr
1333
--all &all\_flag &display all test results if set\cr
1334
--baud &baud &set the default baud rate to something other
1335
than 9600.\cr
1336
--connect &connectmode &@samp{rlogin}, @samp{telnet}, @samp{rsh},
1337
@samp{kermit}, @samp{tip}, or @samp{mondfe}\cr
1338
--mail &mailing\_list&address list for mailing test output\cr
1339
--name &targetname &network name of testing target or its host\cr
1340
--outdir &outdir &directory for @file{@var{tool}.sum} and @file{@var{tool}.log}\cr
1341
--objdir &objdir &directory for compiled binaries\cr
1342
--reboot &reboot &reboot the target if set to @code{"1"};
1343
do not reboot if set to @code{"0"} (the default)\cr
1344
--srcdir &srcdir &directory of test subdirectories\cr
1345
--strace &tracelevel &a number: Tcl trace depth\cr
1346
--tool &tool &name of tool to test; identifies init, test subdir\cr
1347
--verbose &verbose &verbosity level. As option, use multiple times;
1348
as variable, set a number, 0 or greater\cr
1349
--target &target\_triplet
1350
&The canonical configuration string for the target.\cr
1351
--host &host\_triplet &The canonical configuration string for the host.\cr
1352
--build &build\_triplet &The canonical configuration string for the
1353
build host.\cr
1354
}}
1355
@end tex
1356
1357
@node Master Config File, Local Config File, Config Values, Customizing
1358
@subsection Master Config File
1359
@cindex master @file{site.exp}
1360
@cindex @file{site.exp} for all of DejaGnu
1361
The master config file is where all the target specific config variables
1362
get set for a whole site get set. The idea is that for a centralized
1363
testing lab where people have to share a target between multiple
1364
developers. There are settings for both remote targets and remote hosts.
1365
Here's an example of a Master Config File (also called the Global config
1366
file) for a @emph{canadian cross}. A canadian cross is when you build
1367
and test a cross compiler on a machine other than the one it's to be
1368
hosted on.
1369
1370
Here we have the config settings for our California office. Note that
1371
all config values are site dependant. Here we have two sets of values
1372
that we use for testing m68k-aout cross compilers. As both of these
1373
target boards has a different debugging protocol, we test on both of
1374
them in sequence.
1375
1376
@cartouche
1377
@smallexample
1378
global CFLAGS
1379
global CXXFLAGS
1380
1381
case "$target_triplet" in @{
1382
@{ "native" @} @{
1383
set target_abbrev unix
1384
@}
1385
@{ "m68*-unknown-aout" @} @{
1386
set target_abbrev "rom68k"
1387
# IDP target # IDP board with rom68k monitor
1388
set target_info(idp,name) "idp"
1389
set target_info(idp,ldflags) "-Tidp.ld"
1390
set target_info(idp,config) m68k-unknown-aout
1391
set target_info(idp,cflags) ""
1392
set target_info(idp,connect) telnet
1393
set target_info(idp,target) "s7"
1394
set target_info(idp,serial) "tstty12"
1395
set target_info(idp,netport) "truckin:1007"
1396
set target_info(idp,baud) "9600"
1397
# MVME target # Motorola MVME 135 with BUG monitor
1398
set target_info(mvme,name) "mvme"
1399
set target_info(mvme,ldflags) "-Tmvme.ld"
1400
set target_info(mvme,config) m68k-unknown-aout
1401
set target_info(mvme,cflags) ""
1402
set target_info(mvme,connect) telnet
1403
set target_info(mvme,target) "s4"
1404
set target_info(mvme,serial) "tstty8"
1405
set target_info(mvme,netport) "truckin:1004"
1406
set target_info(mvme,baud) "9600"
1407
@}
1408
@}
1409
@end smallexample
1410
@end cartouche
1411
1412
In this case, we have support for several remote hosts for
1413
our m68k-aout cross compiler. Typically the remote Unix hosts run
1414
DejaGnu locally, but we also use them for debugging the testsuites when
1415
we find problems in running on remote hosts. Expect won't run on NT, so
1416
DejaGnu is run on the local build machine, and it'll connect to the NT
1417
host and run all the tests for this cross compiler on that host.
1418
1419
@smallexample
1420
@cartouche
1421
case "$host_triplet" in @{
1422
"native" @{
1423
@}
1424
"i?86-*-linux*" @{ # Linux host
1425
set target_info(host,name) "linux-host"
1426
set target_info(host,config) $host_triplet
1427
set target_info(host,connect) rlogin
1428
set target_info(host,target) chinadoll
1429
@}
1430
"i?86-*-winnt # NT host
1431
set target_info(host,name) "nt-host"
1432
set target_info(host,config) i386-unknown-winnt
1433
set target_info(host,connect) telnet
1434
set target_info(host,target) ripple
1435
@}
1436
"hppa*-hp-hpux*" @{ # HP-UX host
1437
set target_info(host,name) "hpux-host"
1438
set target_info(host,config) $host_triplet
1439
set target_info(host,connect) rlogin
1440
set target_info(host,target) slipknot
1441
@}
1442
"sparc-sun-sunos*" @{ # SunOS (sun4)
1443
set target_info(host,name) "sunos-host"
1444
set target_info(host,config) $host_triplet
1445
set target_info(host,connect) rlogin
1446
set target_info(host,target) darkstar
1447
@}
1448
@}
1449
@end cartouche
1450
@end smallexample
1451
1452
@node Local Config File, Personal Config File, Master Config File, Customizing
1453
@subsection Local Config File
1454
@cindex local @file{site.exp}
1455
@cindex @file{site.exp} for each tool
1456
It is usually more convenient to keep these ``manual overrides'' in the
1457
@file{site.exp} local to each test directory, rather than in the
1458
``master'' @file{site.exp} in the DejaGnu library.
1459
1460
All local @file{site.exp} usually files have two sections, separated by
1461
comment text. The first section is the part that is generated by
1462
@code{make}. It is essentially a collection of Tcl variable definitions
1463
based on @file{Makefile} environment variables. Since they are generated
1464
by @code{make}, they contain the values as specified by
1465
@code{configure}. (You can also customize these values by using the
1466
@samp{--site} option to @code{configure}.) In particular, this section
1467
contains the @file{Makefile} variables for host and target configuration
1468
data. Do not edit this first section; if you do, your changes are replaced
1469
next time you run @code{make}.
1470
1471
The first section starts with:
1472
1473
@cartouche
1474
@smallexample
1475
## these variables are automatically generated by make ##
1476
# Do not edit here. If you wish to override these values
1477
# add them to the last section
1478
@end smallexample
1479
@end cartouche
1480
1481
In the second section, you can override any default values (locally to
1482
DejaGnu) for all the variables. The
1483
second section can also contain your preferred defaults for all the
1484
command line options to @code{runtest}. This allows you to easily
1485
customize @code{runtest} for your preferences in each configured
1486
test-suite tree, so that you need not type options repeatedly on the
1487
command line. (The second section may also be empty, if you do not wish
1488
to override any defaults.)
1489
1490
The first section ends with this line:
1491
1492
@cartouche
1493
@smallexample
1494
## All variables above are generated by configure. Do Not Edit ##
1495
@end smallexample
1496
@end cartouche
1497
1498
You can make any changes under this line. If you wish to redefine a
1499
variable in the top section, then just put a duplicate value in this
1500
second section. Usually the values defined in this config file are
1501
related to the configuration of the test run. This is the ideal place to
1502
set the variables @code{host_triplet}, @code{build_triplet},
1503
@code{target_triplet}. All other variables are tool dependant, i.e., for
1504
testing a compiler, the value for @var{CC} might be set to a freshly
1505
built binary, as opposed to one in the user's path.
1506
1507
@node Personal Config File, , Local Config File, Customizing
1508
@subsection Personal Config File
1509
@cindex personal config @file{site.exp}
1510
@cindex @file{site.exp} for each person
1511
The personal config file is used to customize @code{runtest's} behaviour
1512
for each person. It's typically used to set the user prefered setting
1513
for verbosity, and any experimental Tcl procedures. My personal
1514
@file{~/.dejagnurc} file looks like:
1515
1516
@cartouche
1517
@smallexample
1518
set all_flag 1
1519
set RLOGIN /usr/ucb/rlogin
1520
set RSH /usr/ucb/rsh
1521
@end smallexample
1522
@end cartouche
1523
1524
Here I set @code{all_flag} so I see all the test cases that PASS along
1525
with the ones that FAIL. I also set @var{RLOGIN} and @code{RSH} to the
1526
BSD version. I have @code{kerberos} installed, and when I rlogin to a
1527
target board, it usually isn't supported. So I use the non secure
1528
versions of these programs rather than the default that's in my path.
1529
1530
@node Internals
1531
@chapter The DejaGnu Implementation
1532
@cindex operating principles
1533
@cindex internal details
1534
1535
DejaGnu is entirely written in @code{expect}, which uses Tcl as a
1536
command language. @code{expect} serves as a very programmable shell;
1537
you can run any program, as with the usual Unix command shells---but
1538
once the program is started, your @code{expect} script has fully
1539
programmable control of its input and output. This does not just apply
1540
to the programs under test; @code{expect} can also run any auxiliary
1541
program, such as @code{diff} or @code{sh}, with full control over its
1542
input and output.
1543
1544
DejaGnu itself is merely a framework for the set of test suites
1545
distributed separately for each @sc{gnu} tool. Future releases of
1546
@sc{gnu} tools will include even more tests, developed throughout the
1547
free software community.
1548
1549
@kindex runtest.exp
1550
@code{runtest} is the glue to tie together and manage the test scripts.
1551
The @code{runtest} program is actually a simple Bourne shell script that
1552
locates a copy of the @code{expect} shell and then starts the main Tcl
1553
code, @code{runtest.exp}. @code{runtest.exp} itself has these essential
1554
functions:
1555
1556
@enumerate
1557
@item
1558
Parse the command line options, load the library files, and load the
1559
default configuration files.
1560
1561
@item
1562
Locating the individual test scripts. @code{runtest.exp} locates the tests
1563
by exploiting a straightforward naming convention based on the string
1564
you specify with the @samp{--tool} option.
1565
1566
@item
1567
Providing an extended test environment, by defining additional Tcl
1568
procedures beyond those already in @code{expect}.
1569
1570
@item
1571
Locating target-dependent functions, to standardize the test environment
1572
across a wide variety of test platforms.
1573
@end enumerate
1574
1575
@menu
1576
* Names:: Conventions for using tool names
1577
* Init Module:: Initialization module
1578
* DejaGnu Builtins:: DejaGnu provides these Tcl procedures
1579
* Target Dependent:: Procedures supplied by the init module
1580
* Cross Targets:: Remote targets supported
1581
* Input Files:: The files DejaGnu depends on
1582
* Output Files:: The files DejaGnu produces
1583
@end menu
1584
1585
@node Names
1586
@section Conventions for using tool names
1587
1588
@cindex @code{--tool} and naming conventions
1589
@cindex tool names and naming conventions
1590
@cindex naming conventions
1591
DejaGnu uses @samp{$tool}, the name of the tool under test, to tie
1592
together the testing configuration in a straightforward but flexible
1593
way. If there is only one testsuite for a particular application, then
1594
@samp{$tool} is optional.
1595
1596
@samp{$tool} is @emph{not} used to invoke the tool, since sites that run
1597
multiple configurations of a particular tool often call each
1598
configuration by a different name. @code{runtest} uses the
1599
configuration-dependent variables captured in @file{site.exp} to
1600
determine how to call each tool.
1601
1602
@cindex directory names and @code{--tool}
1603
@cindex test directories, naming
1604
@code{runtest} uses tool names to find directories containing tests.
1605
@code{runtest} scans the source directory (specified with
1606
@code{--srcdir}) for all directories whose names start with the tool
1607
name. It is a common practice to put a period after the tool part of the
1608
name. For instance, directories that start with
1609
@samp{g++.} contain @sc{g++} tests. To add a new test, just put it in
1610
any directory (create an entirely new directory, if you wish) whose name
1611
follows this convention.
1612
1613
@cindex @code{exp} filename suffix
1614
@cindex test filename
1615
@cindex filename for test files
1616
A test is any file in an appropriately named subdirectory whose name
1617
ends in @samp{.exp} (the conventional way of naming @code{expect}
1618
scripts). These simple naming conventions make it as simple as possible
1619
to install new tests: all you must do is put the test in the right
1620
directory.
1621
1622
@cindex order of tests
1623
@cindex tests, running order
1624
@code{runtest} sorts the tests in each subdirectory by name (using the
1625
Tcl @code{lsort} command) and runs them in the resulting order.
1626
1627
@node Init Module
1628
@section Initialization module
1629
@cindex tool initialization
1630
@cindex setting up targets
1631
1632
@c FIXME! should this node be merged with "Target dependent"?
1633
1634
@cindex init file, purpose
1635
@cindex starting interactive tools
1636
@cindex initialization
1637
The initialization module (or ``init file'') has two purposes: to
1638
provide tool and target dependent procedures, and to start up an
1639
interactive tool to the point where it is ready to operate. The latter
1640
includes establishing communications with the target. All the tests for
1641
interactive programs assume that the tool is already running and
1642
communicating. Initialization modules for non-interactive programs may
1643
only need to supply the support functions.
1644
1645
@cindex init file name
1646
@cindex name, initialization module
1647
Each test suite directory must contain (in its @file{config}
1648
subdirectory) a separate initialization module for each target. The
1649
appropriate init file is can be named several ways. The prefered name is
1650
the @emph{os} part of the canonical configuration name with @code{.exp}
1651
as the suffix. An example would be that for an @code{m68k-coff} system,
1652
the @code{target_os} part would be @code{coff}. The next way is for
1653
system where there are short filenames, or a shortcut is desired to
1654
refer to the OS name for that target. This is uses the value of
1655
@code{$target_abbrev} rather than the @code{target_os}.
1656
1657
The final file looked for is simply @file{default.exp}. If there is only
1658
one operating system to support, then this file can be used. It's main
1659
purpose is to offer some support for new operating systems, or for
1660
unsupported cross targets. The last file looked for is
1661
@file{unknown.exp}. This is usually limited to error handling for
1662
unsupported targets. It's whole contents is typically.
1663
1664
@cartouche
1665
@smallexample
1666
perror "Sorry, there is no support for this target"
1667
exit 1
1668
@end smallexample
1669
@end cartouche
1670
1671
At the beginning of the init file, you must first determine the proper
1672
executable name of the tool to execute, since the actual name of the
1673
tool to be tested my vary from system to system. Here's an example
1674
for the @sc{gnu} C compiler.
1675
1676
@cartouche
1677
@smallexample
1678
global AR
1679
# look for the archiver ar
1680
if ![info exists AR] @{
1681
set AR [findfile $base_dir/../../binutils/ar $base_dir/../../binutils/ar [tr
1682
ansform ar]]
1683
verbose "AR defaulting to $AR" 2
1684
@}
1685
@}
1686
1687
global CFLAGS
1688
if ![info exists CFLAGS] then @{
1689
set CFLAGS ""
1690
@}
1691
@end smallexample
1692
@end cartouche
1693
1694
It is always a good idea to first check the variable, and only set it if
1695
it has not yet been defined. Often the proper value of @code{AR} is set
1696
on the command line that invokes @file{runtest}.
1697
1698
@kindex findfile
1699
The @code{findfile} procedure takes as it's first argument a file name
1700
to look for. The second argument is returned if the file is found, and
1701
the third argument is returned if the file is not found. @code{base_dir}
1702
is set internally by DejaGnu to the top level directory of the object
1703
tree.
1704
1705
@kindex transform
1706
The @code{transform} procedure takes as its argument the native name of
1707
a tool (such as @samp{gcc} for the compiler), and returns the name as
1708
configured for that tool in the current installation. (For example, a
1709
cross-compiling version of @sc{gnu} CC that generates MIPS code may be
1710
installed with a name like @code{mips-idt-ecoff-gcc}.)
1711
1712
In a test running native, writing the Tcl code for initialization is
1713
usually quite simple. For cross configurations, however, more elaborate
1714
instructions are usually needed to describe how to talk to a remote
1715
target.
1716
1717
Each initialization module defines up to four procedures with standard
1718
names and purposes. The names of these procedures begin with
1719
@samp{$tool}, the string that identifies tests for a particular tool:
1720
@code{$tool_start}, @code{$tool_load}, @code{$tool_exit}, and
1721
@code{$tool_version}. For example, the start procedure for @sc{gdb} is
1722
called @code{gdb_start}. (Since start procedures are used differently
1723
for batch and interactive tools, however, @code{runtest} itself never
1724
calls the start procedure. Init files for interactive tools are
1725
expected to end by running the start procedure.)
1726
1727
@cindex utilities, loading from init file
1728
@cindex defaults, setting in init file
1729
The initialization module is also a good place to call @code{load_lib}
1730
to get any collections of utility procedures meant for a family of test
1731
cases, and to set up default values for any additional Tcl variables
1732
needed for a specific set of tests.
1733
1734
@xref{Target Dependent,,Target dependent procedures}, for full
1735
descriptions of these procedures.
1736
1737
@node DejaGnu Builtins
1738
@section DejaGnu procedures
1739
@cindex built in procedures, DejaGnu
1740
1741
DejaGnu provides these Tcl procedures for use in test scripts.
1742
You can also use any standard @code{expect} or Tcl function. These
1743
procedures are stored in libraries, which DejaGnu loads at
1744
runtime. Here's explanation of the library procedures that get loaded at
1745
runtime. All other librarys are optional, and need to be loaded by the
1746
testsuite.
1747
1748
@menu
1749
* framework.exp:: Core Internal Procedures.
1750
* remote.exp:: Procedures for remote communication.
1751
* utils.exp:: Utility procedures.
1752
* target.exp:: Cross target procedures.
1753
* debugger.exp:: Procedures for debugging your Tcl code.
1754
@end menu
1755
1756
@node framework.exp, remote.exp, ,DejaGnu Builtins
1757
@subsection Core Internal Procedures
1758
@cindex Core Internal Procedures
1759
1760
@xref{Posix,,A POSIX conforming test framework}, for more detailed
1761
explanations of the test outcomes (@samp{FAIL}, @samp{PASS},
1762
@samp{UNTESTED}, @samp{UNRESOLVED}, @samp{UNSUPPORTED}).
1763
1764
@ftable @code
1765
@item perror "@var{string} @var{number}"
1766
@cindex test case, ERROR in
1767
@kindex ERROR
1768
Declares a severe error in the testing framework itself.
1769
@code{perror} writes in the log files a message beginning with
1770
@samp{ERROR}, appending the argument @var{string}. If the optional
1771
@var{number} is supplied, then this is used to set the internal count of
1772
errors to that value.
1773
1774
As a side effect, @code{perror} also changes the effect of the next
1775
@code{pass} or @code{fail} command: the test outcome becomes
1776
@samp{UNRESOLVED}, since an automatic @samp{PASS} or @samp{FAIL} cannot
1777
be trusted after a severe error in the test framework. If the optional
1778
numeric value is @samp{0}, then there are no further side effects to
1779
calling this function, and the following test outcome doesn't become
1780
@samp{UNRESOLVED}. This can be used for errors with no known side
1781
effects.
1782
1783
@item warning "@var{string} @var{number}"
1784
@cindex test case, WARNING in
1785
@kindex WARNING
1786
Declares detection of a minor error in the test case itself.
1787
@code{warning} writes in the log files a message beginning with
1788
@samp{WARNING}, appending the argument @var{string}. Use @code{warning}
1789
rather than @code{error} for cases (such as communication failure
1790
to be followed by a retry) where the test case can recover from the
1791
error. If the optional @var{number} is supplied, then this is used to
1792
set the internal count of warnings to that value.
1793
1794
As a side effect, @code{warning_threshold} or more calls to
1795
@code{warning} in a single test case also changes the effect of the next
1796
@code{pass} or @code{fail} command: the test outcome becomes
1797
@samp{UNRESOLVED} since an automatic @samp{PASS} or @samp{FAIL} may not
1798
be trustworthy after many warnings. If the optional numeric value is
1799
@samp{0}, then there are no further side effects to calling this
1800
function, and the following test outcome doesn't become
1801
@samp{UNRESOLVED}. This can be used for errors with no known side
1802
effects.
1803
1804
@item note "@var{string}"
1805
@cindex test case, informational messages
1806
@kindex NOTE
1807
Appends an informational message to the log file.
1808
@code{note} writes in the log files a message beginning with
1809
@samp{NOTE}, appending the argument @var{string}. Use @code{note}
1810
sparingly. @code{verbose} should be used for most such messages,
1811
but in cases where a message is needed in the log file regardless of
1812
the verbosity level use @code{note}.
1813
1814
@item pass "@var{string}"
1815
@cindex test case, declaring success
1816
Declares a test to have passed. @code{pass} writes in the
1817
log files a message beginning with @samp{PASS}
1818
(or @code{XPASS}/@code{KPASS}, if failure was expected),
1819
appending the argument @var{string}.
1820
1821
@item fail "@var{string}"
1822
@cindex test case, declaring failure
1823
Declares a test to have failed. @code{fail} writes in the
1824
log files a message beginning with @samp{FAIL} (or @code{XFAIL}, if
1825
failure was expected), appending the argument @var{string}.
1826
1827
@item unresolved "@var{string}"
1828
@cindex test case, declaring ambiguity
1829
Declares a test to have an unresolved outcome. @code{unresolved} writes
1830
in the log file a message beginning with @samp{UNRESOLVED}, appending
1831
the argument @var{string}. This usually means the test did not execute
1832
as expected, and a human being must go over results to determine if it
1833
passed or failed (and to improve the test case).
1834
1835
@item untested "@var{string}"
1836
@cindex test case, declaring no test
1837
Declares a test was not run. @code{untested} writes in the log file a
1838
message beginning with @samp{UNTESTED}, appending the argument
1839
@var{string}. For example, you might use this in a dummy test whose
1840
only role is to record that a test does not yet exist for some feature.
1841
1842
@item unsupported "@var{string}"
1843
@cindex test case, declaring no support
1844
Declares that a test case depends on some facility that does not exist
1845
in the testing environment. @code{unsupported} writes in the log file a
1846
message beginning with @samp{UNSUPPORTED}, appending the argument
1847
@var{string}.
1848
1849
@item get_warning_threshold
1850
@cindex test case, WARNING threshold
1851
Returns the current value of @code{warning_threshold}.
1852
The default value is 3.
1853
1854
@item set_warning_threshold @var{threshold}
1855
@cindex test case, WARNING threshold
1856
Sets the value of @code{warning_threshold}.
1857
A value of @code{0} disables it: calls to @code{warning} will not turn
1858
a @samp{PASS} or @samp{FAIL} into an @samp{UNRESOLVED}.
1859
1860
@item transform "@var{toolname}"
1861
@cindex transform tool name
1862
@cindex installed tool name
1863
@cindex tool name, as installed
1864
@cindex name transformations
1865
Generates a string for the name of a tool as it was configured and
1866
installed, given its native name (as the argument @var{toolname}).
1867
This makes the assumption that all tools are installed using the same
1868
naming conventions: it extrapolates from the invocation name for
1869
@file{runtest}. For example, if you call @code{runtest} as
1870
@file{m68k-vxworks-runtest}, the result of @w{@samp{ transform "gcc" }}
1871
is @samp{m68k-vxworks-gcc}.
1872
1873
@item ishost "@var{host}"
1874
@cindex host configuration test
1875
Tests for a particular @emph{host} environment. If the currently
1876
configured host matches the argument string, the result is @code{1};
1877
otherwise the result is @code{0}. @var{host} must be a full three-part
1878
@code{configure} host name; in particular, you may not use the shorter
1879
nicknames supported by @code{configure} (but you can use wildcard
1880
characters, using shell syntax, to specify sets of names).
1881
1882
@item istarget "@var{target}"
1883
@cindex target configuration test
1884
Tests for a particular @emph{target} environment. If the currently
1885
configured target matches the argument string, the result is @code{1};
1886
otherwise the result is @code{0}. @var{target} must be a full
1887
three-part @code{configure} target name; in particular, you may not use
1888
the shorter nicknames supported by @code{configure} (but you can use
1889
wildcard characters, using shell syntax, to specify sets of names). If it is
1890
passed a @code{NULL} string, then it returns the name of the build
1891
canonical configuration.
1892
1893
@item isbuild "@var{host}"
1894
@cindex build host configuration test
1895
Tests for a particular @emph{build host} environment. If the currently
1896
configured host matches the argument string, the result is @code{1};
1897
otherwise the result is @code{0}. @var{host} must be a full three-part
1898
@code{configure} host name; in particular, you may not use the shorter
1899
nicknames supported by @code{configure} (but you can use wildcard
1900
characters, using shell syntax, to specify sets of names). If it is
1901
passed a @code{NULL} string, then it returns the name of the build
1902
canonical configuration.
1903
1904
item is3way "@var{host}"
1905
@cindex canadian cross configuration test
1906
Tests for a canadian cross. This is when the tests will be run on a
1907
remotly hosted cross compiler. If it is a canadian cross, then the
1908
result is @code{1}; otherwise the result is @code{0}.
1909
1910
@item isnative
1911
@cindex native configuration test
1912
Tests whether the current configuration has the same host and target.
1913
When it runs in a @emph{native} configuration this procedure returns a
1914
@code{1}; otherwise it returns a @code{0}.
1915
1916
@item load_lib "@var{library-file}"
1917
@cindex load library file
1918
Loads the file @var{library-file} by searching a fixed path built into
1919
@code{runtest}. If DejaGnu has been installed, it looks in a path
1920
starting with the installed library directory. If you are running
1921
DejaGnu directly from a source directory, without first running
1922
@samp{make install}, this path defaults to the current directory. In
1923
either case, it then looks in the current directory for a directory
1924
called @code{lib}. If there are duplicate definitions, the last one
1925
loaded takes precedence over the earlier ones.
1926
1927
@item setup_xfail "@var{config} @r{[}@var{bugid}@r{]}"
1928
@c two spaces above to make it absolutely clear there's whitespace---a
1929
@c crude sort of italic correction!
1930
@cindex test case, expecting failure
1931
@cindex failure, expected
1932
@cindex expected failure
1933
Declares that the test is expected to fail on a particular set of
1934
configurations. The @var{config} argument must be a list of full
1935
three-part @code{configure} target name; in particular, you may not use
1936
the shorter nicknames supported by @code{configure} (but you can use the
1937
common shell wildcard characters to specify sets of names). The
1938
@var{bugid} argument is optional, and used only in the logging file
1939
output; use it as a link to a bug-tracking system such as @sc{gnats}
1940
(@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
1941
1942
See notes under setup_kfail (below).
1943
1944
@item setup_kfail "@var{config} @r{[}@var{bugid}@r{]}"
1945
@c two spaces above to make it absolutely clear there's whitespace---a
1946
@c crude sort of italic correction!
1947
@cindex test case, known failure
1948
@cindex failure, known
1949
@cindex known failure
1950
Declares that the test is known to fail on a particular set of
1951
configurations. The @var{config} argument must be a list of full
1952
three-part @code{configure} target name; in particular, you may not use
1953
the shorter nicknames supported by @code{configure} (but you can use the
1954
common shell wildcard characters to specify sets of names). The
1955
@var{bugid} argument is mandatory, and used only in the logging file
1956
output; use it as a link to a bug-tracking system such as @sc{gnats}
1957
(@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
1958
1959
@cindex @code{XFAIL}, producing
1960
@cindex @code{KFAIL}, producing
1961
@cindex @code{XPASS}, producing
1962
@cindex @code{KPASS}, producing
1963
Once you use @code{setup_xfail} or @code{setup_kfail}, the @code{fail}
1964
and @code{pass} procedures produce the messages @samp{XFAIL} or @samp{KFAIL}
1965
and @samp{XPASS} or @samp{KPASS} respectively, allowing you to distinguish
1966
expected/known failures (and unexpected success!) from other test outcomes.
1967
1968
If a test is marked as both expected to fail and known to fail for a
1969
certain configuration, a @samp{KFAIL} message will be generated.
1970
As @samp{KFAIL} messages are expected to draw more attention than
1971
the @samp{XFAIL} ones this will hopefuly ensure the test result is not
1972
overlooked.
1973
1974
@emph{Warning:} you must clear the expected/known failure after using
1975
@code{setup_xfail} or @code{setup_kfail} in a test case. Any call to
1976
@code{pass} or @code{fail} clears the expectedknown failure implicitly;
1977
if the test has some other outcome, e.g. an error, you can call
1978
@code{clear_xfail} to clear the expected failure or @code{clear_kfail}
1979
to clear the known failure explicitly. Otherwise, the expected-failure
1980
declaration applies to whatever test runs next, leading to surprising
1981
results.
1982
1983
@item check_conditional_xfail @var{message} @var{targets} @var{includes} @var{excludes}
1984
@cindex test case, expecting a conditional failure
1985
@cindex failure, conditional expected
1986
@cindex conditional expected failure
1987
1988
This procedure adds a conditional xfail, based on compiler options used to
1989
create a test case executable. If an include options is found in the
1990
compiler flags, and it's the right architecture, it'll trigger an
1991
XFAIL. Otherwise it'll produce an ordinary FAIL. You can also specify
1992
flags to exclude. This makes a result be a FAIL, even if the included
1993
options are found. To set the conditional, set the variable
1994
@var{compiler_conditional_xfail_data} to the fields "[message string] [targets
1995
list] [includes list] [excludes list]" (descriptions below). This is the
1996
checked at pass/fail decision time, so there is no need to call the
1997
procedure yourself, unless you wish to know if it gets triggered. After
1998
a pass/fail, the variable is reset, so it doesn't effect other tests.
1999
2000
The parameters are:
2001
2002
@table @code
2003
@item message
2004
is the message to print with the normal test result
2005
2006
@item targets
2007
is a string with the targets to activate this conditional on.
2008
2009
@item includes
2010
is a list of sets of options to search for in the compiler options to
2011
activate this conditional. If the list of sets of options is empty or
2012
if any set of the options matches, then this conditional is true.
2013
(It may be useful to specify an empty list of include sets if the
2014
conditional is always true unless one of the exclude sets matches.)
2015
2016
@item excludes
2017
is a list of sets of options to search for in the compiler options to
2018
activate this conditional. If any set of the options matches,
2019
(regardless of whether any of the include sets match) then this
2020
conditional is de-activated.
2021
@end table
2022
2023
returns:
2024
2025
@table @code
2026
@item 1
2027
if the conditional is true
2028
@item 0
2029
if the conditional is false
2030
@end table
2031
2032
An example of setting the variable would be:
2033
2034
@cartouche
2035
@smallexample
2036
set compiler_conditional_xfail_data @{@ \
2037
"I sure wish I knew why this was hosed" \
2038
"sparc*-sun*-* *-pc-*-*" \
2039
@{@"-Wall -v" "-O3"@}@ \
2040
@{@"-O1" "-Map" @}@ \
2041
@}@
2042
@end smallexample
2043
@end cartouche
2044
2045
What this does is it matches only for these two targets if "-Wall -v" or
2046
"-O3" is set, but neither "-O1" or "-Map" is set.
2047
2048
For a set to match, the options specified are searched for independantly of
2049
each other, so a "-Wall -v" matches either "-Wall -v" or "-v -Wall". A space
2050
seperates the options in the string. Glob-style regular expressions are also
2051
permitted.
2052
2053
@item clear_xfail @var{config}
2054
@cindex cancelling expected failure
2055
@cindex expected failure, cancelling
2056
Cancel an expected failure (previously declared with @code{setup_xfail})
2057
for a particular set of configurations. The @var{config} argument is a
2058
list of configuration target names. It is only necessary to call
2059
@code{clear_xfail} if a test case ends without calling either
2060
@code{pass} or @code{fail}, after calling @code{setup_xfail}.
2061
2062
@item clear_kfail @var{config}
2063
@cindex cancelling known failure
2064
@cindex known failure, cancelling
2065
Cancel a known failure (previously declared with @code{setup_kfail})
2066
for a particular set of configurations. The @var{config} argument is a
2067
list of configuration target names. It is only necessary to call
2068
@code{clear_kfail} if a test case ends without calling either
2069
@code{pass} or @code{fail}, after calling @code{setup_kfail}.
2070
2071
@item verbose @r{[}-log@r{]} @r{[}-n@r{]} @r{[}--@r{]} @r{[}--@r{]} "@var{string}" @var{number}
2072
@cindex @code{verbose} builtin function
2073
Test cases can use this function to issue helpful messages depending on
2074
the number of @samp{--verbose} options on the @code{runtest} command
2075
line. It prints @var{string} if the value of the variable
2076
@code{verbose} is higher than or equal to the optional @var{number}. The
2077
default value for @var{number} is 1. Use the optional @samp{-log} argument
2078
to cause @var{string} to always be added to the log file, even if it won't
2079
be printed. Use the optional @samp{-n} argument to print @var{string}
2080
without a trailing newline. Use the optional @samp{--} argument if
2081
@var{string} begins with "-". To log test results to a XML file use the
2082
@samp{-x} argument.
2083
2084
@end ftable
2085
2086
@noindent
2087
@node remote.exp, utils.exp, framework.exp, DejaGnu Builtins
2088
@subsection Remote Communication Procedures
2089
2090
@kindex remote.exp
2091
@kindex lib/remote.exp
2092
@cindex remote connection procedures
2093
@cindex communications procedures
2094
@file{lib/remote.exp} defines these functions, for establishing and
2095
managing communications:
2096
2097
@emph{Procedures to establish a connection:} Each of these procedures
2098
tries to establish the connection up to three times before returning.
2099
Warnings (if retries will continue) or errors (if the attempt is
2100
abandoned) report on communication failures. The result for any of
2101
these procedures is either @code{-1}, when the connection cannot be
2102
established, or the spawn ID returned by the @code{expect} command
2103
@code{spawn}.
2104
2105
It use the value of the @code{connect} field in the @code{target_info}
2106
array (was @code{connectmode} as the type of connection to make. Current
2107
supported connection types are tip, kermit, telnet, rsh, rlogin, and
2108
netdata. If the @code{--reboot} option was used on the runtest command
2109
line, then the target is rebooted before the connection is made.
2110
2111
@ftable @code
2112
2113
@item remote_open @var{type}
2114
@cindex Opening a remote connection
2115
@emph{Remote Connection Procedure.} This is passed @emph{host} or
2116
@emph{target}. Host or target refers to whether it is a connection to a
2117
remote target, or a remote host. This opens the connection to the
2118
desired target or host using the default values in the configuration
2119
system. It returns that @code{spawn_id} of the process that manages the
2120
connection. This value can be used in @code{expect} or @code{exp_send}
2121
statements, or passed to other procedures that need the connection
2122
process's id. This also sets the @code{fileid} field in the
2123
@code{target_info} array.
2124
2125
2126
@item remote_close @var{shellid}
2127
@cindex Closing a remote connection
2128
@emph{shellid} is value returned by a call to @code{remote_open}. This
2129
closes the connection to the target so resources can be used by
2130
others. This parameter can be left off if the @code{fileid} field in the
2131
@code{target_info} array is set.
2132
2133
@item telnet @var{hostname} @var{port}
2134
@itemx rlogin @var{hostname}
2135
@itemx rsh @var{hostname}
2136
@cindex IP network procedures
2137
@cindex network (IP) procedures
2138
@emph{IP network procedures.} @var{hostname} refers to the IP address or
2139
name (for example, an entry in @file{/etc/hosts}) for this target. The
2140
procedure names reflect the Unix utility used to establish a
2141
connection. The optional @var{port} is used to specify the IP port
2142
number. The value of the @code{netport} field in the @code{target_info}
2143
array is used. (was @code{$netport}) This value has two parts, the
2144
hostname and the port number, seperated by a @emph{:}. If @code{host} or
2145
@code{target} is used in the @code{hostname} field, than the config
2146
array is used for all information.
2147
2148
@item tip @var{port}
2149
@cindex serial line connection, @code{tip}
2150
@emph{Serial line procedure.} Connect using the Unix utility @code{tip}.
2151
@var{port} must be a name from the @code{tip} configuration file
2152
@file{/etc/remote}. Often, this is called @samp{hardwire}, or something
2153
like @samp{ttya}. This file holds all the configuration data for
2154
the serial port. The value of the @code{serial} field in the
2155
@code{target_info} array is used. (was @code{$serialport}) If
2156
@code{host} or @code{target} is used in the @code{port} field, than
2157
the config array is used for all information.
2158
2159
@item kermit @var{port} @var{bps}
2160
@cindex serial line connection, @code{kermit}
2161
@emph{Serial line procedure.} Connect using the program @code{kermit}.
2162
@var{port} is the device name, e.g. @file{/dev/ttyb}. @var{bps} is
2163
the line speed to use (in bits per second) for the connection. The value
2164
of the @code{serial} field in the @code{target_info} array is used. (was
2165
@code{$serialport}) If @code{host} or @code{target} is used in the
2166
@code{port} field, than the config array is used for all information.
2167
2168
@end ftable
2169
2170
@noindent
2171
@emph{Procedures to manage a connection:}
2172
2173
@ftable @code
2174
@item tip_download @var{spawnid} @var{file}
2175
@cindex download, @code{tip}
2176
@cindex serial download, @code{tip}
2177
Download @file{@var{file}} to the process @var{spawnid} (the value
2178
returned when the connection was established), using the @code{~put}
2179
command under @code{tip}. Most often used for single board computers
2180
that require downloading programs in @sc{ascii} S-records. Returns
2181
@code{1} if an error occurs, @code{0} otherwise.
2182
2183
@item exit_remote_shell @var{spawnid}
2184
@cindex terminating remote connection
2185
@cindex remote connection, ending
2186
Exits a remote process started by any of the connection procedures.
2187
@var{spawnid} is the result of the connection procedure that started the
2188
remote process.
2189
2190
@item download @var{file} @r{[} @var{spawnid} @r{]}
2191
@cindex download a file
2192
After you establish a connection to a target, you can download programs
2193
using this command. @code{download} reads in @var{file} (object code in
2194
S-record format) and writes it to the device controlling this
2195
@var{spawnid}. (From the point of view of the target, the S-record file
2196
comes in via standard input.)
2197
2198
If you have more than one target active, you can use the optional argument
2199
@var{spawnid} to specify an alternative target (the default is the most
2200
recently established @var{spawnid}.)
2201
@end ftable
2202
2203
@noindent
2204
@node utils.exp, target.exp, remote.exp, DejaGnu Builtins
2205
@subsection Utility Procedures
2206
2207
@kindex utils.exp
2208
@kindex lib/utils.exp
2209
@file{lib/utils.exp} defines these utility procedures:
2210
2211
@ftable @code
2212
@item getdirs @var{dir}
2213
@itemx getdirs @var{dir} @var{pattern}
2214
@cindex directories matching a pattern
2215
@cindex pattern match, directory
2216
Returns a list of all the directories in the single directory @var{dir}
2217
that match @var{pattern}. If you do not specify @var{pattern},
2218
@code{getdirs} assumes @samp{*}. You may use the common shell wildcard
2219
characters in @var{pattern}. If no directories match the pattern, then a
2220
@code{NULL} string is returned.
2221
2222
@item find @var{dir} @var{pattern}
2223
@cindex files matching a pattern
2224
@cindex pattern match, filenames
2225
Search for files whose names match @var{pattern} (using shell wildcard
2226
characters for filename expansion). Search subdirectories recursively,
2227
starting at @var{dir}. The result is the list of files whose names
2228
match; if no files match, the result is empty. Filenames in the result
2229
include all intervening subdirectory names. If no files match the
2230
pattern, then a @code{NULL} string is returned.
2231
2232
@item which @var{binary}
2233
@cindex path lookup
2234
Searches the execution path for an executable file @var{binary}, like
2235
the the BSD @code{which} utility. This procedure uses the shell
2236
environment variable @samp{PATH}. It returns @code{0} if the binary is
2237
not in the path, or if there is no @samp{PATH} environment variable. If
2238
@var{binary} is in the path, it returns the full path to @var{binary}.
2239
2240
@item grep @var{filename} @var{regexp}
2241
@item grep @var{filename} @var{regexp} line
2242
@cindex regular expression, file contents
2243
@cindex searching file contents
2244
Search the file called @var{filename} (a fully specified path) for lines
2245
that contain a match for regular expression @var{regexp}. The result is
2246
a list of all the lines that match. If no lines match, the result is an
2247
empty string. Specify @var{regexp} using the standard regular
2248
expression style used by the Unix utility program @code{grep}.
2249
2250
Use the optional third argument @samp{line} to start lines in the result
2251
with the line number in @var{filename}. (This argument is simply an
2252
option flag; type it just as shown---@samp{line}.)
2253
2254
@item diff @var{filename} @var{filename}
2255
@cindex finding file differences
2256
@cindex comparing files
2257
Compares the two files and returns a 1 if they match, or a 0 if they
2258
don't. If @code{verbose} is set, then it'll print the differences to the
2259
screen.
2260
2261
@item slay @var{name}
2262
@cindex slaying processes
2263
This look in the process table for @var{name} and send it a unix
2264
@code{SIGINT}, killing the process.
2265
2266
@item absolute @var{path}
2267
@cindex converting relative paths to absolute
2268
This procedure takes the relative @var{path}, and converts it to an
2269
absolute path.
2270
2271
@item psource @var{filename}
2272
@cindex sourcing Tcl files
2273
This sources the file @var{filename}, and traps all errors. It also
2274
ignores all extraneous output. If there was an error it returns a 1,
2275
otherwise it returns a 0.
2276
2277
@item prune @var{list} @var{pattern}
2278
@cindex list, pruning
2279
Remove elements of the Tcl list @var{list}. Elements are fields
2280
delimited by spaces. The result is a copy of @var{list}, without any
2281
elements that match @var{pattern}. You can use the common shell
2282
wildcard characters to specify @var{pattern}.
2283
2284
@item setenv @var{var} @var{val}
2285
@cindex setting environment variables
2286
Sets the variable @var{var} to the value @var{val}.
2287
2288
@item unsetenv @var{var}
2289
@cindex unsetting environment variables
2290
Unsets the environment variable @var{var}
2291
2292
@item getenv @var{var}
2293
@cindex getting environment variables
2294
returns the value of @var{var} in the environment if it exists,
2295
otherwise it returns @code{NULL}.
2296
2297
@item runtest_file_p @var{runtests} @var{testcase}
2298
@cindex selecting a range of tests
2299
@cindex tests, running specifically
2300
Search @var{runtests} for @var{testcase} and return 1 if found, 0 if not.
2301
@var{runtests} is a list of two elements. The first is a copy of what was
2302
on the right side of the @code{=} if @samp{foo.exp="@dots{}"} was specified,
2303
or an empty string if no such argument is present. The second is the
2304
pathname of the current testcase under consideration.
2305
This is used by tools like compilers where each testcase is a file.
2306
2307
@item prune_system_crud @var{system} @var{text}
2308
@cindex pruning system output, examining program output
2309
For system @var{system}, delete text the host or target operating system might
2310
issue that will interfere with pattern matching of program output in
2311
@var{text}. An example is the message that is printed if a shared library
2312
is out of date.
2313
2314
@end ftable
2315
2316
@noindent
2317
@node target.exp, debugger.exp, utils.exp, DejaGnu Builtins
2318
@subsection Cross target procedure
2319
2320
@kindex target.exp
2321
@kindex lib/target.exp
2322
@file{lib/target.exp} defines these utility procedures:
2323
2324
@ftable @code
2325
2326
@item push_target @emph{name}
2327
@cindex set current target
2328
This makes the target named @emph{name} be the current target
2329
connection. The value of @emph{name} is an index into the
2330
@code{target_info} array and is set in the global config file.
2331
2332
@item pop_target
2333
@cindex unset current target
2334
This unsets the current target connection.
2335
2336
@item list_targets
2337
@cindex lists supported targets
2338
This lists all the supported targets for this architecture.
2339
2340
@item push_host @emph{name}
2341
@cindex set current host
2342
This makes the host named @emph{name} be the current remote host
2343
connection. The value of @emph{name} is an index into the
2344
@code{target_info} array and is set in the global config file.
2345
2346
@item pop_host
2347
@cindex unset current host
2348
This unsets the current host connection.
2349
2350
@c @item compile @emph{file}
2351
@cindex compile a file
2352
This invokes the compiler as set by @code{CC} to compile the file
2353
@emph{file}. The default options for many cross compilation targets are
2354
@emph{guessed} by DejaGnu, and these options can be added to by passing
2355
in more parameters as arguments to @code{compile}. Optionally, this will
2356
also use the value of the @code{cflags} field in the target config
2357
array. If the host is not the same as the build machines, then then
2358
compiler is run on the remote host using @code{execute_anywhere}.
2359
2360
@c @item archive @emph{file}
2361
@cindex archive object files
2362
This produces an archive file. Any parameters passed to @code{archive}
2363
are used in addition to the default flags. Optionally, this will
2364
also use the value of the @code{arflags} field in the target config
2365
array. If the host is not the same as the build machines, then then
2366
archiver is run on the remote host using @code{execute_anywhere}.
2367
2368
@c @item ranlib @emph{file}
2369
@cindex ranlib a file
2370
This generates an index for the archive file for systems that aren't
2371
POSIX yet. Any parameters passed to @code{ranlib} are used in for the
2372
flags.
2373
2374
@item execute_anywhere @emph{cmdline}
2375
@cindex executing commands remotely
2376
This executes the @emph{cmdline} on the proper host. This should be used
2377
as a replacement for the Tcl command @code{exec} as this version
2378
utilizes the target config info to execute this command on the build
2379
machine or a remote host. All config information for the remote host
2380
must be setup to have this command work. If this is a canadian cross,
2381
(where we test a cross compiler that runs on a different host then where
2382
DejaGnu is running) then a connection is made to the remote host and
2383
the command is executed there. It returns either @emph{REMOTERROR} (for
2384
an error) or the output produced when the command was executed. This is
2385
used for running the tool to be tested, not a test case.
2386
2387
@end ftable
2388
2389
@node debugger.exp, , target.exp, DejaGnu Builtins
2390
@subsection Debugging Procedures
2391
2392
@kindex debugger.exp
2393
@kindex lib/debugger.exp
2394
@file{lib/debugger.exp} defines these utility procedures:
2395
2396
@ftable @code
2397
2398
@item dumpvars @emph{expr}
2399
@cindex Print global variable values
2400
This takes a csh style regular expression (glob rules) and prints the
2401
values of the global variable names that match. It is abbreviated as
2402
@code{dv}
2403
2404
@item dumplocals @emph{expr}
2405
@cindex Print local variable value
2406
This takes a csh style regular expression (glob rules) and prints the
2407
values of the local variable names that match. It is abbreviated as
2408
@code{dl}.
2409
2410
@item dumprocs @emph{expr}
2411
@cindex Print procedure bodies
2412
This takes a csh style regular expression (glob rules) and prints the
2413
body of all procs that match. It is abbreviated as @code{dp}
2414
2415
@item dumpwatch @emph{expr}
2416
@cindex Print watchpoints
2417
This takes a csh style regular expression (glob rules) and prints all
2418
the watchpoints. It is abbreviated as @code{dw}.
2419
2420
@c FIXME: finish these when the code is fixed.
2421
@c @item watcharray @emph{element} @emph{type}
2422
@c @cindex Set a watchpoint on an array
2423
@c This sets an watchpoint of the @emph{element-type} on the
2424
@c @item watchvar v null type
2425
@c @cindex Set a watchpoint on a variable
2426
2427
@item watchunset @emph{var}
2428
@cindex Watch when a variable is unset
2429
This breaks program execution when the variable @emph{var} is unset. It
2430
is abbreviated as @code{wu}.
2431
2432
@item watchwrite @emph{var}
2433
@cindex Watch when a variable is written
2434
This breaks program execution when the variable @emph{var} is
2435
written. It is abbreviated as @code{ww}.
2436
2437
@item watchread @emph{var}
2438
@cindex Watch when a variable is read
2439
This breaks program execution when the variable @emph{var} is read. It
2440
is abbreviated as @code{wr}.
2441
2442
@item watchdel @emph{watch}
2443
@cindex Delete a watchpoint.
2444
This deletes a the watchpoint for @emph{watch}. It is abbreviated as
2445
@code{wd}.
2446
2447
@item print @emph{var}
2448
@cindex Printing variable values
2449
This prints the value of the variable @emph{var}. It is abbreviated as
2450
@code{p}.
2451
2452
@item quit
2453
@cindex Quiting DejaGnu
2454
This makes runtest exit. It is abbreviated as @code{q}.
2455
2456
@item bt
2457
@cindex Print a backtrace
2458
This prints a backtrace of the executed Tcl commands.
2459
2460
@end ftable
2461
2462
@node Target Dependent
2463
@section Target dependent procedures
2464
@cindex target dependent procedures
2465
2466
@c FIXME? These may be renamed to just "start", "load", "exit", and
2467
@c "version" eventually.
2468
2469
Each combination of target and tool requires some target-dependent
2470
procedures. The names of these procedures have a common form: the tool
2471
name, followed by an underbar @samp{_}, and finally a suffix describing
2472
the procedure's purpose. For example, a procedure to extract the
2473
version from @sc{gdb} is called @samp{gdb_version}. @xref{Init Module,,
2474
Initialization Module}, for a discussion of how DejaGnu arranges to find
2475
the right procedures for each target.
2476
2477
@code{runtest} itself calls only two of these procedures,
2478
@code{@var{tool}_exit} and @code{@var{tool}_version}; these procedures use
2479
no arguments.
2480
2481
The other two procedures, @code{@var{tool}_start} and
2482
@code{@var{tool}_load}, are only called by the test suites themselves
2483
(or by testsuite-specific initialization code); they may take arguments
2484
or not, depending on the conventions used within each test suite.
2485
2486
@ftable @code
2487
@item @var{tool}_start
2488
@cindex start procedure, tested tools
2489
Starts a particular tool. For an interactive tool,
2490
@code{@var{tool}_start} starts and initializes the tool, leaving the
2491
tool up and running for the test cases; an example is @code{gdb_start},
2492
the start function for @sc{gdb}. For a batch oriented tool,
2493
@code{@var{tool}_start} is optional; the recommended convention is to
2494
let @code{@var{tool}_start} run the tool, leaving the output in a
2495
variable called @code{comp_output}. Test scripts can then analyze
2496
@samp{$comp_output} to determine the test results. An example of this
2497
second kind of start function is @code{gcc_start}, the start function
2498
for @sc{gcc}.
2499
2500
@code{runtest} itself @emph{does not call} @code{@var{tool}_start}. The
2501
initialization module @file{@var{tool}_init.exp} must call
2502
@code{@var{tool}_start} for interactive tools; for batch-oriented tools,
2503
each individual test script calls @code{@var{tool}_start} (or makes
2504
other arrangements to run the tool).
2505
2506
@item @var{tool}_load
2507
@cindex load procedure, tested tools
2508
Loads something into a tool. For an interactive tool, this conditions
2509
the tool for a particular test case; for example, @code{gdb_load} loads
2510
a new executable file into the debugger. For batch oriented tools,
2511
@code{@var{tool}_load} may do nothing---though, for example, the
2512
@sc{gcc} support uses @code{gcc_load} to load and run a binary on the
2513
target environment. Conventionally, @code{@var{tool}_load} leaves the
2514
output of any program it runs in a variable called @samp{exec_output}.
2515
Writing @code{@var{tool}_load} can be the most complex part of extending
2516
DejaGnu to a new tool or a new target, if it requires much communication
2517
coding or file downloading.
2518
2519
Test scripts call @code{@var{tool}_load}.
2520
2521
@item @var{tool}_exit
2522
@cindex exit procedure, tested tools
2523
Cleans up (if necessary) before @code{runtest} exits. For interactive
2524
tools, this usually ends the interactive session. You can also use
2525
@code{@var{tool}_exit} to remove any temporary files left over from the
2526
tests.
2527
2528
@code{runtest} calls @code{@var{tool}_exit}.
2529
2530
@item @var{tool}_version
2531
@cindex version procedure, tested tools
2532
Prints the version label and number for @var{tool}. This is called by
2533
the DejaGnu procedure that prints the final summary report. The output
2534
should consist of the full path name used for the tested tool, and its
2535
version number.
2536
2537
@code{runtest} calls @code{@var{tool}_version}.
2538
@end ftable
2539
2540
The usual convention for return codes from any of these procedures
2541
(although it is not required by @code{runtest}) is to return @code{0} if
2542
the procedure succeeded, @code{1} if it failed, and @code{-1} if there
2543
was a communication error.
2544
2545
@node Cross Targets
2546
@section Remote targets supported
2547
2548
@cindex targets
2549
@cindex remote testing
2550
The DejaGnu distribution includes support for the following remote
2551
targets. You can set the target name and the connect mode in the
2552
@file{site.exp} file (using the Tcl variables @samp{targetname} and
2553
@samp{connectmode}, respectively), or on the @code{runtest} command line
2554
(using @samp{--name} and @samp{--connect}).
2555
2556
@table @strong
2557
@item @sc{amd} 29000, with UDI protocol
2558
Configure DejaGnu for target @samp{a29k-amd-udi}. (Cygnus
2559
@code{configure} also recognizes the abbreviation @samp{udi29k}.) Then,
2560
to run tests, use the @code{runtest} target name to specify whether you
2561
want to use a simulator, or a particular hardware board. The particular
2562
string to use with @samp{--name} will depend on your UDI setup file,
2563
@file{udi_soc} (if @file{udi_soc} is not in your working directory, the
2564
environment variable @samp{UDICONF} should contain a path to this file).
2565
For example, if your UDI setup file includes these lines:
2566
@end table
2567
@c table "ends" *only* to allow wider example below
2568
2569
@cartouche
2570
@smallexample
2571
iss AF_UNIX * isstip -r /home/gnu/29k/src/osboot/sim/osboot
2572
mon AF_UNIX * montip -t serial -baud 9600 -com /dev/ttyb
2573
@end smallexample
2574
@end cartouche
2575
2576
@table @strong
2577
@item @w{ }
2578
@c fake out table/item into continuing w/same margin as before
2579
You can use @samp{--name iss} to run tests on the simulator, and
2580
@samp{--name mon} to run tests on the 29K hardware. See the
2581
manufacturer's manuals for more information on UDI and @file{udi_soc}.
2582
@c FIXME! Is there a better ref than "the manufacturer's manuals"?
2583
2584
@kindex mondfe
2585
The default connect protocol is @samp{mondfe} with either back end.
2586
@code{mondfe} is the only shell DejaGnu supports for UDI targets.
2587
@code{mondfe} is an @sc{amd} specific monitor program freely available
2588
from @sc{amd}.
2589
2590
@emph{Warning:} This target requires @sc{gdb} version 4.7.2 (or
2591
greater). Earlier versions of @sc{gdb} do not fully support the
2592
@code{load} command on this target, so DejaGnu has no way to load
2593
executable files from the debugger.
2594
2595
@item Motorola 680x0 boards, a.out or @sc{coff} object format
2596
Configure DejaGnu for any remote target matching @samp{m68k-*}.
2597
2598
@emph{Warning:} Most @samp{m68k-*} configurations run all tests only for
2599
native testing (when the target is the same as the host). When you
2600
specify most of these targets for a cross configuration, you will only be
2601
able to use tests that run completely within the host (for example,
2602
tests of the binary utilities such as the archiver; or compiler tests
2603
that only generate code rather than running it).
2604
2605
To run a.out or @sc{coff} binaries on a remote M68K, you must configure
2606
DejaGnu for a particular target board. @samp{m68k-abug} is an example.
2607
(In general for an embedded environment, because it does not have absolute
2608
addresses, a.out is not a good choice for output format in any case; most
2609
often S-records or Hex-32 are used instead.)
2610
2611
@item Motorola 68K MVME 135 board running ABug boot monitor
2612
Configure for @samp{m68k-abug-aout} or @samp{m68k-abug-coff} (as a
2613
target). This boot monitor can only download S-records; therefore, the
2614
DejaGnu tests for this environment require a linker command script to
2615
convert either output format to S-records, setting the default addresses
2616
for @code{.text}, @code{.bss}, and @code{.data}.
2617
2618
With this configuration, the default for @samp{--connect} is @samp{tip}.
2619
@samp{tip} is the only communications protocol supported for connecting
2620
to @samp{m68k-abug-*} targets. @samp{tip} uses an @sc{ascii} downloader
2621
(the @code{~put} command) to load S-records into the target board. The
2622
@samp{--name} string must be a machine name that @code{tip}
2623
understands (for example, on some @code{tip} implementations it must be
2624
an entry from the initialization file for @code{tip}; this file is
2625
sometimes called @file{/etc/remote}).
2626
2627
See your system documentation for information on how to create new
2628
entries in @file{/etc/remote}. (Some @sc{unix} systems are distributed
2629
with at least one default entry with a name resembling @samp{hardwire};
2630
if your system has one, you can edit it, or make a modified copy with a
2631
new name.) When you have a working @file{/etc/remote} entry
2632
@var{abugtarget}, you should be able to type @samp{tip
2633
@var{abugtarget}}, and get the prompt @samp{135ABUG>} from the board.
2634
Use the same @var{abugtarget} string with @samp{runtest --name}.
2635
2636
@item Motorola IDP board running the rom68k boot monitor
2637
@c FIXME 1: this doesn't really say anything! OK, so functionality is
2638
@c the same. Is object code the same (srecords)? Do we configure with
2639
@c the same triplets? What is the default for --connect? Is
2640
@c any comms method other than tip supported? What prompt to expect
2641
@c when tip connected?
2642
@c FIXME 2: should @code{BUG} below be @code{ABUG}?
2643
This is the same in functionality as the MVME board running the
2644
@code{BUG} boot monitor. Only the monitor commands and the addresses are
2645
different.
2646
2647
@item VxWorks (Motorola 68K or Intel 960)
2648
Configure DejaGnu for either @samp{m68k-wrs-vxworks} (abbreviated
2649
@samp{vxworks68}) or @samp{i960-wrs-vxworks} (abbreviated
2650
@samp{vxworks960}). Since both targets support IP addressing, specify
2651
the network address (for example, a host name from @file{/etc/hosts})
2652
with @samp{--name}.
2653
2654
The default connect protocol is @samp{rlogin}, but you can use any of
2655
@samp{--connect rlogin}, @samp{--connect telnet}, or @samp{--connect
2656
rsh}.
2657
2658
Test scripts need no special code to load programs into these targets;
2659
since VxWorks supports NFS, all you must do is ensure test programs are
2660
on an exported filesystem.
2661
2662
@cindex VxWorks, link with @samp{-r}
2663
When you compile for VxWorks, use the linker @samp{-r} option to make
2664
the linker output relocatable---at least if you want to use library
2665
routines. Many standard C routines are included in VxWorks; often no
2666
additional libraries are needed. See your VxWorks system documentation
2667
for additional details.
2668
@end table
2669
2670
@node Input Files
2671
@section The files DejaGnu reads
2672
@cindex input files
2673
2674
The @code{runtest} program used to invoke DejaGnu is a short shell
2675
script generated by @code{make} during the configuration process. Its
2676
main task is to read the main test framework driver, @file{runtest.exp}.
2677
2678
@file{runtest.exp}, in turn, reads @code{expect} code from certain other
2679
files, in this order:
2680
2681
@enumerate
2682
@item
2683
Each of the @file{site.exp} local definition files available.
2684
@xref{Customizing,,Setting @code{runtest} defaults}, for details.
2685
2686
@item
2687
@file{lib/utils.exp}, a collection of utility procedures. @xref{DejaGnu
2688
Builtins,,DejaGnu Builtins}, for descriptions of these procedures.
2689
2690
@item
2691
@file{lib/framework.exp}, a file of subroutines meant for @code{runtest}
2692
itself rather than for general-purpose use in both @code{runtest} and
2693
test suites.
2694
2695
@item
2696
@file{debugger.exp}, Don Libes' Tcl Debugger. (See @cite{A Debugger for
2697
Tcl Applications} by Don Libes. This paper is distributed with
2698
@code{expect} in PostScript form as the file
2699
@file{expect/tcl-debug.ps}.)
2700
2701
@item
2702
@file{lib/remote.exp}, a collection of subroutines meant for connecting
2703
to remote machines.
2704
2705
@item
2706
@file{lib/target.exp}, a collection of subroutines used for the
2707
configuration systems in DejaGnu. These procedures typically manipulate
2708
or utilize the configuration system.
2709
2710
@item
2711
@c FIXME! A comment in runtest.exp claims a system default is used if
2712
@c no tool-specific init file is not available; I couldn't see where
2713
@c the program flow actually does this, though---pesch 30jul1993.
2714
An initialization file @code{@var{tool}_init.exp}. @xref{Init
2715
Module,,Initialization module}, for more discussion of init files.
2716
@end enumerate
2717
2718
@c This hard page break is mainly intended for smallbook formatting;
2719
@c some examples in this section come out better if this starts at a
2720
@c page boundary.
2721
@page
2722
@node Output Files
2723
@section The files DejaGnu writes
2724
@cindex output files
2725
2726
@code{runtest} always writes two kinds of output files: summary logs and
2727
detailed logs. The contents of both of these are determined by your
2728
tests.
2729
2730
For troubleshooting, a third kind of output file is useful: use
2731
@samp{--debug} to request an output file showing details of what
2732
@code{expect} is doing internally.
2733
2734
@menu
2735
* Summary:: Files that summarize tests
2736
* Detail:: Files that contain complete test results
2737
* Debug:: Logging expect internal actions
2738
@end menu
2739
2740
@node Summary
2741
@subsection Summary log
2742
@cindex summary log
2743
2744
@code{runtest} always produces a summary output file
2745
@file{@var{tool}.sum}. This summary shows the names of all test files
2746
run; for each test file, one line of output from each @code{pass}
2747
command (showing status @samp{PASS}, @samp{XPASS} or @samp{KPASS}) or
2748
@code{fail} command (status @samp{FAIL}, @samp{XFAIL} or @samp{KFAIL});
2749
trailing summary statistics that count passing and failing tests (expected and
2750
unexpected); and the full pathname and version number of the tool
2751
tested. (All possible outcomes, and all errors, are always reflected in
2752
the summary output file, regardless of whether or not you specify
2753
@samp{--all}.)
2754
2755
If any of your tests use the procedures @code{unresolved},
2756
@code{unsupported}, or @code{untested}, the summary output also
2757
tabulates the corresponding outcomes.
2758
2759
For example, after @samp{runtest --tool binutils}, look for a summary
2760
log in @file{binutils.sum}. Normally, @code{runtest} writes this file
2761
in your current working directory; use the @samp{--outdir} option to
2762
select a different directory.
2763
2764
@need 3500
2765
@noindent
2766
Here is a short sample summary log:
2767
2768
@cartouche
2769
@smallexample
2770
Test Run By rob on Mon May 25 21:40:57 PDT 1992
2771
=== gdb tests ===
2772
Running ./gdb.t00/echo.exp ...
2773
PASS: Echo test
2774
Running ./gdb.all/help.exp ...
2775
PASS: help add-symbol-file
2776
PASS: help aliases
2777
PASS: help breakpoint "bre" abbreviation
2778
FAIL: help run "r" abbreviation
2779
Running ./gdb.t10/crossload.exp ...
2780
PASS: m68k-elf (elf-big) explicit format; loaded
2781
XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed
2782
C types
2783
=== gdb Summary ===
2784
# of expected passes 5
2785
# of expected failures 1
2786
# of unexpected failures 1
2787
/usr/latest/bin/gdb version 4.6.5 -q
2788
@end smallexample
2789
@end cartouche
2790
2791
@node Detail
2792
@subsection Detailed log
2793
@cindex detailed log
2794
2795
@code{runtest} also saves a detailed log file @file{@var{tool}.log},
2796
showing any output generated by tests as well as the summary output.
2797
For example, after @samp{runtest --tool binutils}, look for a detailed
2798
log in @file{binutils.log}. Normally, @code{runtest} writes this file
2799
in your current working directory; use the @samp{--outdir} option to
2800
select a different directory.
2801
2802
@need 4000
2803
@noindent
2804
Here is a brief example showing a detailed log for @sc{g++} tests:
2805
2806
@cartouche
2807
@smallexample
2808
Test Run By rob on Mon May 25 21:40:43 PDT 1992
2809
2810
=== g++ tests ===
2811
2812
--- Running ./g++.other/t01-1.exp ---
2813
PASS: operate delete
2814
2815
--- Running ./g++.other/t01-2.exp ---
2816
FAIL: i960 bug EOF
2817
p0000646.C: In function `int warn_return_1 ()':
2818
p0000646.C:109: warning: control reaches end of non-void function
2819
p0000646.C: In function `int warn_return_arg (int)':
2820
p0000646.C:117: warning: control reaches end of non-void function
2821
p0000646.C: In function `int warn_return_sum (int, int)':
2822
p0000646.C:125: warning: control reaches end of non-void function
2823
p0000646.C: In function `struct foo warn_return_foo ()':
2824
p0000646.C:132: warning: control reaches end of non-void function
2825
2826
--- Running ./g++.other/t01-4.exp ---
2827
FAIL: abort
2828
900403_04.C:8: zero width for bit-field `foo'
2829
--- Running ./g++.other/t01-3.exp ---
2830
FAIL: segment violation
2831
900519_12.C:9: parse error before `;'
2832
900519_12.C:12: Segmentation violation
2833
/usr/latest/bin/gcc: Internal compiler error: program cc1plus got
2834
fatal signal
2835
2836
=== g++ Summary ===
2837
2838
# of expected passes 1
2839
# of expected failures 3
2840
/usr/ps/bin/g++ version cygnus-2.0.1
2841
@end smallexample
2842
@end cartouche
2843
2844
@node Debug
2845
@subsection Logging @code{expect} internal actions
2846
@cindex debug log
2847
2848
With the @samp{--debug} option, you can request a log file showing the
2849
output from @code{expect} itself, running in debugging mode. This file
2850
(@file{dbg.log}, in the directory where you start @code{runtest}) shows
2851
each pattern @code{expect} considers in analyzing test output.
2852
2853
This file reflects each @code{send} command, showing the string sent as
2854
input to the tool under test; and each @code{expect} command, showing
2855
each pattern it compares with the tool output.
2856
2857
The log messages for @code{expect} begin with a message of the form
2858
2859
@smallexample
2860
expect: does @{@var{tool output}@} (spawn_id @var{n}) match pattern
2861
@{@var{expected pattern}@}?
2862
@end smallexample
2863
2864
@noindent
2865
For every unsuccessful match, @code{expect} issues a @samp{no} after
2866
this message; if other patterns are specified for the same
2867
@code{expect} command, they are reflected also, but without the first
2868
part of the message (@samp{expect@dots{}match pattern}).
2869
2870
When @code{expect} finds a match, the log for the successful match ends
2871
with @samp{yes}, followed by a record of the @code{expect} variables set
2872
to describe a successful match. Here is an excerpt from the debugging
2873
log for a @sc{gdb} test:
2874
2875
@c FIXME! Why is the second spawn_id shown 0 rather than 6?
2876
@cartouche
2877
@smallexample
2878
send: sent @{break gdbme.c:34\n@} to spawn id 6
2879
expect: does @{@} (spawn_id 6) match pattern @{Breakpoint.*at.* file
2880
gdbme.c, line 34.*\(gdb\) $@}? no
2881
@{.*\(gdb\) $@}? no
2882
expect: does @{@} (spawn_id 0) match pattern @{<return>@}? no
2883
@{\(y or n\) @}? no
2884
@{buffer_full@}? no
2885
@{virtual@}? no
2886
@{memory@}? no
2887
@{exhausted@}? no
2888
@{Undefined@}? no
2889
@{command@}? no
2890
break gdbme.c:34
2891
Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
2892
(gdb) expect: does @{break gdbme.c:34\r\nBreakpoint 8 at 0x23d8:
2893
file gdbme.c, line 34.\r\n(gdb) @} (spawn_id 6) match pattern
2894
@{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $@}? yes
2895
expect: set expect_out(0,start) @{18@}
2896
expect: set expect_out(0,end) @{71@}
2897
expect: set expect_out(0,string) @{Breakpoint 8 at 0x23d8: file
2898
gdbme.c, line 34.\r\n(gdb) @}
2899
expect: set expect_out(spawn_id) @{6@}
2900
expect: set expect_out(buffer) @{break gdbme.c:34\r\nBreakpoint 8
2901
at 0x23d8: file gdbme.c, line 34.\r\n(gdb) @}
2902
PASS: 70 0 breakpoint line number in file
2903
@end smallexample
2904
@end cartouche
2905
2906
@noindent
2907
This example exhibits three properties of @code{expect} and DejaGnu that
2908
might be surprising at first glance:
2909
2910
@itemize @bullet
2911
@item
2912
Empty output for the first attempted match. The first set of attempted
2913
matches shown ran against the output @samp{@{@}}---that is, no output.
2914
@code{expect} begins attempting to match the patterns supplied
2915
immediately; often, the first pass is against incomplete output (or
2916
completely before all output, as in this case).
2917
2918
@item
2919
Interspersed tool output. The beginning of the log entry for the second
2920
attempted match may be hard to spot: this is because the prompt
2921
@samp{(gdb) } appears on the same line, just before the @samp{expect:}
2922
that marks the beginning of the log entry.
2923
2924
@item
2925
Fail-safe patterns. Many of the patterns tested are fail-safe patterns
2926
provided by @sc{gdb} testing utilities, to reduce possible
2927
indeterminacy. It is useful to anticipate potential variations
2928
caused by extreme system conditions (@sc{gdb} might issue the message
2929
@samp{virtual memory exhausted} in rare circumstances), or by changes in
2930
the tested program (@samp{Undefined command} is the likeliest outcome if
2931
the name of a tested command changes).
2932
2933
The pattern @samp{@{<return>@}} is a particularly interesting fail-safe
2934
to notice; it checks for an unexpected @key{RET} prompt. This may
2935
happen, for example, if the tested tool can filter output through a
2936
pager.
2937
2938
These fail-safe patterns (like the debugging log itself) are primarily
2939
useful while developing test scripts. Use the @code{error} procedure to
2940
make the actions for fail-safe patterns produce messages starting with
2941
@samp{ERROR} on the @code{runtest} standard output, and in the detailed
2942
log file.
2943
@end itemize
2944
2945
@node Tests
2946
@chapter How To Write a Test Cases
2947
@cindex writing a test case
2948
@cindex test case, writing
2949
2950
@menu
2951
* Writing:: Writing a test case
2952
* Debugging:: Debugging a test case
2953
* Adding:: Adding a test case to a test suite
2954
* Hints:: Hints on writing a test case
2955
* Variables:: Special variables used by test cases
2956
@end menu
2957
2958
@node Writing
2959
@section Writing a test case
2960
2961
The easiest way to prepare a new test case is to base it on an existing
2962
one for a similar situation. There are two major categories of tests:
2963
batch or interactive. Batch oriented tests are usually easier to write.
2964
2965
The @sc{gcc} tests are a good example of batch oriented tests. All
2966
@sc{gcc} tests consist primarily of a call to a single common procedure,
2967
since all the tests either have no output, or only have a few warning
2968
messages when successfully compiled. Any non-warning output is a test
2969
failure. All the C code needed is kept in the test directory. The test
2970
driver, written in @code{expect}, need only get a listing of all the C
2971
files in the directory, and compile them all using a generic procedure.
2972
This procedure and a few others supporting for these tests are kept in
2973
the library module @file{lib/c-torture.exp} in the @sc{gcc} test suite.
2974
Most tests of this kind use very few @code{expect} features, and are
2975
coded almost purely in Tcl.
2976
2977
@noindent
2978
Writing the complete suite of C tests, then, consisted of these steps:
2979
2980
@enumerate
2981
@item
2982
@cindex Granlund, Torbjorn
2983
@cindex C torture test
2984
Copying all the C code into the test directory. These tests were based on
2985
the C-torture test created by Torbjorn Granlund (on behalf of the Free
2986
Software Foundation) for @sc{gcc} development.
2987
2988
@item
2989
Writing (and debugging) the generic @code{expect} procedures for
2990
compilation.
2991
2992
@item
2993
Writing the simple test driver: its main task is to search the directory
2994
(using the Tcl procedure @code{glob} for filename expansion with
2995
wildcards) and call a Tcl procedure with each filename. It also checks
2996
for a few errors from the testing procedure.
2997
@end enumerate
2998
2999
Testing interactive programs is intrinsically more complex. Tests for most
3000
interactive programs require some trial and error before they are complete.
3001
3002
However, some interactive programs can be tested in a simple fashion
3003
reminiscent of batch tests. For example, prior to the creation of
3004
DejaGnu, the @sc{gdb} distribution already included a wide-ranging
3005
testing procedure. This procedure was very robust, and had already
3006
undergone much more debugging and error checking than many recent
3007
DejaGnu test cases. Accordingly, the best approach was simply to
3008
encapsulate the existing @sc{gdb} tests, for reporting purposes.
3009
Thereafter, new @sc{gdb} tests built up a family of @code{expect}
3010
procedures specialized for @sc{gdb} testing.
3011
3012
@file{gdb.t10/crossload.exp} is a good example of an interactive test.
3013
@c FIXME! Check what *kind* of example it is---work-intensive, or generic...
3014
3015
@node Debugging
3016
@section Debugging a test case
3017
@cindex debugging a test case
3018
@cindex test case, debugging
3019
3020
@noindent
3021
These are the kinds of debugging information available from DejaGnu:
3022
3023
@enumerate
3024
@item
3025
Output controlled by test scripts themselves, explicitly allowed for by
3026
the test author. This kind of debugging output appears in the detailed
3027
output recorded in the @file{@var{tool}.log} file. To do the same for
3028
new tests, use the @code{verbose} procedure (which in turn uses the
3029
variable also called @code{verbose}) to control how much output to
3030
generate. This will make it easier for other people running the test to
3031
debug it if necessary. Whenever possible, if @samp{$verbose} is
3032
@code{0}, there should be no output other than the output from
3033
@code{pass}, @code{fail}, @code{error}, and @code{warning}. Then, to
3034
whatever extent is appropriate for the particular test, allow
3035
successively higher values of @samp{$verbose} to generate more
3036
information. Be kind to other programmers who use your tests: provide
3037
for a lot of debugging information.
3038
3039
@item
3040
Output from the internal debugging functions of Tcl and @code{expect}.
3041
There is a command line options for each; both forms of debugging output
3042
are recorded in the file @code{dbg.log} in the current directory.
3043
3044
Use @samp{--debug} for information from the @code{expect} level; it
3045
generates displays of the @code{expect} attempts to match the tool
3046
output with the patterns specified (@pxref{Debug,,Debug Log}). This
3047
output can be very helpful while developing test scripts, since it shows
3048
precisely the characters received. Iterating between the latest attempt
3049
at a new test script and the corresponding @file{dbg.log} can allow you
3050
to create the final patterns by ``cut and paste''. This is sometimes
3051
the best way to write a test case.
3052
3053
Use @samp{--strace} to see more detail at the Tcl level; this shows how Tcl
3054
procedure definitions expand, as they execute. The associated number
3055
controls the depth of definitions expanded; see the discussion of
3056
@samp{--strace} in @ref{Invoking runtest,,Running the Tests}.
3057
3058
@item
3059
Finally, if the value of @samp{verbose} is 3 or greater, @code{runtest}
3060
turns on the @code{expect} command @code{log_user}. This command prints
3061
all @code{expect} actions to the @code{expect} standard output, to the
3062
detailed log file, and (if @samp{--debug} is on) to @file{dbg.log}.
3063
@end enumerate
3064
3065
@node Adding
3066
@section Adding a test case to a test suite
3067
@cindex adding a test case
3068
3069
There are two slightly different ways to add a test case. One is to add
3070
the test case to an existing directory. The other is to create a new
3071
directory to hold your test. The existing test directories represent
3072
several styles of testing, all of which are slightly different; examine
3073
the directories for the tool of interest to see which (if any) is most
3074
suitable.
3075
3076
Adding a @sc{gcc} test can be very simple: just add the C code to any
3077
directory beginning with @samp{gcc.} and it runs on the next
3078
@samp{runtest --tool gcc}.
3079
3080
To add a test to @sc{gdb}, first add any source code you will need to
3081
the test directory. Then you can either create a new @code{expect} file,
3082
or add your test to an existing one (any file with a @samp{.exp}
3083
suffix). Creating a new @samp{.exp} file is probably a better idea if
3084
the test is significantly different from existing tests. Adding it as a
3085
separate file also makes upgrading easier. If the C code has to be
3086
already compiled before the test will run, then you'll have to add it to
3087
the @file{Makefile.in} file for that test directory, then run
3088
@code{configure} and @code{make}.
3089
3090
Adding a test by creating a new directory is very similar:
3091
3092
@enumerate
3093
@item
3094
Create the new directory. All subdirectory names begin with the name of
3095
the tool to test; e.g. @sc{g++} tests might be in a directory called
3096
@file{g++.other}. There can be multiple test directories that start with
3097
the same tool name (such as @samp{g++}).
3098
3099
@item
3100
Add the new directory name to the @samp{configdirs} definition in the
3101
@file{configure.in} file for the test suite directory. This way when
3102
@code{make} and @code{configure} next run, they include the new directory.
3103
3104
@item
3105
Add the new test case to the directory, as above.
3106
3107
@item
3108
To add support in the new directory for configure and make, you must
3109
also create a @code{Makefile.in} and a @code{configure.in}. @xref{What
3110
Configure Does,,What Configure Does, configure.info, Cygnus Configure}.
3111
@end enumerate
3112
3113
@c FIXME! Expand this sentence to at least a section, maybe a chapter...
3114
@c The @file{admin} directory contains templates for a few common forms
3115
@c of test.
3116
3117
@node Hints
3118
@section Hints on writing a test case
3119
@cindex hints on test case writing
3120
3121
There may be useful existing procedures already written for your test in
3122
the @file{lib} directory of the DejaGnu distribution. @xref{DejaGnu
3123
Builtins,,DejaGnu Builtins}.
3124
3125
It is safest to write patterns that match @emph{all} the output
3126
generated by the tested program; this is called @dfn{closure}. If a
3127
pattern does not match the entire output, any output that remains will
3128
be examined by the @emph{next} @code{expect} command. In this
3129
situation, the precise boundary that determines which @code{expect}
3130
command sees what is very sensitive to timing between the @code{expect}
3131
task and the task running the tested tool. As a result, the test may
3132
sometimes appear to work, but is likely to have unpredictable results.
3133
(This problem is particularly likely for interactive tools, but can also
3134
affect batch tools---especially for tests that take a long time to finish.)
3135
The best way to ensure closure is to use the @samp{-re} option for the
3136
@code{expect} command to write the pattern as a full regular
3137
expressions; then you can match the end of output using a @samp{$}. It
3138
is also a good idea to write patterns that match all available output by
3139
using @samp{.*\} after the text of interest; this will also match any
3140
intervening blank lines. Sometimes an alternative is to match end of
3141
line using @samp{\r} or @samp{\n}, but this is usually too dependent on
3142
terminal settings.
3143
@c FIXME!! explain what "end of output" means for interactive task.
3144
@c (Timeout or EOF, right?)
3145
3146
Always escape punctuation, such as @samp{(} or @samp{"}, in your
3147
patterns; for example, write @samp{\(}. If you forget to escape
3148
punctuation, you will usually see an error message like @samp{extra
3149
characters after close-quote}.
3150
3151
If you have trouble understanding why a pattern does not match the
3152
program output, try using the @samp{--debug} option to @code{runtest},
3153
and examine the debug log carefully. @xref{Debug,,Debug Log}.
3154
3155
Be careful not to neglect output generated by setup rather than by the
3156
interesting parts of a test case. For example, while testing @sc{gdb},
3157
I issue a send @samp{set height 0\n} command. The purpose is simply to
3158
make sure @sc{gdb} never calls a paging program. The @samp{set height}
3159
command in @sc{gdb} does not generate any output; but running @emph{any}
3160
command makes @sc{gdb} issue a new @samp{(gdb) } prompt. If there were
3161
no @code{expect} command to match this prompt, the output @samp{(gdb) }
3162
begins the text seen by the next @code{expect} command---which might
3163
make @emph{that} pattern fail to match.
3164
3165
To preserve basic sanity, I also recommended that no test ever pass if
3166
there was any kind of problem in the test case. To take an extreme
3167
case, tests that pass even when the tool will not spawn are misleading.
3168
Ideally, a test in this sort of situation should not fail either.
3169
Instead, print an error message by calling one of the DejaGnu procedures
3170
@code{error} or @code{warning}.
3171
3172
@node Variables
3173
@section Special variables used by test cases
3174
@cindex special variables
3175
3176
@cindex variables for all tests
3177
Your test cases can use these variables, with conventional meanings (as
3178
well as the variables saved in @file{site.exp}
3179
@pxref{Customizing,,Setting @code{runtest} defaults}):
3180
3181
@quotation
3182
@emph{These variables are available to all test cases.}
3183
@end quotation
3184
3185
@ftable @code
3186
@item prms_id
3187
@cindex PRMS bug number
3188
@cindex GNATS bug number
3189
@cindex bug number
3190
The tracking system (e.g. @sc{gnats}) number identifying a corresponding
3191
bugreport. (@samp{0} if you do not specify it in the test script.)
3192
3193
@item bug_id
3194
@cindex bug number, extra
3195
An optional bug id; may reflect a bug identification from another
3196
organization. (@samp{0} if you do not specify it.)
3197
3198
@item subdir
3199
@cindex current test subdirectory
3200
The subdirectory for the current test case.
3201
@end ftable
3202
3203
@quotation
3204
@emph{These variables should never be changed. They appear in most
3205
tests.}
3206
@end quotation
3207
3208
@ftable @code
3209
@item expect_out(buffer)
3210
@cindex last command output
3211
The output from the last command. This is an internal variable set by
3212
@code{expect}.
3213
3214
@item exec_output
3215
This is the output from a @code{@var{tool}_load} command. This only
3216
applies to tools like @sc{gcc} and @sc{gas} which produce an object
3217
file that must in turn be executed to complete a test.
3218
3219
@item comp_output
3220
This is the output from a @code{@var{tool}_start} command. This is
3221
conventionally used for batch oriented programs, like @sc{gcc} and
3222
@sc{gas}, that may produce interesting output (warnings, errors) without
3223
further interaction.
3224
@end ftable
3225
3226
@node Extending
3227
@chapter New Tools, Targets, or Hosts
3228
3229
The most common ways to extend the DejaGnu framework are: adding a suite
3230
of tests for a new tool to be tested; adding support for testing on a
3231
new target; and porting @code{runtest} to a new host.
3232
3233
@menu
3234
* Adding Tools:: How to add tests for a new tool
3235
* Adding Targets:: How to add a new target
3236
* Porting:: Porting DejaGnu to a new host
3237
@end menu
3238
3239
@node Adding Tools
3240
@section Writing tests for a new tool
3241
3242
In general, the best way to learn how to write (code or even prose) is
3243
to read something similar. This principle applies to test cases and to
3244
test suites. Unfortunately, well-established test suites have a way of
3245
developing their own conventions: as test writers become more
3246
experienced with DejaGnu and with Tcl, they accumulate more utilities,
3247
and take advantage of more and more features of @code{expect} and Tcl in
3248
general.
3249
3250
Inspecting such established test suites may make the prospect of
3251
creating an entirely new test suite appear overwhelming. Nevertheless,
3252
it is quite straightforward to get a new test suite going.
3253
3254
@cindex Lupton, Robert
3255
There is one test suite that is guaranteed not to grow more elaborate
3256
over time: both it and the tool it tests were created expressly to
3257
illustrate what it takes to get started with DejaGnu. The
3258
@file{example/} directory of the DejaGnu distribution contains both an
3259
interactive tool called @code{calc}, and a test suite for it. Reading
3260
this test suite, and experimenting with it, is a good way to supplement
3261
the information in this section. (Thanks to Robert Lupton for creating
3262
@code{calc} and its test suite---and also the first version of this
3263
section of the manual!)
3264
3265
To help orient you further in this task, here is an outline of the steps
3266
to begin building a test suite for a program @var{example}.
3267
3268
@enumerate
3269
@item
3270
Create or select a directory to contain your new collection of tests.
3271
Change to that directory (shown here as @code{testsuite}):
3272
3273
@example
3274
eg$ cd testsuite/
3275
@end example
3276
3277
@item
3278
Create a @file{configure.in} file in this directory, to control
3279
configuration-dependent choices for your tests. So far as DejaGnu is
3280
concerned, the important thing is to set a value for the variable
3281
@code{target_abbrev}; this value is the link to the init file you will
3282
write soon. (For simplicity, we assume the environment is Unix, and use
3283
@samp{unix} as the value.)
3284
3285
What else is needed in @file{configure.in} depends on the requirements
3286
of your tool, your intended test environments, and which
3287
@code{configure} system you use. This example is a minimal
3288
@code{configure.in} for use with Cygnus Configure. (For an alternative
3289
based on the FSF @code{autoconf} system, see the @code{calc} example
3290
distributed with DejaGnu.) Replace @var{example} with the name of your
3291
program:
3292
3293
@cartouche
3294
@smallexample
3295
# This file is a shell script fragment
3296
# for use with Cygnus configure.
3297
3298
srctrigger="@var{example}.0"
3299
srcname="The DejaGnu @var{example} tests"
3300
3301
# per-host:
3302
3303
# per-target:
3304
3305
# everything defaults to unix for a target
3306
target_abbrev=unix
3307
3308
# post-target:
3309
3310
@end smallexample
3311
@end cartouche
3312
3313
@item
3314
Create @file{Makefile.in}, the source file used by @code{configure} to
3315
build your @file{Makefile}. Its leading section should as usual contain
3316
the values that @code{configure} may override:
3317
3318
@cartouche
3319
@smallexample
3320
srcdir = .
3321
prefix = /usr/local
3322
3323
exec_prefix = $(prefix)
3324
bindir = $(exec_prefix)/bin
3325
libdir = $(exec_prefix)/lib
3326
tooldir = $(libdir)/$(target_alias)
3327
3328
datadir = $(exec_prefix)/lib/dejagnu
3329
3330
RUNTEST = runtest
3331
RUNTESTFLAGS =
3332
FLAGS_TO_PASS =
3333
3334
#### host, target, site specific Makefile frags come in here.
3335
@end smallexample
3336
@end cartouche
3337
3338
This should be followed by the standard targets at your site. To begin
3339
with, they need not do anything---for example, these definitions will
3340
do:
3341
3342
@cartouche
3343
@smallexample
3344
3345
all:
3346
3347
info:
3348
3349
install-info:
3350
3351
install:
3352
uninstall:
3353
3354
clean:
3355
-rm -f *~ core *.info*
3356
3357
@end smallexample
3358
@end cartouche
3359
3360
It is also a good idea to make sure your @file{Makefile} can rebuild
3361
itself if @file{Makefile.in} changes, with a target like this (which
3362
works for either Cygnus or FSF Configure):
3363
3364
@cartouche
3365
@smallexample
3366
Makefile : $(srcdir)/Makefile.in $(host_makefile_frag) \
3367
$(target_makefile_frag)
3368
$(SHELL) ./config.status
3369
@end smallexample
3370
@end cartouche
3371
3372
You also need to include two targets important to DejaGnu: @code{check},
3373
to run the tests, and @code{site.exp}, to set up the Tcl copies of
3374
configuration-dependent values. The @code{check} target must run
3375
@samp{runtest --tool @var{example}}:
3376
3377
@cartouche
3378
@smallexample
3379
check: site.exp all
3380
$(RUNTEST) $(RUNTESTFLAGS) $(FLAGS_TO_PASS) \
3381
--tool @var{example} --srcdir $(srcdir)
3382
@end smallexample
3383
@end cartouche
3384
3385
The @code{site.exp} target should usually set up (among other things!) a
3386
Tcl variable for the name of your program:
3387
3388
@cartouche
3389
@smallexample
3390
site.exp: ./config.status Makefile
3391
@@echo "Making a new config file..."
3392
-@@rm -f ./tmp?
3393
@@touch site.exp
3394
3395
-@@mv site.exp site.bak
3396
@@echo "## these variables are automatically\
3397
generated by make ##" > ./tmp0
3398
@@echo "# Do not edit here. If you wish to\
3399
override these values" >> ./tmp0
3400
@@echo "# add them to the last section" >> ./tmp0
3401
@@echo "set host_os $@{host_os@}" >> ./tmp0
3402
@@echo "set host_alias $@{host_alias@}" >> ./tmp0
3403
@@echo "set host_cpu $@{host_cpu@}" >> ./tmp0
3404
@@echo "set host_vendor $@{host_vendor@}" >> ./tmp0
3405
@@echo "set target_os $@{target_os@}" >> ./tmp0
3406
@@echo "set target_alias $@{target_alias@}" >> ./tmp0
3407
@@echo "set target_cpu $@{target_cpu@}" >> ./tmp0
3408
@@echo "set target_vendor $@{target_vendor@}" >> ./tmp0
3409
@@echo "set host_triplet $@{host_canonical@}" >> ./tmp0
3410
@@echo "set target_triplet $@{target_canonical@}">>./tmp0
3411
@@echo "set tool binutils" >> ./tmp0
3412
@@echo "set srcdir $@{srcdir@}" >> ./tmp0
3413
@@echo "set objdir `pwd`" >> ./tmp0
3414
@@echo "set @var{examplename} @var{example}" >> ./tmp0
3415
@@echo "## All variables above are generated by\
3416
configure. Do Not Edit ##" >> ./tmp0
3417
@@cat ./tmp0 > site.exp
3418
@@sed < site.bak \
3419
-e '1,/^## All variables above are.*##/ d' \
3420
>> site.exp
3421
-@@rm -f ./tmp?
3422
@end smallexample
3423
@end cartouche
3424
3425
@item
3426
Create a directory (in @file{testsuite/}) called @file{config/}:
3427
3428
@example
3429
eg$ mkdir config
3430
@end example
3431
3432
@item
3433
Make an init file in this directory; its name must start with the
3434
@code{target_abbrev} value, so call it @file{config/unix.exp}.
3435
This is the file that contains the target-dependent procedures;
3436
fortunately, most of them do not have to do very much in order for
3437
@code{runtest} to run.
3438
3439
If @var{example} is not interactive, you can get away with this minimal
3440
@file{unix.exp} to begin with:
3441
3442
@cartouche
3443
@smallexample
3444
proc foo_exit @{@} @{@}
3445
proc foo_version @{@} @{@}
3446
@end smallexample
3447
@end cartouche
3448
3449
If @var{example} is interactive, however, you might as well define a
3450
start routine @emph{and invoke it} by using an init file like this:
3451
3452
@cartouche
3453
@smallexample
3454
proc foo_exit @{@} @{@}
3455
proc foo_version @{@} @{@}
3456
3457
proc foo_start @{@} @{
3458
global @var{examplename}
3459
spawn $@var{examplename}
3460
expect @{
3461
-re "" @{@}
3462
@}
3463
@}
3464
foo_start
3465
@end smallexample
3466
@end cartouche
3467
3468
@item
3469
Create a directory whose name begins with your tool's name, to contain
3470
tests:
3471
3472
@example
3473
eg$ mkdir @var{example}.0
3474
@end example
3475
3476
@item
3477
Create a sample test file in @file{@var{example}.0}. Its name must end
3478
with @samp{.exp}; you can use @samp{first-try.exp} To begin with, just
3479
write there a line of Tcl code to issue a message:
3480
3481
@cartouche
3482
@smallexample
3483
send_user "Testing: one, two...\n"
3484
@end smallexample
3485
@end cartouche
3486
3487
@item
3488
Back in the @file{testsuite/} (top level) directory, run
3489
3490
@example
3491
eg$ configure
3492
@end example
3493
3494
(You may have to specify more of a path, if a suitable @code{configure}
3495
is not available in your execution path.)
3496
3497
@item
3498
You are now ready to triumphantly type @samp{make check} or
3499
@samp{runtest --tool @var{example}}. You should see something like this:
3500
3501
@cartouche
3502
@smallexample
3503
Test Run By rhl on Fri Jan 29 16:25:44 EST 1993
3504
3505
=== @var{example} tests ===
3506
3507
Running ./@var{example}.0/first-try.exp ...
3508
Testing: one, two...
3509
3510
=== @var{example} Summary ===
3511
3512
@end smallexample
3513
@end cartouche
3514
3515
There is no output in the summary, because so far the example does not
3516
call any of the procedures that establish a test outcome.
3517
3518
@item
3519
Begin writing some real tests. For an interactive tool, you should
3520
probably write a real exit routine in fairly short order; in any case,
3521
you should also write a real version routine soon.
3522
@end enumerate
3523
3524
@node Adding Targets
3525
@section Adding a target
3526
@cindex adding a target
3527
3528
DejaGnu has some additional requirements for target support, beyond the
3529
general-purpose provisions of Cygnus @code{configure}. @code{runtest}
3530
must actively communicate with the target, rather than simply generating
3531
or managing code for the target architecture. Therefore, each tool
3532
requires an initialization module for each target. For new targets, you
3533
must supply a few Tcl procedures to adapt DejaGnu to the target. This
3534
permits DejaGnu itself to remain target independent. @xref{Init
3535
Module,,Initialization module}, for a discussion of the naming
3536
conventions that enable DejaGnu to locate and use init files.
3537
3538
Usually the best way to write a new initialization module is to edit an
3539
existing initialization module; some trial and error will be required.
3540
If necessary, you can use the @samp{--debug} option to see what
3541
is really going on.
3542
3543
When you code an initialization module, be generous in printing
3544
information controlled by the @code{verbose} procedure (@pxref{DejaGnu
3545
Builtins, DejaGnu procedures}).
3546
3547
Most of the work is in getting the communications right. Communications
3548
code (for several situations involving IP networks or serial lines) is
3549
available in a DejaGnu library file, @file{lib/remote.exp}.
3550
@xref{DejaGnu Builtins,,DejaGnu Builtins}.
3551
3552
@c FIXME! Say something about Tcl debugger here.
3553
If you suspect a communication problem, try running the connection
3554
interactively from @code{expect}. (There are three ways of running
3555
@code{expect} as an interactive interpreter. You can run @code{expect}
3556
with no arguments, and control it completely interactively; or you can
3557
use @samp{expect -i} together with other command-line options and
3558
arguments; or you can run the command @code{interpreter} from any
3559
@code{expect} procedure. Use @code{return} to get back to the calling
3560
procedure (if any), or @code{return -tcl} to make the calling procedure
3561
itself return to its caller; use @code{exit} or end-of-file to leave
3562
@code{expect} altogether.) Run the program whose name is recorded in
3563
@samp{$connectmode}, with the arguments in @samp{$targetname}, to
3564
establish a connection. You should at least be able to get a prompt
3565
from any target that is physically connected.
3566
3567
@node Porting
3568
@section Porting to a new host
3569
@cindex porting to a new host
3570
3571
The task of porting DejaGnu is basically that of porting Tcl and
3572
@code{expect}. Tcl and @code{expect}, as distributed with DejaGnu, both
3573
use @code{autoconf}; they should port automatically to most Unix
3574
systems.
3575
3576
Once Tcl and @code{expect} are ported, DejaGnu should run. Most system
3577
dependencies are taken care of by using @code{expect} as the main
3578
command shell.
3579
3580
@node Installation
3581
@appendix Installing DejaGnu
3582
3583
@cindex host, explained
3584
@cindex target, explained
3585
@cindex DejaGnu configuration
3586
@cindex configuring DejaGnu
3587
Once you have the DejaGnu source unpacked and available, you must first
3588
configure the software to specify where it is to run (and the associated
3589
defaults); then you can proceed to installing it.
3590
3591
@menu
3592
* Configuring DejaGnu::
3593
* Installing DejaGnu::
3594
@end menu
3595
3596
@node Configuring DejaGnu
3597
@section Configuring the DejaGnu test driver
3598
3599
It is usually best to configure in a directory separate
3600
from the source tree, specifying where to find the source with the
3601
optional @samp{--srcdir} option to @code{configure}. DejaGnu uses the
3602
GNU @code{autoconf} to configure itself. For more info on using
3603
autoconf, read the GNU autoconf manual. To configure, execute the
3604
@file{configure} program, no other options are required. For an example,
3605
to configure in a seperate tree for objects, execute the configure
3606
script from the source tree like this:
3607
3608
@smallexample
3609
../dejagnu-1.3/configure
3610
@end smallexample
3611
3612
DejaGnu doesn't care at config time if it's for testing a native system
3613
or a cross system. That is determined at runtime by using the config
3614
files.
3615
3616
@cindex @code{prefix}, configure options
3617
@cindex @code{exec_prefix}, configure options.
3618
You may also want to use the @code{configure} option @samp{--prefix} to
3619
specify where you want DejaGnu and its supporting code installed. By
3620
default, installation is in subdirectories of @file{/usr/local}, but you
3621
can select any alternate directory @var{altdir} by including
3622
@samp{--prefix=@var{altdir}} on the @code{configure} command line.
3623
(This value is captured in the Makefile variables @code{prefix}
3624
and @code{exec_prefix}.)
3625
3626
@cindex auxiliary programs
3627
@cindex test suite distributions
3628
@cindex @code{make} builds part of tests
3629
Save for a small number of example tests, the DejaGnu distribution
3630
itself does not include any test suites; these are available separately.
3631
Test suites for the @sc{gnu} compiler (testing both GCC and G++) and for
3632
the @sc{gnu} binary utilities are distributed in parallel with the
3633
DejaGnu distribution (but packaged as separate files). The test suite
3634
for the @sc{gnu} debugger is distributed in parallel with each release
3635
of GDB itself, starting with GDB 4.9. After configuring the top-level
3636
DejaGnu directory, unpack and configure the test directories for the
3637
tools you want to test; then, in each test directory, run @code{make} to
3638
build auxiliary programs required by some of the tests.
3639
3640
@node Installing DejaGnu
3641
@section Installing DejaGnu
3642
3643
@cindex installing DejaGnu
3644
To install DejaGnu in your filesystem (either in @file{/usr/local}, or
3645
as specified by your @samp{--prefix} option to @code{configure}), execute
3646
3647
@example
3648
eg$ make install
3649
@end example
3650
3651
@noindent
3652
@samp{make install} does these things for DejaGnu:
3653
3654
@enumerate
3655
@item
3656
Look in the path specified for executables (@file{$exec_prefix}) for
3657
directories called @file{lib} and @file{bin}. If these directories do
3658
not exist, @samp{make install} creates them.
3659
3660
@item
3661
Create another directory in the @file{lib} directory, called
3662
@file{dejagnu}.
3663
3664
@item
3665
Copy the @code{runtest} shell script into @file{$exec_prefix/bin}.
3666
3667
@item
3668
Copy all the library files (used to support the framework) into
3669
@file{$exec_prefix/lib/dejagnu}.
3670
3671
@item
3672
Copy @file{runtest.exp} into @file{$exec_prefix/lib/dejagnu}. This is
3673
the main Tcl code implementing DejaGnu.
3674
3675
@end enumerate
3676
3677
Each test suite collection comes with simple installation instructions
3678
in a @file{README} file; in general, the test suites are designed to be
3679
unpacked in the source directory for the corresponding tool, and extract
3680
into a directory called @file{testsuite}.
3681
3682
@node Index
3683
@unnumbered Index
3684
3685
@printindex cp
3686
3687
@contents
3688
3689
@bye
Loggerhead is a web-based interface for Breezy
Version: 2.0.1