2
Understanding the runtest script in DejaGnu
6
philip.wilsey@cliftonlabs.com
12
DejaGnu is a tcl program to help run test scripts for batch and
13
interactive programs. If you have not yet read the DejaGnu
14
documentation don't, read this document first and play with this
15
example. If you have, you have my sympathies. Hopefully this document
16
and example will provide a high level overview of what goes on in
17
runtest. You will still have to read the DejaGnu documentation. It
18
contains many details of the low level functions and operation that I do
19
not address in this document.
21
FIRST: I AM NOT A DEJAGNU OR TCL EXPERT. I have simply spent the past
22
week studying and trying to understand the execution behavior of the
23
runtest script. As part of my studies I built the accompanying example
24
that you may (or may not) find helpful in understanding just what is
25
going on with the runtest script. Fortunately tcl is a pretty simply
26
language and a quick scan of Ousterhout's text on tcl has been more than
27
sufficient for me to conduct these experiments. Again, this document
28
only records my observations.
30
SECOND: I am not interested in or concerned with testing
31
cross-compilation systems or anything exotic like remote execution. I
32
was/am simply interested in using a test harness in a general purpose
33
conformance suite. Consequently I have not examined all the switches
34
and variable related to -host_board, -target, or remote execution. I
35
was simply trying to gain an overall understanding of what and when the
36
various tcl/expect files and procedures are executed.
38
THIRD: There are really two parts to my studies, specifically: (i) the
39
study of runtest, and (ii) the use of the auto- tools to realize a
40
"make check" target that will invoke the runtest script for you. As a
41
result, this document is organized into three parts:
43
Part I: An overview of runtest
44
Part II: Building a "make check" target with automake
45
Part III: Running the example
47
For small projects, you may not be interested in the autoconf/automake
48
tools and can safely skip Part II of this document.
50
FOURTH: This example is setup only for recording the files and
51
procedures that runtest uses. Technically it will compile a simple c++
52
hello world program, however the test system does not verify it's
53
operation. If you run make check, it will simply dump out a bunch of
54
DEBUG statements and exit. Please don't expect anything more.
56
FINALLY: For many years now we have been managing our regression suite
57
using hand rolled perl scripts that require continual maintenance. We
58
turned to DejaGnu in hopes to reduce some of our efforts in this
59
direction. Unfortunately, I found understanding DejaGnu to be very
60
painful and at several times during my study I was ready to chuck it
61
all. Now that I have a better understanding of the system, I believe
62
that in the long run I will be much happier that I stuck with it.
63
Hopefully you will think so as well.
65
NOTE: I have only used/studied runtest on a Debian Linux box. I do not
66
know how well this knowledge will generalize or be useful to you.
67
Please use at your own risk.
71
PART I: AN OVERVIEW OF RUNTEST
72
------------------------------
74
I.1 The essence of Runtest
75
--------------------------
77
runtest is designed to be run from a test subdirectory where all tests
78
are stored in subdirectories with a naming convention of TOOL.TESTNAME.
79
Subdirectories not conforming to the TOOL.TESTNAME naming convention are
80
not examined. runtest examines recursively all the subdirectories under
81
each TOOL.TESTNAME subdirectories for expect scripts to be executed (it
82
assumes that everything with the ".exp" suffix is an expect script). It
83
will then attempt to execute every expect script it finds. For example,
84
let's take a look at the contents of the helloworld.* subdirectories in
85
the testsuite subdirectory of this example:
87
peabody>ls -R testsuite/helloworld.test*
88
~/test/dejagnu/helloDemo
89
testsuite/helloworld.test1:
91
4 helloworld.test1-1/ 4 test1.exp
93
testsuite/helloworld.test1/helloworld.test1-1:
97
testsuite/helloworld.test2:
99
4 test2.exp 4 test3.exp
101
testsuite/helloworld.test3:
103
4 config/ 4 lib/ 4 non_compliant_dir_name/
105
testsuite/helloworld.test3/config:
109
testsuite/helloworld.test3/lib:
113
testsuite/helloworld.test3/non_compliant_dir_name:
118
Once everything is setup, issuing the command "runtest" from the
119
testsuite subdirectory on my machine causes all of the expect scripts to
120
be executed in this order:
122
Running ./helloworld.test1/helloworld.test1-1/test1-1.exp ...
123
Running ./helloworld.test1/test1.exp ...
124
Running ./helloworld.test2/test2.exp ...
125
Running ./helloworld.test2/test3.exp ...
126
Running ./helloworld.test3/non_compliant_dir_name/non_compliant.exp ...
128
Looks like a depth first search; however, I can find nothing in the
129
documentation that indicates that a specific order will be followed. So
130
you should not count on it. Also notice that the TOOL.TESTNAME naming
131
convention is only enforced at the root directory from where the runtest
132
command is issued -- beneath the first level subdirectories all
133
subdirectories except those named "lib" and "config" are examined for
136
I.2 The nuts and bolts of Runtest
137
---------------------------------
139
A few more points. runtest is setup to look for scripts to load for
140
configuration information and runtime procedures. In many cases if
141
runtest cannot find a script it will continue. Four important tcl
142
procedures that you should define (in testsuite/lib/TOOL.exp) are:
149
TOOL_init is called prior to running each expect script it locates under
150
the testsuite subdirectory. runtest invokes TOOL_finish after executing
151
each expect script. TOOL_exit is invoked after TOOL_finish for the last
152
expect script executes. Finally TOOL_version is invoked to allow the
153
testsute to report the version of the tool that was just tested.
155
If you want to add tool specific arguments on the on the runtest command
156
line you can also define:
161
TOOL_option_proc is invoked as runtest parses the command line
162
arguments. TOOL_option_help is invoked when runtest is invoked with the
163
"-help" command line argument (this is actually pretty cool).
165
In general runtest will take the following steps (the expect scripts
166
named in these steps are named relative to the directory where the
167
runtest command is issued):
169
1. It will look for and load the file ~/.dejagnurc for command line
170
options. I have not experimented with this and cannot comment on
171
what can and cannot be placed in this file.
172
2. It will try to load load the file ./site.exp (if you use automake,
173
this file will be automagically created for you). Technically this
174
file is organized into two parts, one that is set when the .configure
175
script is executed; the second part can be edited and changed by the
177
3. It will try to load the file ./lib/TOOL.exp (this is probably a good
178
place to locate definitions for the TOOL_init/TOOL_exit/TOOL_version
180
4. It will try to load some configuration files. Specifically
181
4a. It will try to load a base-config.exp file. On my system it
182
searches, in order, the following subdirectories:
187
4b. It will try to load a system specific configuration file. I believe
188
you can/should use this to setup different configuration information
189
for various operating systems that the test suite is to run in. On
190
my system it searches, in order, for (the search is terminated after
198
./../config/default.exp
199
./../config/unknown.exp
200
./../../config/unix.exp
201
./../../config/gnu.exp
202
./../../config/default.exp
203
./../../config/unknown.exp
204
./../../../config/unix.exp
205
./../../../config/gnu.exp
206
./../../../config/default.exp
207
./../../../config/unknown.exp
208
5. Now it will recursively search the ./TOOL.* subdirectories and for
209
each expect script it locates it will:
210
5a. Invoke the procedure TOOL_init giving the relative pathname where
211
the expect script is found as an argument.
212
5b. Run the expect script.
213
5c. Invoke the proecture TOOL_finish with no arguments.
214
6. After running all of the located expect script it will invoke the
216
7. Next the TOOL_version procedure is invoked.
217
8. Finally, runtest exits.
219
That's it. Sadly it took me almost a week to learn this....
222
I.3 Some notes I made during my week of study
223
---------------------------------------------
225
1. site.exp is loaded twice. Why? I have no clue.
227
2. As far as I can tell the "testsuite" name is completely artificial.
228
runtest does not appear to depend on it. The autoconf/automake
229
family also do not seem to depend on it.
231
3. Searching for files to load: There is a method to this madness.
232
However, it's not very regular: for example why does 4a not look into
233
the testsuite subdirectory and 4b does.
235
3a. Look into srcdir (where the runtest command was issued) for
236
lib/tool.exp; I guess the principle concept is to put tool specific
237
and platform independent expect scripts here.
239
3b. Next look for for platform scripts in --srcdir/config (on my linux
240
system it looks for one of unix.exp, gnu.exp, and default.exp (in
241
this order and stops with the first one it finds).
243
My confusion on the search patterns of runtest is magnified by the
244
fact that I am trying to issue the runtest command from outside the
245
testsuite subdirectory....don't do this.
247
4. Executing expect scripts.
249
4a. If an expect script is named on the command line it (and only it) is
252
4b. runtest looks in the testsuite directory and does a glob on
253
TOOL.*/*.exp (ok, it's a recursive glob TOOL.*/.../*.exp) and
254
executes all expect script it sees (ok, it appears to ignore all
255
subdirectories named "lib" or "config"). Before execution of each
256
script, runtest invokes TOOL_init with the relative path (relative
257
to where the directory where the runtest command was issued) name of
258
that script as it's only argument. After execution of each script
259
runtest invokes TOOL_finish with no arguments.
261
5. load_lib: load_lib is a tcl script in the DejaGnu system that you can
262
use to load tcl/expect files. It has an odd search path. In
263
particular, on my machine, the search path is:
265
5a. the dejagnu install directory (/usr/share/dejagnu) and its
266
accompanying library directory (/usr/share/dejagnu/lib)
267
5b. in a dejagnu/lib directory one above the current directory (where
268
runtest was run). that is in ../dejagnu/lib
269
5c. in the --srcdir lib
270
5d. in the directory where runtest was issued
271
5e and this is the funny one, in a dejagnu/lib directory two directories
272
above (../../dejagnu/lib).
274
6. hmmmm how does DEJAGNULIBS alter the library search path?? I did not
277
7. COOL: I can name an expect script in the --ignore command line option
278
to prune it from the set of scripts that are executed.
284
1. As far as I can tell the "testsuite" name is completely artificial
285
and you can store your tests in whatever subdirectory name you like.
287
2. Don't try to issue the runtest command from outside the testsuite
288
subdirectory using the "--srcdir target" command line argument.
289
Unless you really know what's going on, it will cause you grief. For
290
some reason the runtest will look for some things in the $target
291
location and for others it will look in the directory where the
292
runtest command was issued.
294
3. The search paths used to look for things varies all over the places.
295
The system has tremendous configurability for building and
296
controlling the expect scripts. However the flexibility for
297
controlling what and where things are loaded by runtest is really
298
quite limited. For example, site.exp must be located in the
299
directory where the runtest command is issued. I can discover no way
300
to alter its location.
304
PART II: BUILDING a "make check" TARGET WITH AUTOMAKE
305
-----------------------------------------------------
307
NOTE: before I began this study I did not know anything about
308
autoconf/automake and friends (I still don't).
310
1. If you understand these tools it's really quite simple. In the
311
directory containing the testsuite you must have a Makefile.am to
312
build the "make check" target. You only need to define the dejagnu
313
option for AUTOMAKE_OPTIONS. If you want you can also add runtest
314
command line arguments to the RUNTESTDEFAULTFLAGS variable.
316
The only problem I had with this was realizing that the Makefile.am
317
to locate these definitions in is the one located *in* the testsuite
318
subdirectory. DO NOT make these definitions in the root Makefile.am
319
file (unless you intend to place you TOOL.* subdirectories there).
321
2. I cannot figure out a good way to add alternate check targets (e.g.,
322
make check-test1) to run alternate tests on the command line. Since
323
I really don't know automake very well, this could very likely be
324
very easy to do, but my ignorance prevents me from discovering how.
325
Presently I have simply defined the alternate target by hand (see
326
make check-demo in the testsuite/Makefile.am file).
329
PART III: RUNNING THE EXAMPLE
330
-----------------------------
332
The example contains a bunch of expect/tcl files, some of them are
333
unused (actually I don't believe I ever used anything outside of
334
standard tcl in any of my scripts). I created this example to explore
335
what was going on so you will see, for example, unix.exp in the lib,
336
testsuite/lib, and testsuite/config subdirectories. I did this to learn
337
where things were loaded from. Consequently you should *not* use my
338
setup to build a testing framework from. This system should only be
339
used to discover the runtime behaviors of runtest.
341
To build this example, you will need a system configured with a c++
342
compiler and the autoconf and automake tools installed. With these
343
tools in place, the build procedure is simply:
351
Pretty much that's it (you may or may not need the aclocal step). After
352
the ./configure, the makefile in the testsuite subdirectory has two
353
relevant targets, namely:
added
removed
example/hello/README
