6
The bash-completion package contains an automated test suite. Running the
7
tests should help verifying that bash-completion works as expected. The tests
8
are also very helpful in uncovering software regressions at an early stage.
10
The bash-completion test suite is written on top of the
11
http://www.gnu.org/software/dejagnu/[DejaGnu] testing framework. DejaGnu is
12
written in http://expect.nist.gov[Expect], which in turn uses
13
http://tcl.sourceforge.net[Tcl] -- Tool command language.
19
The bash-completion test suite tries to adhere to this
20
http://wiki.tcl.tk/708[Tcl Style Guide].
23
Installing dependencies
24
-----------------------
26
Installing dependencies should be easy using your local package manager.
32
On Debian/Ubuntu you can use `apt-get`:
34
sudo apt-get install dejagnu tcllib
36
This should also install the necessary `expect` and `tcl` packages.
41
On Fedora and RHEL/CentOS (with EPEL) you can use `yum`:
43
sudo yum install dejagnu tcllib
45
This should also install the necessary `expect` and `tcl` packages.
53
Main areas (DejaGnu tools)
54
~~~~~~~~~~~~~~~~~~~~~~~~~~
56
The tests are grouped into different areas, called _tool_ in DejaGnu:
59
Functional tests per completion.
61
Functional tests for installation and caching of the main bash-completion
64
Unit tests for bash-completion helper functions.
66
Each tool has a slightly different way of loading the test fixtures, see
67
<<Test_context,Test context>> below.
73
Completion tests are spread over two directories: `completion/\*.exp` calls
74
completions in `lib/completions/\*.exp`. This two-file system stems from
75
bash-completion-lib (http://code.google.com/p/bash-completion-lib/, containing
76
dynamic loading of completions) where tests are run twice per completion; once
77
before dynamic loading and a second time after to confirm that all dynamic
78
loading has gone well.
83
set test "Completion via comp_load() should be installed"
84
set cmd "complete -p awk"
87
-re "^$cmd\r\ncomplete -o filenames -F comp_load awk\r\n/@$" { pass "$test" }
88
-re /@ { fail "$test at prompt" }
92
source "lib/completions/awk.exp"
95
set test "Completion via _longopt() should be installed"
96
set cmd "complete -p awk"
99
-re "^$cmd\r\ncomplete -o filenames -F _longopt awk\r\n/@$" { pass "$test" }
100
-re /@ { fail "$test at prompt" }
104
source "lib/completions/awk.exp"
107
Looking to the completion tests from a broader perspective, every test for a
108
command has two stages which are now reflected in the two files:
110
. Tests concerning the command completions' environment (typically in
111
`test/completion/foo`)
112
. Tests invoking actual command completion (typically in
113
`test/lib/completions/foo`)
119
The tests are run by calling `runtest` command in the test directory:
120
-----------------------
121
runtest --outdir log --tool completion
122
runtest --outdir log --tool install
123
runtest --outdir log --tool unit
124
-----------------------
125
The commands above are already wrapped up in shell scripts within the `test`
127
-----------------------
131
-----------------------
132
To run a particular test, specify file name of your test as an argument to
133
`runCompletion` script:
134
-----------------------
135
./runCompletion ssh.exp
136
-----------------------
137
That will run `test/completion/ssh.exp`.
140
Running tests via cron
141
~~~~~~~~~~~~~~~~~~~~~~
143
The test suite requires a connected terminal (tty). When invoked via cron, no
144
tty is connected and the test suite may respond with this error:
145
---------------------------------------------
146
can't read "multipass_name": no such variable
147
---------------------------------------------
149
To run the tests successfully via cron, connect a terminal by redirecting
150
stdin from a tty, e.g. /dev/tty40. (In Linux, you can press alt-Fx or
151
ctrl-alt-Fx to switch the console from /dev/tty1 to tty7. There are many more
152
/dev/tty* which are not accessed via function keys. To be safe, use a tty
155
----------------------
156
./runUnit < /dev/tty40
157
----------------------
159
If the process doesn't run as root (recommended), root will have to change the
160
owner and permissions of the tty:
161
-------------------------
162
sudo chmod o+r /dev/tty40
163
-------------------------
165
To make this permission permanent (at least on Debian) - and not revert back on
166
reboot - create the file `/etc/udev/rules.d/10-mydejagnu.rules`, containing:
167
----------------------------
168
KERNEL=="tty40", MODE="0666"
169
----------------------------
171
To start the test at 01:00, set the crontab to this:
172
----------------------------
173
* 1 * * * cd bash-completion/test && ./cron.sh < /dev/tty40
174
----------------------------
176
Here's an example batch file `cron.sh`, to be put in the bash-completion `test`
177
directory. This batch file only e-mails the output of each test-run if the
181
---------------------------------------------------------------------
184
set -e # Exit if simple command fails
185
set -u # Error if variable is undefined
188
LOG=/tmp/bash-completion.log~
190
# Retrieve latest sources
193
# Run tests on bash-4
195
./runUnit --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG
196
./runCompletion --outdir log/bash-4 --tool_exec /opt/bash-4.0/bin/bash > $LOG || cat $LOG
199
[ -f $LOG ] && rm $LOG
200
---------------------------------------------------------------------
202
Specifying bash binary
203
~~~~~~~~~~~~~~~~~~~~~~
205
The test suite standard uses `bash` as found in the tcl path (/bin/bash).
206
Using `--tool_exec` you can specify which bash binary you want to run the test
210
./runUnit --tool_exec /opt/bash-4.0/bin/bash
220
Adding a completion test
221
~~~~~~~~~~~~~~~~~~~~~~~~
223
You can run `cd test && ./generate cmd` to add a test for the `cmd` command.
224
This will add two files with a very basic tests:
225
----------------------------------
226
test/completion/cmd.exp
227
test/lib/completions/cmd.exp
228
----------------------------------
229
Place any additional tests into `test/lib/completions/cmd.exp`.
232
Fixing a completion test
233
~~~~~~~~~~~~~~~~~~~~~~~~
234
Let's consider this real-life example where an ssh completion bug is fixed.
235
First you're triggered by unsuccessful tests:
237
----------------------------------
240
=== completion Summary ===
242
# of expected passes 283
243
# of unexpected failures 8
244
# of unresolved testcases 2
245
# of unsupported tests 47
246
----------------------------------
248
Take a look in `log/completion.log` to find out which specific command is
251
-----------------------
252
$ vi log/completion.log
253
-----------------------
255
Search for `UNRESOLVED` or `FAIL`. From there scroll up to see which `.exp`
258
---------------------------------------------------------
259
/@Running ./completion/ssh.exp ...
261
UNRESOLVED: Tab should complete ssh known-hosts at prompt
262
---------------------------------------------------------
264
In this case it appears `ssh.exp` is causing the problem. Isolate the `ssh`
265
tests by specifying just `ssh.exp` to run. Furthermore add the `--debug` flag,
266
so output gets logged in `dbg.log`:
268
----------------------------------
269
$ ./runCompletion ssh.exp --debug
271
=== completion Summary ===
273
# of expected passes 1
274
# of unresolved testcases 1
275
----------------------------------
277
Now we can have a detailed look in `dbg.log` to find out what's going wrong.
278
Open `dbg.log` and search for `UNRESOLVED` (or `FAIL` if that's what you're
281
---------------------------------------------------------
282
UNRESOLVED: Tab should complete ssh known-hosts at prompt
283
---------------------------------------------------------
285
From there, search up for the first line saying:
287
-------------------------------------------------
288
expect: does "..." match regular expression "..."
289
-------------------------------------------------
291
This tells you where the actual output differs from the expected output. In
292
this case it looks like the test "ssh -F fixtures/ssh/config <TAB>" is
293
expecting just hostnames, whereas the actual completion is containing commands
295
So what should be expected after "ssh -F fixtures/ssh/config <TAB>" are *both*
296
commands and hostnames. This means both the test and the completion need
297
fixing. Let's start with the test.
299
----------------------------
300
$ vi lib/completions/ssh.exp
301
----------------------------
303
Search for the test "Tab should complete ssh known-hosts". Here you could've
304
seen that what was expected were hostnames ($hosts):
306
-----------------------------------------
307
set expected "^$cmd\r\n$hosts\r\n/@$cmd$"
308
-----------------------------------------
310
Adding *all* commands (which could well be over 2000) to 'expected', seems a
311
bit overdone so we're gonna change things here. Lets expect the unit test for
312
`_known_hosts` assures all hosts are returned. Then all we need to do here is
313
expect one host and one command, just to be kind of sure that both hosts and
314
commands are completed.
316
Looking in the fixture for ssh:
318
-----------------------------
319
$ vi fixtures/ssh/known_hosts
320
-----------------------------
322
it looks like we can add an additional host 'ls_known_host'. Now if we would
323
perform the test "ssh -F fixtures/ssh/config ls<TAB>" both the command `ls` and
324
the host `ls_known_host` should come up. Let's modify the test so:
326
--------------------------------------------------------
327
$ vi lib/completions/ssh.exp
329
set expected "^$cmd\r\n.*ls.*ls_known_host.*\r\n/@$cmd$"
330
--------------------------------------------------------
332
Running the test reveals we still have an unresolved test:
334
----------------------------------
335
$ ./runCompletion ssh.exp --debug
337
=== completion Summary ===
339
# of expected passes 1
340
# of unresolved testcases 1
341
----------------------------------
343
But if now look into the log file `dbg.log` we can see the completion only
344
returns commands starting with 'ls' but fails to match our regular expression
345
which also expects the hostname `ls_known_host':
347
-----------------------
350
expect: does "ssh -F fixtures/ssh/config ls\r\nls lsattr lsb_release lshal lshw lsmod lsof lspci lspcmcia lspgpot lss16toppm\r\nlsusb\r\n/@ssh -F fixtures/ssh/config ls" (spawn_id exp9) match regular expression "^ssh -F fixtures/ssh/config ls\r\n.*ls.*ls_known_host.*\r\n/@ssh -F fixtures/ssh/config ls$"? no
351
-----------------------
353
Now let's fix ssh completion:
360
until the test shows:
362
----------------------------------
363
$ ./runCompletion ssh.exp
365
=== completion Summary ===
367
# of expected passes 2
368
----------------------------------
372
Now let's consider a unit test failure. First you're triggered by unsuccessful
375
----------------------------------
380
# of expected passes 1
381
# of unexpected failures 1
382
----------------------------------
384
Take a look in `log/unit.log` to find out which specific command is failing.
390
Search for `UNRESOLVED` or `FAIL`. From there scroll up to see which `.exp`
393
------------------------------------------
394
/@Running ./unit/_known_hosts_real.exp ...
396
FAIL: Environment should stay clean
397
------------------------------------------
399
In this case it appears `_known_hosts_real.exp` is causing the problem.
400
Isolate the `_known_hosts_real` test by specifying just `_known_hosts_real.exp`
401
to run. Furthermore add the `--debug` flag, so output gets logged in
404
----------------------------------
405
$ ./runUnit _known_hosts_real.exp --debug
407
=== completion Summary ===
409
# of expected passes 1
410
# of unexpected failures 1
411
----------------------------------
413
Now, if we haven't already figured out the problem, we can have a detailed look
414
in `dbg.log` to find out what's going wrong. Open `dbg.log` and search for
415
`UNRESOLVED` (or `FAIL` if that's what you're looking for):
417
-----------------------------------
418
FAIL: Environment should stay clean
419
-----------------------------------
421
From there, search up for the first line saying:
423
-------------------------------------------------
424
expect: does "..." match regular expression "..."
425
-------------------------------------------------
427
This tells you where the actual output differs from the expected output. In
428
this case it looks like the the function `_known_hosts_real` is unexpectedly
429
modifying global variables `cur` and `flag`. In case you need to modify the
432
-----------------------------------
433
$ vi lib/unit/_known_hosts_real.exp
434
-----------------------------------
443
Test suite or testsuite
444
^^^^^^^^^^^^^^^^^^^^^^^
445
The primary Wikipedia page is called
446
http://en.wikipedia.org/wiki/Test_suite[test suite] and not testsuite, so
447
that's what this document sticks to.
451
The name and location of this code generation script come from Ruby on Rails'
452
http://en.wikibooks.org/wiki/Ruby_on_Rails/Tools/Generators[script/generate].
459
Within test scripts the following library functions can be used:
464
The test environment needs to be put to fixed states when testing. For
465
instance the bash prompt (PS1) is set to the current test directory, followed
466
by an at sign (@). The default settings for `bash` reside in `config/bashrc`
467
and `config/inputrc`.
469
For each tool (completion, install, unit) a slightly different context is in
472
=== What happens when tests are run?
476
When the completions are tested, invoking DejaGnu will result in a call to
477
`completion_start()` which in turn will start `bash --rcfile config/bashrc`.
479
.What happens when completion tests are run?
481
| runtest --tool completion
483
+----------+-----------+
484
| lib/completion.exp |
486
| config/default.exp |
487
+----------+-----------+
490
+----------+-----------+ +---------------+ +----------------+
491
| completion_start() +<---+ config/bashrc +<---| config/inputrc |
492
| (lib/completion.exp) | +---------------+ +----------------+
493
+----------+-----------+
494
| ,+----------------------------+
495
| ,--+-+ "Actual completion tests" |
496
V / +------------------------------+
497
+----------+-----------+ +-----------------------+
498
| completion/*.exp +<---| lib/completions/*.exp |
499
+----------+-----------+ +-----------------------+
500
| \ ,+--------------------------------+
501
| `----------------------+-+ "Completion invocation tests" |
502
V +----------------------------------+
503
+----------+-----------+
504
| completion_exit() |
505
| (lib/completion.exp) |
506
+----------------------+
508
Setting up bash once within `completion_start()` has the speed advantage that
509
bash - and bash-completion - need only initialize once when testing multiple
512
runtest --tool completion alias.exp cd.exp
516
.What happens when install tests are run?
518
| runtest --tool install
525
+------------+---------------+
526
| (file: config/default.exp) |
527
+------------+---------------+
530
+------------+------------+
531
| (file: lib/install.exp) |
532
+-------------------------+
537
.What happens when unit tests are run?
539
| runtest --tool unit
546
+----------+-----------+
548
| (file: lib/unit.exp) |
549
+----------------------+
554
This is the bash configuration file (bashrc) used for testing:
557
---------------------------------------------------------------------
559
---------------------------------------------------------------------
564
This is the readline configuration file (inputrc) used for testing:
567
---------------------------------------------------------------------
569
---------------------------------------------------------------------
added
removed
doc/testing.txt
