~ubuntu-branches/ubuntu/feisty/dejagnu/feisty

<para>This chapter was originally written by Niklaus Giger (ngiger@mus.ch) because he lost a week to figure out how DejaGnu works and how to write a first test.</para>

<para>Follow these instructions as closely a possible in order get a good insight into how DejaGnu works, else you might run into a lot of subtle problems. You have been warned.</para>

<para>It should be no big problems installing DejaGnu using your package manager or from the source code. Under a Debian/GNU/Linux systems just type (as root) <programlisting>apt-get dejagnu</programlisting>. These examples were run on a primary machine with a AMD K6 and a Mac Powerbook G3 serving as a remote target.</para>

<para> The tests for Windows were run under Windows NT using the actual Cygwin version (1.3.x as of October 2001). It's target system was a PPC embedded system running vxWorks.

</para>

<sect1>

<title>Test your installation</title>

<para>Create a new user called "dgt" (DejaGnuTest), which uses bash as it login shell. PS1 must be set to '\u:\w\$ ' in its ~/.bashrc. Login as this user, create an empty directory and change the working directory to it. e.g</para>

dgt:~$ mkdir ~/dejagnu.test

dgt:~$ cd ~/dejagnu.test

</programlisting>

<para>Now you are ready to test DejaGnu's main program called runtest. The expecteted output is shown</para>

<title>Runtest output in a empty directory

</title>

dgt:~/dejagnu.test$ runtest

WARNING: Couldn't find the global config file.

WARNING: No tool specified Test

Run By dgt on Sun Nov 25 17:07:03 2001 Native configuration is i586-pc-linux-gnu

=== tests ===

Schedule of variations: unix

Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.

Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.

ERROR: Couldn't find tool config file for unix.

=== Summary ===</programlisting>

<para>We will show you later how to get rid of all the WARNING- and ERROR-messages. The files testrun.sum and testrun.log have been created, which do not interest us at this point. Let's remove them.</para>

<programlisting>:~/dejagnu.test$ rm testrun.sum testrun.log

</programlisting>

</example>

<sect2>

<title>Windows</title>

<para>On Windows systems DejaGnu is part of a port of a lot of Unix tools to the Windows OS, called Cygwin. Cygwin may be downloaded and installed from a mirror of http://www.cygwin.com/. All examples were also run on Windows NT. If nothing is said, you can assume that you should get the same output as on a Unix system.</para>

<para>You will need a telnet daemon if you want to use a Windows box as a remote target. There seems to be a freeware telnet daemon at http://www.fictional.net/.</para>

</sect2>

<sect2>

<title>Getting the source code for the calc example</title>

<para>If you are running a Debian distribution you can find the examples under /usr/share/doc/dejagnu/examples.

These examples seem to be missing in Red Hat's RPM.

In this case download the sources of DejaGnu and adjust the pathes to the DejaGnu examples accordingly.</para>

</sect2>

</sect1>

<sect1>

<title>Create a minimal project, e.g. calc</title>

<para>In this section you will to start a small project,

using the sample application calc, which is part of your DejaGnu distribution</para>

<sect2><title>A simple project without the GNU autotools</title>

<para>The runtest program can be run standalone. All the autoconf/automake support is just cause those programs are commonly used for other GNU applications. The key to running runtest standalone is having the local site.exp file setup correctly, which automake does.</para>

<para>The generated site.exp should like like:</para>

set tool calc

set srcdir .

set objdir /home/dgt/dejagnu.test

</programlisting></sect2>

<sect2>

<title>Using autoconf/autoheader/automake</title>

<para>We have to prepare some input file in order to run autocon and automake. There is book “GNU autoconf, automake and libtool” by Garry V. Vaughan, et al. NewRider, ISBN 1-57870-190-2 which describes this process thoroughly.</para>

<para>From the calc example distributed with the DejaGnu documentation you should copy the program file itself (calc.c) and some additional files, which you might examine a little bit close to derive their meanings.</para>

dgt:~/dejagnu.test$ cp -r /usr/share/doc/dejagnu/examples/calc/\

{configure.in,Makefile.am,calc.c,testsuite} .

</programlisting>

<para>In Makemake.am note the presence of the AUTOMAKE_OPTIONS = dejagnu. This option is needed.</para>

<para>Run aclocal to generate aclocal.m4, which is a collection of macros needed by configure.in</para>

dgt:~/dejagnu.test$ aclocal

</programlisting>

<para>autoconf is another part of the auto-tools.

100

Run it to generate configure based on information contained in configure.in.</para>

101

102

103

dgt:~/dejagnu.test$ autoconf

104

</programlisting>

105

106

<para>autoheader is another part of the auto-tools.

107

Run it to generate calc.h.in. </para>

108

109

110

dgt:~/dejagnu.test$ autoheader

111

</programlisting>

112

113

<para>The Makefile.am of this example was developed as port of the DejaGnu

114

distribution.

115

Adapt Makefile.am for this test. Replace the line

116

“&num;noinst_PROGRAMS = calc” to

117

“bin_PROGRAMS = calc”.

118

Change the RUNTESTDEFAULTFLAGS from

119

“&dollar;&dollar;srcdir/testsuite” to

120

“./testsuite”.</para>

121

122

<para>Running automake at this point contains a series of warning in its output as shown in the following example:</para>

123

124

125

<title>Sample output of automake with missing files</title>

126

127

dgt:~/dejagnu.test$ automake --add-missing

128

automake: configure.in: installing `./install-sh'

129

automake: configure.in: installing `./mkinstalldirs'

130

automake: configure.in: installing `./missing'

131

automake: Makefile.am: installing `./INSTALL'

132

automake: Makefile.am: required file `./NEWS' not found

133

automake: Makefile.am: required file `./README' not found

134

automake: Makefile.am: installing `./COPYING'

135

automake: Makefile.am: required file `./AUTHORS' not found

136

automake: Makefile.am: required file `./ChangeLog' not found

137

configure.in: 4: required file `./calc.h.in' not found

138

Makefile.am:6: required directory ./doc does not exist

139

</programlisting>

140

</example>

141

142

<para>Create a empty directory doc and empty files

143

INSTALL, NEWS, README, AUTHORS, ChangeLog and COPYING.

144

The default COPYING will point to the GNU Public License (GPL).

145

In a real project it would be time to add some meaningfull text in each file.

146

</para>

147

148

<para>Adapt calc to your environment by calling configure.</para>

149

150

<title>Sample output of configure

151

</title>

152

153

dgt:~/dejagnu.test$ ./configure

154

creating cache ./config.cache

155

checking whether to enable maintainer-specific portions of Makefiles... no

156

checking for a BSD compatible install... /usr/bin/install -c

157

checking whether build environment is sane... yes

158

checking whether make sets ${MAKE}... yes

159

checking for working aclocal... found

160

checking for working autoconf... found

161

checking for working automake... found

162

checking for working autoheader... found

163

checking for working makeinfo... found

164

checking for gcc... gcc checking whether the C compiler (gcc ) works... yes

165

checking whether the C compiler (gcc ) is a cross-compiler... no

166

checking whether we are using GNU C... yes

167

checking whether gcc accepts -g... yes

168

checking for a BSD compatible install... /usr/bin/install -c

169

checking how to run the C preprocessor... gcc -E

170

checking for stdlib.h... yes

171

checking for strcmp... yes

172

updating cache ./config.cache

173

creating ./config.status

174

creating Makefile creating calc.h

175

</programlisting>

176

</example>

177

178

<para>If you are familiar with GNU software,

179

this output should not contain any surprise to you.

180

Any errors should be easy to fix for such a simple program.</para>

181

182

<para>Build the calc executable:</para>

183

184

185

<title>Sample output building calc

186

</title>

187

188

dgt:~/dejagnu.test$ make

189

gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c

190

gcc -g -O2 -o calc calc.o

191

</programlisting>

192

</example>

193

194

<para>You prepared a few files and then called some commands.

195

Respecting the right order assures a automatic and correctly compiled calc program. The following example resumes the correct order.</para>

196

197

198

<title>Creating the calc program using the GNU autotools</title>

199

200

dgt:~/dejagnu.test$ aclocal

201

dgt:~/dejagnu.test$ autoconf

202

dgt:~/dejagnu.test$ autoheader

203

dgt:~/dejagnu.test$ automake --add-missing

204

dgt:~/dejagnu.test$ ./configure

205

dgt:~/dejagnu.test$ make

206

207

</programlisting>

208

</example>

209

210

<para>Play with calc and verify whether it works correctly.

211

A sample session might look like this:</para>

212

213

214

dgt:~/dejagnu.test$ ./calc

215

calc: version

216

Version: 1.1

217

calc:<emphasis> </emphasis>add 3 4

218

219

calc: multiply 3 4<emphasis> </emphasis>

220

221

calc: multiply 2 4<emphasis> </emphasis>

222

223

calc: quit

224

225

</programlisting>

226

227

<para>Look at the intentional bug that 2 times 4 equals 12.</para>

228

<para>The tests run by DejaGnu need a file called site.exp,

229

which is automatically generated if we call “make site.exp”.

230

This was the purpose of the “AUTOMAKE_OPTIONS = dejagnu” in Makefile.am.</para>

231

232

233

<title>Sample output generating a site.exp</title>

234

235

dgt: make site.exp

236

dgt:~/dejagnu.test$ make site.exp

237

Making a new site.exp file...

238

</programlisting>

239

</example>

240

241

242

</sect2>

243

</sect1>

244

245

<sect1>

246

<title>Our first automated tests</title>

247

<sect2><title>Running the test for the calc example</title>

248

249

<para>Now we are ready to call the automated tests </para>

250

251

252

<title>Sample output of runtest in a configured directory</title>

253

254

255

dgt:~/dejagnu.test$ make check

256

make check-DEJAGNU

257

make[1]: Entering directory `/home/dgt/dejagnu.test' srcdir=`cd . && pwd`; export srcdir; \

258

EXPECT=expect; export EXPECT; \ runtest=runtest; \

259

if /bin/sh -c "$runtest --version" > /dev/null 2>&1; then \

260

$runtest --tool calc CALC=`pwd`/calc --srcdir ./testsuite ; \

261

else echo "WARNING: could not find \`runtest'" 1>&2; :;\

262

263

WARNING: Couldn't find the global config file.

264

WARNING: Couldn't find tool init file

265

Test Run By dgt on Sun Nov 25 21:42:21 2001

266

Native configuration is i586-pc-linux-gnu

267

268

=== calc tests ===

269

270

Schedule of variations:

271

unix

272

273

Running target unix

274

Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.

275

Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.

276

Using ./testsuite/config/unix.exp as tool-and-target-specific interface file.

277

Running ./testsuite/calc.test/calc.exp ...

278

FAIL: multiply2 (bad match)

279

280

=== calc Summary ===

281

282

# of expected passes 5

283

# of unexpected failures 1

284

/home/Dgt/dejagnu.test/calc version Version: 1.1

285

make[1]: *** [check-DEJAGNU] Fehler 1

286

make[1]: Leaving directory `/home/Dgt/dejagnu.test' make: *** [check-am] Fehler 2

287

</programlisting>

288

</example>

289

290

<para>Did you see the line “FAIL:“? The test cases for calc catch the bug in the calc.c file. Fix the error in calc.c later as the following examples assume a unchanged calc.c.</para>

291

292

<para>Examine the output files calc.sum and calc.log.

293

Try to understand the testcases written in ~/dejagnu.test/testsuite/calc.test/calc.exp.

294

To understand Expect you might take a look at the book "Exploring Expect",

295

which is an excellent resource for learning and using Expect. (Pub: O'Reilly, ISBN 1-56592-090-2)

296

The book contains hundreds of examples and also includes a tutorial on Tcl.

297

Exploring Expect is 602 pages long. </para>

298

</sect2>

299

300

301

<sect2><title>The various config files or how to avoid warnings</title>

302

303

<para>DejaGnu may be customized by each user. It first searches for a file called ~/.dejagnurc. Create the file ~/.dejagnurc and insert the following line:</para>

304

305

306

puts "I am ~/.dejagnurc"

307

</programlisting>

308

309

<para>Rerun make check. Test whether the output contains "I am ~/.dejagnurc".

310

Create ~/my_dejagnu.exp and insert the following line:</para>

311

312

313

puts "I am ~/my_dejagnu.exp"

314

</programlisting>

315

316

<para>In a Bash-Shell enter</para>

317

318

319

dgt:~/dejagnu.test$ export DEJAGNU=~/my_dejagnu.exp

320

</programlisting>

321

322

<para>Run “make check” again. The output should not contain

323

“WARNING: Couldn't find the global config file.”.

324

Create the sub-directory lib. Create the file “calc.exp” in it and insert the following line:</para>

325

326

327

puts "I am lib/calc.exp"

328

</programlisting>

329

330

<para>The last warning “WARNING: Couldn't find tool init file”

331

should not be part of the output of make check.

332

Create the directory &tilde;/boards. Create the file &tilde;/boards/standard.exp and insert the following line:</para>

333

334

335

puts "I am boards/standard.exp"

336

</programlisting>

337

338

<para>If the variable DEJAGNU is still not empty then the (abbreviated) output of “make check” should look like this:</para>

339

340

341

<title>Sample output of runtest with the usual configuration files</title>

342

343

dgt:~/dejagnu.test$ make check

344

<...>

345

346

I am ~/.dejagnurc

347

I am ~/my_dejagnu.exp

348

I am lib/calc.exp

349

Test Run By dgt on Sun Nov 25 22:19:14 2001

350

Native configuration is i586-pc-linux-gnu

351

352

=== calc tests ===

353

Using /home/Dgt/boards/standard.exp as standard board description\

354

file for build.

355

I am ~/boards/standard.exp

356

Using /home/Dgt/boards/standard.exp as standard board description\

357

file for host.

358

I am ~/boards/standard.exp

359

360

Schedule of variations:

361

unix

362

363

Running target unix

364

Using /home/Dgt/boards/standard.exp as standard board description\

365

file for target.

366

I am ~/boards/standard.exp

367

Using /usr/share/dejagnu/baseboards/unix.exp as board description file\

368

for target.

369

<...>

370

</programlisting>

371

</example>

372

373

<para>It is up to you to decide when and where to use any of the above

374

mentioned config files for customizing.

375

This chapters showed you where and in which order the different config

376

files are run.</para>

377

</sect2>

378

379

<sect2>

380

<title>When trouble strikes</title>

381

382

<para>Calling runtest with the '-v'-flag shows you in even more details which files are searched in which order. Passing it several times gives more and more details. </para>

383

384

<title>Displaying details about runtest execution</title>

385

386

runtest -v -v -v --tool calc CALC=`pwd`/calc --srcdir ./testsuite

387

</programlisting>

388

</example>

389

390

<para>Calling runtest with the '--debug'-flag logs a lot of details to dbg.log where you can analyse it afterwards. </para>

391

392

<para>In all test cases you can temporary adjust the verbosity of information by adding the following Tcl-command to any tcl file that gets loaded by

393

dejagnu, for instance, ~/.dejagnurc:</para>

394

395

396

set verbose 9

397

</programlisting>

398

399

</sect2>

400

401

<sect2>

402

<title>Testing “Hello world” locally</title>

403

404

<para>This test checks, whether the built-in shell command “echo Hello world”

405

will really write “Hello world” on the console.

406

Create the file ~/dejagnu.test/testsuite/calc.test/local_echo.exp.

407

It should contain the following lines</para>

408

409

410

<title>A first (local) test case</title>

411

412

set test "Local Hello World"

413

send "echo Hello World"

414

expect {

415

-re "Hello World" { pass "$test" }

416

}

417

</programlisting>

418

</example>

419

420

<para>Run runtest again and verify the output “calc.log”</para>

421

</sect2>

422

</sect1>

423

424

<sect1>

425

<title>A first remote test</title>

426

427

<para>Testing remote targets is a lot trickier especially if you are using an

428

embedded target

429

which has no built in support for things like a compiler, ftp server or a Bash-shell.

430

Before you can test calc on a remote target you have to acquire a few basics skills.</para>

431

432

<sect2>

433

<title>Setup telnet to your own host</title>

434

<para>The easiest remote host is usually the host you are working on.

435

In this example we will use telnet to login in your own workstation.

436

For security reason you should never have a telnet deamon running on

437

machine connected on the internet, as password and usernames are transmitted

438

in clear text.

439

We assume you know how to setup your machine for a telnet daemon.</para>

440

441

<para>Next try whether you may login in your own host by issuing the

442

command “telnet localhost.1”. In order to be able to

443

distinguish between a normal session an a telnet login add the following lines to /home/dgt/.bashrc.</para>

444

445

446

if [ "$REMOTEHOST" ]

447

then

448

PS1='remote:\w\$ '

449

450

</programlisting>

451

452

<para>Now on the machine a “remote” login looks like this:</para>

453

454

455

<title>Sample log of a telnet login to localhost</title>

456

457

dgt:~/dejagnu.test$ telnet localhost

458

Trying 127.0.0.1...

459

Connected to 127.0.0.1.

460

Escape character is '^]'.

461

Debian GNU/Linux testing/unstable Linux

462

K6Linux login: dgt

463

Password:

464

Last login: Sun Nov 25 22:46:34 2001 from localhost on pts/4

465

Linux K6Linux 2.4.14 #1 Fre Nov 16 19:28:25 CET 2001 i586 unknown

466

No mail.

467

remote:~$ exit

468

logout

469

Connection closed by foreign host.

470

</programlisting>

471

</example>

472

</sect2>

473

474

<sect2>

475

<title>A test case for login via telnet</title>

476

<para>In order to define a correct setup we have add a line containing

477

“set target unix” either to ~/.dejagnurc or to ~/my_dejagnu.exp.

478

In ~/boards/standard.exp add the following four lines to define a few patterns for the DejaGnu telnet login procedure.</para>

479

480

481

<title>Defining a remote target board</title>

482

483

set_board_info shell_prompt "remote:"

484

set_board_info telnet_username "dgt"

485

set_board_info telnet_password "top_secret"

486

set_board_info hostname "localhost"

487

488

</programlisting>

489

</example>

490

491

<para>As DejaGnu will be parsing the telnet session output for some well

492

known pattern the output there are a lot of things that can go wrong.

493

If you have any problems verify your setup:</para>

494

495

496

497

<para>Is <filename>/etc/motd</filename> empty?</para></listitem>

498

499

500

<para>Is <filename>/etc/issue.net</filename> empty?</para></listitem>

501

502

503

<para>Exists a empty <filename>~/.hushlogin</filename>?</para></listitem>

504

505

506

<para>The LANG environment variable must be either empty or set to “C”. </para></listitem>

507

508

</itemizedlist>

509

<para>To test the login via telnet write a sample test case.

510

Create the file ~/dejagnu.test/testsuite/calc.test/remote_echo.exp and

511

add the following few lines:</para>

512

513

514

<title>DejaGnu script for logging in into a remote target</title>

515

516

puts "this is remote_echo.exp target for $target "

517

target_info $target

518

#set verbose 9

519

set shell_id [remote_open $target]

520

set test "Remote login to $target"

521

#set verbose 0

522

puts "Spawn id for remote shell is $shell_id"

523

if { $shell_id > 0 } {

524

pass "$test"

525

} else {

526

fail "Remote open to $target"

527

}

528

</programlisting>

529

</example>

530

531

<para>In the runtest output you should find something like:</para>

532

533

534

Running ./testsuite/calc.test/local_echo.exp ...

535

Running ./testsuite/calc.test/remote_echoo.exp ...

536

this is remote_echo.exp target is unix

537

Spawn id for remote shell is exp7

538

</programlisting>

539

540

<para>Have again a look at calc.log to get a feeling how DejaGnu and expect

541

parse the input. </para></sect2>

542

543

<sect2>

544

<title>Remote testing “Hello world”</title>

545

546

<para>Next you will transform the above “hello world” example to

547

its remote equivalent.

548

This can be done by adding the following lines to our file remote_echo.exp.</para>

549

550

551

<title>A first (local) remote "Hello world" test</title>

552

553

set test "Remote_send Hello World"

554

set status [remote_send $target "echo \"Hello\" \"World\"\n" ]

555

pass "$test"

556

set test "Remote_expect Hello World"

557

remote_expect $target 5 {

558

-re "Hello World" { pass "$test" }

559

}

560

</programlisting>

561

</example>

562

563

<para>Call make check. The output should contain

564

“&num; of expected passes 9” and “&num; of unexcpected failures 1”.</para>

565

566

<para>Have a look at the procedures in /usr/share/dejagnu/remote.exp to have an overview of the offered procedures and their features. </para>

567

568

<para>Now setup a real target.

569

In the following example we assume as target a PowerBook running Debian.

570

As above add a test user "dgt", install telnet and FTP servers.

571

In order to distinguish it from the host add the line

572

<programlisting>PS1='test:>'</programlisting> to /home/dgt/.bash_profile.

573

Also add a corresponding entry "powerbook" to /etc/hosts and verify that you

574

are able to ping, telnet and ftp to the target "powerbook".</para>

575

576

<para>In order to let runtest run its test on the "powerbook" target change the following lines in ~/boards/standard.exp:</para>

577

578

579

<title>Board definition for a remote target</title>

580

581

set_board_info protocol "telnet"

582

set_board_info telnet_username "dgt"

583

set_board_info telnet_password "top_secret"

584

set_board_info shell_prompt "test:> "

585

set_board_info hostname "powerbook"

586

</programlisting>

587

</example>

588

589

<para>Now call runtest again with the same arguments and verify whether all went okay by taking a close look at calc.log.</para>

590

</sect2>

591

592

593

<sect2>

594

<title>Transferring files from/to the target</title>

595

596

<para>A simple procedure like this will do the job for you:</para>

597

598

599

<title>Test script to transfer a file to a remote target</title>

600

601

set test "Remote_download"

602

puts "Running Remote_download"

603

# set verbose 9

604

set remfile /home/dgt/dejagnu2

605

606

set status [remote_download $target /home/dgt/.dejagnurc $remfile]

607

if { "$status" == "" } {

608

fail "Remote download to $remfile on $target"

609

} else {

610

pass "$test"

611

}

612

613

puts "status of remote_download ist $status"

614

# set verbose 0

615

</programlisting>

616

</example>

617

618

<para>After running runtest again, check whether the file dejagnu2 exists on the target.

619

620

This example will only work if the rcp command works with your target.

621

622

If you have a working FTP-server on the target you can use it by adding the

623

following lines to ~/boards/standard.exp:</para>

624

625

626

<title>Defining a board to use FTP as file transport</title>

627

628

set_board_info file_transfer "ftp"

629

set_board_info ftp_username "dgt"

630

set_board_info ftp_password "1234"

631

</programlisting>

632

</example>

633

634

</sect2>

635

636

<sect2>

637

<title>Preparing for crosscompilation</title>

638

639

<para>For crosscompiling you need working binutils, gcc and a base library like

640

libc or glib for your target.

641

It is beyond the scope of this document to describe how to get it working.

642

The following examples assume a cross compiler for PowerPC which is called linux-powerpc-gcc.

643

</para>

644

645

<para>Add AC_CANONICAL_TARGET in dejagnu.test/configure.in at the following location. Copy config.guess from /usr/share/automake to dejagnu.test.</para>

646

647

648

AM_CONFIG_HEADER(calc.h)

649

AC_CANONICAL_TARGET([])

650

AM_INIT_AUTOMAKE(calc, 1.1)

651

</programlisting>

652

653

<para>You need to run automake 2.5 or later.

654

Depending on your installation calling autoconf2.5 instead of autoconf is not needed.

655

The sequence to regenerate all files is:</para>

656

657

658

<title>Using autotools for cross development</title>

659

660

$ autoconf2.5

661

$ autoheader

662

$ automake

663

$ ./configure --host=powerpc-linux --target=powerpc-linux

664

configure: WARNING: If you wanted to set the --build type, don't use --host.

665

If a cross compiler is detected then cross compile mode will be used.

666

checking build system type... ./config.guess: ./config.guess: No such file or directory

667

configure: error: cannot guess build type; you must specify one

668

$ cp /usr/share/automake/config.guess .

669

$ ./configure --host=powerpc-linux --target=powerpc-linux

670

configure: WARNING: If you wanted to set the --build type, don't use --host.

671

If a cross compiler is detected then cross compile mode will be used. \

672

checking build system type... i586-pc-linux-gnu

673

checking host system type... powerpc-unknown-linux-gnu

674

<...>

675

checking whether we are cross compiling... yes

676

<...>

677

Configuration:

678

Source code location: .

679

C Compiler: powerpc-linux-gcc

680

C Compiler flags: -g -O2

681

682

</programlisting>

683

</example>

684

685

<para>Everything should be ready to recompile for the target:</para>

686

687

<programlisting>$ make

688

powerpc-linux-gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c calc.c

689

powerpc-linux-gcc -g -O2 -o calc calc.o

690

691

</programlisting>

692

</sect2>

693

694

<sect2>

695

<title>Remote testing of calc</title>

696

<para>Not yet written, as I have problem getting libc6-dev-powerpc to work. Probably I first have to build my cross compiler. </para>

697

</sect2>

698

699

<sect2>

700

<title>Using Windows as host and vxWorks as target</title>

701

702

<para>A more thorough walk-through will be written in a few weeks.</para>

703

704

<para>In order to test the vxWorks as a target I changed boards/standards.exp to reflect my settings (IP, username, password). Then I reconfigured vxWorks to include a FTP and telnet server (using the same username/password combination ad in boards/standard.exp).</para>

705

706

<para>With this setup and some minor modification (e.g. replacing echo by printf) in my test cases I could test my vxWorks system. It sure does not seem to be a correct setup by DejaGnu standard. For instance, it still loading /usr/share/dejagnu/baseboards/unix.exp instead of vxWorks. In any case I found that (at least under Windows) I did not find out how the command line would let me override settings in my personal config files.</para>

707

708

</sect2>

709

</sect1>

710

</chapter>

711

712

<title>Running Tests</title>

713

714

<para>There are two ways to execute a testsuite. The most

715

common way is when there is existing support in the

716

<filename>Makefile</filename>. This support consists of a

717

<emphasis>check</emphasis> target. The other way is to execute the

718

<command>runtest</command> program directly. To run

719

<command>runtest</command> directcly from the command line requires

720

either all the correct options, or the <xref linkend=local> must be setup

721

correctly.</para>

722

723

724

<title>Make check</title>

725

726

<para>To run tests from an existing collection, first use

727

<command>configure</command> as usual to set up the

728

build directory. Then try typing:</para>

729

730

731

make check

732

</screen>

733

734

<para>If the <emphasis>check</emphasis> target exists, it

735

usually saves you some trouble. For instance, it can set up any

736

auxiliary programs or other files needed by the tests. The most

737

common file the check builds is the

738

<emphasis>site.exp</emphasis>. The site.exp file contains

739

various variables that DejaGnu used to dertermine the

740

configuration of the program being tested. This is mostly for

741

supporting remote testing.</para>

742

743

<para>The <emphasis>check</emphasis> target is supported by GNU

744

<productname>Automake</productname>. To have DejaGnu support added to your

745

generated <filename>Makefile.in</filename>, just add the keyword

746

dejagnu to the AUTOMAKE_OPTIONS variable in your

747

<filename>Makefile.am</filename> file.</para>

748

749

<para>Once you have run <emphasis>make check</emphasis> to build

750

any auxiliary files, you can invoke the test driver

751

<command>runtest</command> directly to repeat the tests.

752

You will also have to execute <command>runtest</command>

753

directly for test collections with no

754

<emphasis>check</emphasis> target in the

755

<filename>Makefile</filename>.</para>

756

757

</sect1>

758

759

760

<title>Runtest</title>

761

762

<para><command>runtest</command> is the executable test driver

763

for DejaGnu. You can specify two kinds of things on the

764

<command>runtest</command> command line: command line options,

765

and Tcl variables for the test scripts. The options are listed

766

alphabetically below.</para>

767

768

<para><command>runtest</command> returns an exit code of

769

<emphasis>1</emphasis> if any test has an unexpected result; otherwise

770

(if all tests pass or fail as expected) it returns <emphasis>0</emphasis>

771

as the exit code.</para>

772

773

774

<title>Output States</title>

775

776

<para><filename>runtest</filename> flags the outcome of each

777

test as one of these cases. <xref linkend=posix> for a

778

discussion of how POSIX specifies the meanings of these

779

cases.</para>

780

781

782

783

784

<listitem><para>The most desirable outcome: the test succeeded, and

785

was expected to succeed.</para></listitem>

786

</varlistentry>

787

788

789

<term>XPASS</term>

790

<listitem><para>A pleasant kind of failure: a test was expected to

791

fail, but succeeded. This may indicate progress; inspect the test

792

case to determine whether you should amend it to stop expecting

793

failure.</para></listitem>

794

</varlistentry>

795

796

797

798

<listitem><para>A test failed, although it was expected to succeed.

799

This may indicate regress; inspect the test case and the failing

800

software to ocate the bug.</para></listitem>

801

</varlistentry>

802

803

804

<term>XFAIL</term>

805

<listitem><para>A test failed, but it was expected to fail. This

806

result indicates no change in a known bug. If a test fails because

807

the operating system where the test runs lacks some facility required

808

by the test, the outcome is <emphasis>UNSUPPORTED</emphasis>

809

instead.</para></listitem>

810

</varlistentry>

811

812

813

<term>UNRESOLVED</term>

814

<listitem><para>Output from a test requires manual inspection; the

815

testsuite could not automatically determine the outcome. For

816

example, your tests can report this outcome is when a test does not

817

complete as expected.</para></listitem>

818

</varlistentry>

819

820

821

<term>UNTESTED</term>

822

<listitem><para>A test case is not yet complete, and in particular

823

cannot yet produce a <emphasis>PASS</emphasis> or

824

<emphasis>FAIL</emphasis>. You can also use this outcome in dummy

825

``tests'' that note explicitly the absence of a real test case for a

826

particular property.</para></listitem>

827

</varlistentry>

828

829

830

<term>UNSUPPORTED</term>

831

<listitem><para>A test depends on a conditionally available feature

832

that does not exist (in the configured testing environment). For

833

example, you can use this outcome to report on a test case that does

834

not work on a particular target because its operating system support

835

does not include a required subroutine.</para></listitem>

836

</varlistentry>

837

</variablelist>

838

839

<para>runtest may also display the following messages:</para>

840

841

842

843

<term>ERROR</term>

844

<listitem><para>Indicates a major problem (detected by the test case

845

itself) in running the test. This is usually an unrecoverable error,

846

such as a missing file or loss of communication to the target. (POSIX

847

testsuites should not emit this message; use

848

<emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or

849

<emphasis>UNRESOLVED</emphasis> instead, as

850

appropriate.)</para></listitem>

851

</varlistentry>

852

853

854

<term>WARNING</term>

855

<listitem><para>Indicates a possible problem in running the

856

test. Usually warnings correspond to recoverable errors, or display

857

an important message about the following tests.</para></listitem>

858

</varlistentry>

859

860

861

862

<listitem><para>An informational message about the test

863

case.</para></listitem>

864

</varlistentry>

865

</variablelist>

866

867

</sect2>

868

869

870

<title>Invoking Runtest</title>

871

872

<para>This is the full set of command line options that

873

<filename>runtest</filename> recognizes. Arguments may be

874

abbreviated to the shortest unique string.</para>

875

876

877

878

879

<listitem><para>Display all test output. By default,

880

<emphasis>runtest</emphasis> shows only the output of tests that

881

produce unexpected results; that is, tests with status

882

<emphasis>FAIL</emphasis> (unexpected failure),

883

<emphasis>XPASS</emphasis> (unexpected success), or

884

<emphasis>ERROR</emphasis> (a severe error in the test case

885

itself). Specify <emphasis>--all</emphasis> to see output for tests

886

with status <emphasis>PASS</emphasis> (success, as expected)

887

<emphasis>XFAIL</emphasis> (failure, as expected), or

888

<emphasis>WARNING</emphasis> (minor error in the test case

889

itself).</para></listitem>

890

</varlistentry>

891

892

893

<term><option>--build [string]</option></term>

894

<listitem><para><emphasis>string</emphasis> is a full configuration

895

``triple'' name as used by <command>configure</command>. This

896

is the type of machine DejaGnu and the tools to be tested are built

897

on. For a normal cross this is the same as the host, but for a

898

canadian cross, they are seperate.</para></listitem>

899

</varlistentry>

900

901

902

<term><option>--host [string]</option></term>

903

<listitem><para><symbol>string</symbol> is a full configuration

904

``triple'' name as used by <emphasis>configure</emphasis>. Use this

905

option to override the default string recorded by your

906

configuration's choice of host. This choice does not change how

907

anything is actually configured unless --build is also specified; it

908

affects <emphasis>only</emphasis> DejaGnu procedures that compare the

909

host string with particular values. The procedures

910

<emphasis>ishost</emphasis>, <emphasis>istarget</emphasis>,

911

<emphasis>isnative</emphasis>, and <emphasis>setup</emphasis>xfail}

912

are affected by <emphasis>--host</emphasis>. In this usage,

913

<emphasis>host</emphasis> refers to the machine that the tests are to

914

be run on, which may not be the same as the

915

<emphasis>build</emphasis> machine. If <emphasis>--build</emphasis>

916

is also specified, then <emphasis>--host</emphasis> refers to the

917

machine that the tests wil, be run on, not the machine DejaGnu is run

918

on.</para></listitem>

919

</varlistentry>

920

921

922

<term><option>--host_board [name]</option></term>

923

<listitem><para>The host board to use.</para></listitem>

924

</varlistentry>

925

926

927

<term><option>--target [string]</option></term>

928

<listitem><para>Use this option to override the default setting

929

(running native tests). <emphasis>string</emphasis> is a full

930

configuration ``triple'' name of the form

931

<emphasis>cpu-vendor-os</emphasis> as used by

932

<command>configure</command>. This option changes the

933

configuration <emphasis>runtest</emphasis> uses for the default tool

934

names, and other setup information.</para></listitem>

935

</varlistentry>

936

937

938

<term><option>--debug</option> (-de)</term>

939

<listitem><para>Turns on the <emphasis>expect</emphasis> internal

940

debugging output. Debugging output is displayed as part of the

941

<emphasis>runtest</emphasis> output, and logged to a file called

942

<filename>dbg.log</filename>. The extra debugging output does

943

<emphasis>not</emphasis> appear on standard output, unless the

944

verbose level is greater than 2 (for instance, to see debug output

945

immediately, specify <emphasis>--debug</emphasis>-v -v}). The

946

debugging output shows all attempts at matching the test output of

947

the tool with the scripted patterns describing expected output. The

948

output generated with <emphasis>--strace</emphasis> also goes into

949

950

</varlistentry>

951

952

953

954

<listitem><para>Prints out a short summary of the

955

<emphasis>runtest</emphasis> options, then exits (even if you also

956

specify other options).</para></listitem>

957

</varlistentry>

958

959

960

<term><option>--ignore [name(s)] </option></term>

961

<listitem><para>The names of specific tests to

962

ignore.</para></listitem>

963

</varlistentry>

964

965

966

<term><option>--objdir [path]</option></term>

967

<listitem><para>Use <emphasis>path</emphasis> as the top directory

968

containing any auxiliary compiled test code. This defaults to

969

<filename>.</filename>. Use this option to locate pre-compiled test

970

code. You can normally prepare any auxiliary files needed with

971

972

</varlistentry>

973

974

975

<term><option>--outdir [path]</option></term>

976

<listitem><para>Write output logs in directory

977

<filename>path</filename>. The default is <emphasis>.},

978

the</emphasis> directory where you start

979

<emphasis>runtest</emphasis>. This option affects only the summary

980

and the detailed log files

981

<filename>tool.sum</filename> and

982

<filename>tool.log</filename>. The DejaGnu debug

983

log <filename>dbg.log</filename> always appears (when requested) in

984

the local directory.</para></listitem>

985

</varlistentry>

986

987

988

<term><option>--reboot [name]</option></term>

989

<listitem><para>Reboot the target board when

990

<emphasis>runtest</emphasis> initializes. Usually, when running tests

991

on a separate target board, it is safer to reboot the target to be

992

certain of its state. However, when developing test scripts,

993

rebooting takes a lot of time.</para></listitem>

994

</varlistentry>

995

996

997

<term><option>--srcdir [path]</option></term>

998

<listitem><para>Use <filename>path</filename> as the top directory

999

for test scripts to run. <emphasis>runtest</emphasis> looks in this

1000

directory for any subdirectory whose name begins with the toolname

1001

(specified with <emphasis>--tool</emphasis>). For instance, with

1002

<emphasis>--tool</emphasis>gdb}, <emphasis>runtest</emphasis> uses

1003

tests in subdirectories <filename>gdb.*</filename> (with the usual

1004

shell-like filename expansion). If you do not use

1005

<emphasis>--srcdir</emphasis>, <emphasis>runtest</emphasis> looks for

1006

test directories under the current working

1007

directory.</para></listitem>

1008

</varlistentry>

1009

1010

1011

<term><option>--strace [number]</option></term>

1012

<listitem><para>Turn on internal tracing for

1013

<emphasis>expect</emphasis>, to n levels deep. By adjusting the

1014

level, you can control the extent to which your output expands

1015

multi-level Tcl statements. This allows you to ignore some levels of

1016

<emphasis>case</emphasis> or <emphasis>if</emphasis> statements.

1017

Each procedure call or control structure counts as one ``level''. The

1018

output is recorded in the same file, <filename>dbg.log</filename>,

1019

used for output from <emphasis>--debug</emphasis>.</para></listitem>

1020

</varlistentry>

1021

1022

1023

<term><option>--connect [program]</option></term>

1024

<listitem><para>Connect to a target testing environment as specified

1025

by <emphasis>type</emphasis>, if the target is not the computer

1026

running <emphasis>runtest</emphasis>. For example, use

1027

<emphasis>--connect</emphasis> to change the program used to connect

1028

to a ``bare board'' boot monitor. The choices for

1029

<emphasis>type</emphasis> in the DejaGnu 1.4 distribution are

1030

<emphasis>rlogin</emphasis>, <emphasis>telnet</emphasis>,

1031

<emphasis>rsh</emphasis>, <emphasis>tip</emphasis>,

1032

<emphasis>kermit</emphasis>, and <emphasis>mondfe</emphasis>.</para>

1033

1034

<para>The default for this option depends on the configuration most

1035

convenient communication method available, but often other

1036

alternatives work as well; you may find it useful to try alternative

1037

connect methods if you suspect a communication problem with your

1038

testing target.</para></listitem>

1039

</varlistentry>

1040

1041

1042

<term><option>--baud [number]</option></term>

1043

<listitem><para>Set the default baud rate to something other than

1044

9600. (Some serial interface programs, like <emphasis>tip</emphasis>,

1045

use a separate initialization file instead of this

1046

value.)</para></listitem>

1047

</varlistentry>

1048

1049

1050

<term><option>--target_board [name(s)]</option></term>

1051

<listitem><para>The list of target boards to run tests

1052

on.</para></listitem>

1053

</varlistentry>

1054

1055

1056

1057

<listitem><para>Specifies which testsuite to run, and what

1058

initialization module to use. <option>--tool</option> is used

1059

<emphasis>only</emphasis> for these two purposes. It is

1060

<emphasis>not</emphasis> used to name the executable program to

1061

test. Executable tool names (and paths) are recorded in

1062

<filename>site.exp</filename> and you can override them by specifying

1063

Tcl variables on the command line.</para>

1064

1065

<para>For example, including "<option>--tool</option> gcc" on the

1066

<emphasis>runtest</emphasis> command line runs tests from all test

1067

subdirectories whose names match <filename>gcc.*</filename>, and uses

1068

one of the initialization modules named

1069

<filename>config/*-gcc.exp</filename>. To specify the name of the

1070

compiler (perhaps as an alternative path to what

1071

<emphasis>runtest</emphasis> would use by default), use

1072

<emphasis>GCC=binname</emphasis> on the <emphasis>runtest</emphasis>

1073

command line.</para></listitem>

1074

</varlistentry>

1075

1076

1077

1078

<listitem><para>The path to the tool executable to

1079

test.</para></listitem>

1080

</varlistentry>

1081

1082

1083

<term><option>--tool_opts [options]</option></term>

1084

<listitem><para>A list of additional options to pass to the

1085

tool.</para></listitem>

1086

</varlistentry>

1087

1088

1089

<term><option>--verbose</option> (-v)</term>

1090

<listitem><para>Turns on more output. Repeating this option increases

1091

the amount of output displayed. Level one (<emphasis>-v</emphasis>)

1092

is simply test output. Level two (<emphasis>-v</emphasis>-v}) shows

1093

messages on options, configuration, and process control. Verbose

1094

messages appear in the detailed (<filename>*.log</filename>) log

1095

file, but not in the summary (<filename>*.sum</filename>) log

1096

file.</para></listitem>

1097

</varlistentry>

1098

1099

1100

<term><option>--version</option> (-V)</term>

1101

<listitem><para>Prints out the version numbers of DejaGnu,

1102

<emphasis>expect</emphasis> and Tcl, and exits without running any

1103

tests.</para></listitem>

1104

</varlistentry>

1105

1106

1107

1108

<listitem><para>Start the internal Tcl debugger. The Tcl debugger

1109

supports breakpoints, single stepping, and other common debugging

1110

activities. See the document "Debugger for Tcl Applications" by Don

1111

Libes. (Distributed in PostScript form with

1112

<emphasis>expect</emphasis> as the file

1113

<filename>expect/tcl-debug.ps.</filename>. If you specify

1114

<emphasis>-D1</emphasis>, the <emphasis>expect</emphasis> shell stops

1115

at a breakpoint as soon as DejaGnu invokes it. If you specify

1116

<emphasis>-D0</emphasis>, DejaGnu starts as usual, but you can enter

1117

the debugger by sending an interrupt (e.g. by typing

1118

<keycombo><keycap>C</keycap><keycap>c</keycap></keycombo>).

1119

</para></listitem>

1120

</varlistentry>

1121

1122

1123

<term><filename>testfile</filename>.exp[=arg(s)]</term>

1124

<listitem><para>Specify the names of testsuites to run. By default,

1125

<emphasis>runtest</emphasis> runs all tests for the tool, but you can

1126

restrict it to particular testsuites by giving the names of the

1127

<emphasis>.exp expect</emphasis> scripts that control

1128

them. <emphasis>testsuite</emphasis>.exp may not include path

1129

information; use plain filenames.</para></listitem>

1130

</varlistentry>

1131

1132

1133

<term><filename>testfile</filename>.exp="testfile1 ..."</term>

1134

<listitem><para>Specify a subset of tests in a suite to run. For

1135

compiler or assembler tests, which often use a single

1136

<emphasis>.exp</emphasis> script covering many different source

1137

files, this option allows you to further restrict the tests by

1138

listing particular source files to compile. Some tools even support

1139

wildcards here. The wildcards supported depend upon the tool, but

1140

typically they are <emphasis>?</emphasis>, <emphasis>*</emphasis>,

1141

and <emphasis>[chars]</emphasis>.</para></listitem>

1142

</varlistentry>

1143

1144

1145

<term><symbol>tclvar</symbol>=value</term>

1146

<listitem><para>You can define Tcl variables for use by your test

1147

scripts in the same style used with <emphasis>make</emphasis> for

1148

environment variables. For example, <emphasis>runtest

1149

GDB=gdb.old</emphasis> defines a variable called

1150

<command>GDB</command>; when your scripts refer to

1151

<symbol>$GDB</symbol> in this run, they use the value

1152

1153

1154

<para>The default Tcl variables used for most tools are defined in

1155

the main DejaGnu <emphasis>Makefile</emphasis>; their values are

1156

captured in the <filename>site.exp</filename> file.</para></listitem>

1157

</varlistentry>

1158

</variablelist>

1159

</sect2>

1160

1161

1162

<title>Common Options</title>

1163

1164

<para>Typically, you don't need must to use any command-line options.

1165

<option>--tool</option> used is only required when there are more than

1166

one testsuite in the same directory. The default options are in the

1167

local site.exp file, created by "make site.exp".</para>

1168

1169

<para>For example, if the directory <filename>gdb/testsuite</filename>

1170

contains a collection of DejaGnu tests for GDB, you can run them like

1171

this:</para>

1172

1173

1174

eg$ cd gdb/testsuite

1175

eg$ runtest --tool gdb

1176

</screen>

1177

1178

<para>Test output follows, ending with:</para>

1179

1180

1181

=== gdb Summary ===

1182

1183

# of expected passes 508

1184

# of expected failures 103

1185

/usr/latest/bin/gdb version 4.14.4 -nx

1186

</screen>

1187

1188

<para>You can use the option <emphasis>--srcdir</emphasis> to point to

1189

some other directory containing a collection of tests:</para>

1190

1191

1192

eg$ runtest--srcdir /devo/gdb/testsuite

1193

</screen>

1194

1195

<para>By default, <command>runtest</command> prints only the

1196

names of the tests it runs, output from any tests that have unexpected

1197

results, and a summary showing how many tests passed and how many

1198

failed. To display output from all tests (whether or not they behave

1199

as expected), use the <emphasis>--all</emphasis> option. For more

1200

verbose output about processes being run, communication, and so on, use

1201

<emphasis>--verbose</emphasis>. To see even more output, use multiple

1202

<emphasis>--verbose</emphasis> options. for a more detailed explanation

1203

of each <command>runtest</command> option.</para>

1204

1205

<para>Test output goes into two files in your current directory:

1206

summary output in <filename>tool.sum</filename>,

1207

and detailed output in <filename>

1208

tool.log</filename>. (<emphasis>tool</emphasis>

1209

refers to the collection of tests; for example, after a run with

1210

<emphasis>--tool</emphasis> gdb, look for output files

1211

<filename>gdb.sum</filename> and

1212

1213

</sect2>

1214

</sect1>

1215

1216

1217

1218

<title>The files DejaGnu produces.</title>

1219

1220

<para>DejaGnu always writes two kinds of output files: summary

1221

logs and detailed logs. The contents of both of these are

1222

determined by your tests.</para>

1223

1224

<para>For troubleshooting, a third kind of output file is useful:

1225

use <option>--debug</option> to request an output file showing

1226

details of what <productname>Expect</productname> is doing

1227

internally.</para>

1228

1229

1230

<title>Summary File</title>

1231

1232

<para>DejaGnu always produces a summary output file

1233

<filename>tool.sum</filename>. This summary shows the names of

1234

all test files run; for each test file, one line of output from

1235

each <command>pass</command> command (showing status

1236

<emphasis>PASS</emphasis> or <emphasis>XPASS</emphasis>) or

1237

<command>fail</command> command (status

1238

<emphasis>FAIL</emphasis> or <emphasis>XFAIL</emphasis>);

1239

trailing summary statistics that count passing and failing tests

1240

(expected and unexpected); and the full pathname and version

1241

number of the tool tested. (All possible outcomes, and all

1242

errors, are always reflected in the summary output file,

1243

regardless of whether or not you specify

1244

1245

1246

<para>If any of your tests use the procedures

1247

<command>unresolved</command>, <command>unsupported</command>,

1248

or <command>runtested</command>, the summary output also

1249

tabulates the corresponding outcomes.</para>

1250

1251

<para>For example, after <command>runtest --tool

1252

binutils</command>, look for a summary log in

1253

<filename>binutils.sum</filename>. Normally, DejaGnu writes this

1254

file in your current working directory; use the

1255

<option>--outdir</option> option to select a different

1256

directory.</para>

1257

1258

1259

<title>Here is a short sample summary log</title>

1260

1261

1262

Test Run By rob on Mon May 25 21:40:57 PDT 1992

1263

=== gdb tests ===

1264

Running ./gdb.t00/echo.exp ...

1265

PASS: Echo test

1266

Running ./gdb.all/help.exp ...

1267

PASS: help add-symbol-file

1268

PASS: help aliases

1269

PASS: help breakpoint "bre" abbreviation

1270

FAIL: help run "r" abbreviation

1271

Running ./gdb.t10/crossload.exp ...

1272

PASS: m68k-elf (elf-big) explicit format; loaded

1273

XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types

1274

=== gdb Summary ===

1275

# of expected passes 5

1276

# of expected failures 1

1277

# of unexpected failures 1

1278

/usr/latest/bin/gdb version 4.6.5 -q

1279

</screen>

1280

</example>

1281

1282

</sect2>

1283

1284

1285

1286

1287

<para>DejaGnu also saves a detailed log file

1288

<filename>tool.log</filename>, showing any output generated by

1289

tests as well as the summary output. For example, after

1290

<command>runtest --tool binutils</command>, look for a detailed

1291

1292

writes this file in your current working directory; use the

1293

<option>--outdir</option> option to select a different

1294

directory.</para>

1295

1296

1297

1298

<title>Here is a brief example showing a detailed log for

1299

<productname>G++</productname> tests</title>

1300

1301

1302

Test Run By rob on Mon May 25 21:40:43 PDT 1992

1303

1304

=== g++ tests ===

1305

1306

--- Running ./g++.other/t01-1.exp ---

1307

PASS: operate delete

1308

1309

--- Running ./g++.other/t01-2.exp ---

1310

FAIL: i960 bug EOF

1311

p0000646.C: In function `int warn_return_1 ()':

1312

p0000646.C:109: warning: control reaches end of non-void function

1313

p0000646.C: In function `int warn_return_arg (int)':

1314

p0000646.C:117: warning: control reaches end of non-void function

1315

p0000646.C: In function `int warn_return_sum (int, int)':

1316

p0000646.C:125: warning: control reaches end of non-void function

1317

p0000646.C: In function `struct foo warn_return_foo ()':

1318

p0000646.C:132: warning: control reaches end of non-void function

1319

1320

--- Running ./g++.other/t01-4.exp ---

1321

FAIL: abort

1322

900403_04.C:8: zero width for bit-field `foo'

1323

--- Running ./g++.other/t01-3.exp ---

1324

FAIL: segment violation

1325

900519_12.C:9: parse error before `;'

1326

900519_12.C:12: Segmentation violation

1327

/usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal

1328

1329

=== g++ Summary ===

1330

1331

# of expected passes 1

1332

# of expected failures 3

1333

/usr/latest/bin/g++ version cygnus-2.0.1

1334

</screen>

1335

</example>

1336

1337

</sect2>

1338

1339

1340

<title>Debug Log File</title>

1341

1342

<para>With the <option>--debug</option> option, you can request

1343

a log file showing the output from

1344

<productname>Expect</productname> itself, running in debugging

1345

mode. This file (<filename>dbg.log</filename>, in the directory

1346

where you start <command>runtest</command>) shows each pattern

1347

<productname>Expect</productname> considers in analyzing test

1348

output.</para>

1349

1350

<para>This file reflects each <command>send</command> command,

1351

showing the string sent as input to the tool under test; and

1352

each <productname>Expect</productname> command, showing each

1353

pattern it compares with the tool output.</para>

1354

1355

1356

<title>The log messages begin with a message of the form</title>

1357

1358

1359

1360

expect: does {<symbol>tool output</symbol>} (spawn_id <symbol>n</symbol>)

1361

match pattern {<emphasis>expected pattern</emphasis>}?

1362

1363

</screen>

1364

</example>

1365

1366

<para>For every unsuccessful match,

1367

<productname>Expect</productname> issues a

1368

<emphasis>no</emphasis> after this message; if other patterns

1369

are specified for the same <productname>Expect</productname>

1370

command, they are reflected also, but without the first part of

1371

the message (<emphasis>expect... match pattern</emphasis>).</para>

1372

1373

<para>When <productname>Expect</productname> finds a match, the

1374

log for the successful match ends with <emphasis>yes</emphasis>,

1375

followed by a record of the <productname>Expect</productname>

1376

variables set to describe a successful match.</para>

1377

1378

1379

<title>Here is an excerpt from the debugging log for a

1380

1381

1382

1383

send: sent {break gdbme.c:34\n} to spawn id 6

1384

expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file

1385

gdbme.c, line 34.*$gdb$ $}? no

1386

{.*$gdb$ $}? no

1387

expect: does {} (spawn_id 0) match pattern {return} ? no

1388

{$y or n$ }? no

1389

{buffer_full}? no

1390

{virtual}? no

1391

{memory}? no

1392

{exhausted}? no

1393

{Undefined}? no

1394

{command}? no

1395

break gdbme.c:34

1396

Breakpoint 8 at 0x23d8: file gdbme.c, line 34.

1397

(gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8:

1398

file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern

1399

{Breakpoint.*at.* file gdbme.c, line 34.*$gdb$ $}? yes

1400

expect: set expect_out(0,start) {18}

1401

expect: set expect_out(0,end) {71}

1402

expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file

1403

gdbme.c, line 34.\r\n(gdb) }

1404

epect: set expect_out(spawn_id) {6}

1405

expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8

1406

at 0x23d8: file gdbme.c, line 34.\r\n(gdb) }

1407

PASS: 70 0 breakpoint line number in file

1408

</screen>

1409

</example>

1410

1411

<para>This example exhibits three properties of

1412

<productname>Expect</productname> and

1413

<productname>DejaGnu</productname> that might be surprising at

1414

first glance:</para>

1415

1416

1417

<listitem><para>Empty output for the first attempted match. The

1418

first set of attempted matches shown ran against the output

1419

<emphasis>{}</emphasis> --- that is, no

1420

output. <productname>Expect</productname> begins

1421

attempting to match the patterns supplied immediately; often,

1422

the first pass is against incomplete output (or completely

1423

before all output, as in this case).</para></listitem>

1424

1425

<listitem><para>Interspersed tool output. The beginning of

1426

the log entry for the second attempted match may be hard to

1427

spot: this is because the prompt <emphasis>{(gdb) }</emphasis>

1428

appears on the same line, just before the

1429

<emphasis>expect:</emphasis> that marks the beginning of the

1430

log entry.</para></listitem>

1431

1432

<listitem><para>Fail-safe patterns. Many of the patterns

1433

tested are fail-safe patterns provided by

1434

<productname>GDB</productname> testing utilities, to reduce

1435

possible indeterminacy. It is useful to anticipate potential

1436

variations caused by extreme system conditions

1437

(<productname>GDB</productname> might issue the message

1438

<emphasis>virtual memory exhausted</emphasis> in rare

1439

circumstances), or by changes in the tested program

1440

(<emphasis>Undefined command</emphasis> is the likeliest

1441

outcome if the name of a tested command changes).</para>

1442

1443

<para>The pattern <emphasis>{return}</emphasis> is a

1444

particularly interesting fail-safe to notice; it checks for an

1445

unexpected <keycap>RET</keycap> prompt. This may happen,

1446

for example, if the tested tool can filter output through a

1447

pager.</para>

1448

1449

<para>These fail-safe patterns (like the debugging log itself)

1450

are primarily useful while developing test scripts. Use the

1451

<command>error</command> procedure to make the actions for

1452

fail-safe patterns produce messages starting with

1453

<emphasis>ERROR</emphasis> on standard output, and in the

1454

detailed log file.</para></listitem>

1455

</itemizedlist>

1456

</sect2>

1457

</sect1>

1458

</chapter>

1459

1460

1461

<title>Customizing DejaGnu</title>

1462

1463

<para>The site configuration file, <filename>site.exp</filename>,

1464

captures configuration-dependent values and propagates them to the

1465

DejaGnu test environment using Tcl variables. This ties the

1466

DejaGnu test scripts into the <command>configure</command> and

1467

<command>make</command> programs. If this file is setup correctly,

1468

it is possible to execute a testsuite merely by typing

1469

<command>runtest</command>.</para>

1470

1471

<para>DejaGnu supports two <filename>site.exp</filename>

1472

files. The multiple instances of <filename>site.exp</filename> are

1473

loaded in a fixed order built into DejaGnu. The first file loaded

1474

is the local file <filename>site.exp</filename>, and then the

1475

optional global <filename>site.exp</filename> file as

1476

pointed to by the <symbol>DEJAGNU</symbol> environment

1477

variable.</para>

1478

1479

<para>There is an optional <emphasis>master</emphasis>

1480

<filename>site.exp</filename>, capturing configuration values that

1481

apply to DejaGnu across the board, in each configuration-specific

1482

subdirectory of the DejaGnu library directory.

1483

<command>runtest</command> loads these values first. The master

1484

<filename>site.exp</filename> contains the default values for all

1485

targets and hosts supported by DejaGnu. This master file is

1486

identified by setting the environment variable

1487

<symbol>DEJAGNU</symbol> to the name of the file. This is also

1488

refered to as the ``global'' config file.</para>

1489

1490

<para>Any directory containing a configured testsuite also has a

1491

local <filename>site.exp</filename>, capturing configuration values

1492

specific to the tool under test. Since <command>runtest</command>

1493

loads these values last, the individual test configuration can

1494

either rely on and use, or override, any of the global values from

1495

the global <filename>site.exp</filename> file.</para>

1496

1497

<para>You can usually generate or update the testsuite's local

1498

<filename>site.exp</filename> by typing <command>make

1499

site.exp</command> in the testsuite directory, after the test

1500

suite is configured.</para>

1501

1502

<para>You can also have a file in your home directory called

1503

<filename>.dejagnurc</filename>. This gets loaded first before the

1504

other config files. Usually this is used for personal stuff, like

1505

setting the <symbol>all_flag</symbol> so all the output gets

1506

printed, or your own verbosity levels. This file is usually

1507

restricted to setting command line options.</para>

1508

1509

<para>You can further override the default values in a

1510

user-editable section of any <filename>site.exp</filename>, or by

1511

setting variables on the <command>runtest</command> command

1512

line.</para>

1513

1514

1515

<title>Local Config File</title>

1516

1517

<para>It is usually more convenient to keep these <emphasis>manual

1518

overrides</emphasis> in the <filename>site.exp</filename>

1519

local to each test directory, rather than in the global

1520

<filename>site.exp</filename> in the installed DejaGnu

1521

library. This file is mostly for supplying tool specific info

1522

that is required by the testsuite.</para>

1523

1524

<para>All local <filename>site.exp</filename> files have

1525

two sections, separated by comment text. The first section is

1526

the part that is generated by <command>make</command>. It is

1527

essentially a collection of Tcl variable definitions based on

1528

<filename>Makefile</filename> environment variables. Since they

1529

are generated by <command>make</command>, they contain the

1530

values as specified by <command>configure</command>. (You can

1531

also customize these values by using the <option>--site</option>

1532

option to <command>configure</command>.) In particular, this

1533

section contains the <filename>Makefile</filename>

1534

variables for host and target configuration data. Do not edit

1535

this first section; if you do, your changes are replaced next

1536

time you run <command>make</command>.</para>

1537

1538

1539

<title>The first section starts with</title>

1540

1541

1542

## these variables are automatically generated by make ##

1543

# Do not edit here. If you wish to override these values

1544

# add them to the last section

1545

</programlisting>

1546

</example>

1547

1548

<para>In the second section, you can override any default values

1549

(locally to DejaGnu) for all the variables. The second section

1550

can also contain your preferred defaults for all the command

1551

line options to <command>runtest</command>. This allows you to

1552

easily customize <command>runtest</command> for your preferences

1553

in each configured test-suite tree, so that you need not type

1554

options repeatedly on the command line. (The second section may

1555

also be empty, if you do not wish to override any defaults.)</para>

1556

1557

1558

<title>The first section ends with this line</title>

1559

1560

1561

## All variables above are generated by configure. Do Not Edit ##

1562

</programlisting>

1563

</example>

1564

1565

<para>You can make any changes under this line. If you wish to

1566

redefine a variable in the top section, then just put a

1567

duplicate value in this second section. Usually the values

1568

defined in this config file are related to the configuration of

1569

the test run. This is the ideal place to set the variables

1570

<symbol>host_triplet</symbol>, <symbol>build_triplet</symbol>,

1571

<symbol>target_triplet</symbol>. All other variables are tool

1572

dependant, i.e., for testing a compiler, the value for

1573

<symbol>CC</symbol> might be set to a freshly built binary, as

1574

opposed to one in the user's path.</para>

1575

1576

<para>Here's an example local site.exp file, as used for

1577

<productname>GCC/G++</productname> testing.</para>

1578

1579

1580

<title>Local Config File</title>

1581

1582

1583

## these variables are automatically generated by make ##

1584

# Do not edit here. If you wish to override these values

1585

# add them to the last section

1586

set rootme "/build/devo-builds/i586-pc-linux-gnulibc1/gcc"

1587

set host_triplet i586-pc-linux-gnulibc1

1588

set build_triplet i586-pc-linux-gnulibc1

1589

set target_triplet i586-pc-linux-gnulibc1

1590

set target_alias i586-pc-linux-gnulibc1

1591

set CFLAGS ""

1592

set CXXFLAGS "-isystem /build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libio -isystem $srcdir/../libg++/src -isystem $srcdir/../libio -isystem $srcdir/../libstdc++ -isystem $srcdir/../libstdc++/stl -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libg++ -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libstdc++"

1593

append LDFLAGS " -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../ld"

1594

set tmpdir /build/devo-builds/i586-pc-linux-gnulibc1/gcc/testsuite

1595

set srcdir "${srcdir}/testsuite"

1596

## All variables above are generated by configure. Do Not Edit ##

1597

1598

</programlisting>

1599

</example>

1600

1601

<para>This file defines the required fields for a local config

1602

file, namely the three config triplets, and the srcdir. It also

1603

defines several other Tcl variables that are used exclusivly by

1604

the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS

1605

are supplied by DejaGnu itself for cross testing, but to test a

1606

compiler, GCC needs to manipulate these itself.</para>

1607

1608

</sect1>

1609

1610

<title>Global Config File</title>

1611

1612

<para>The master config file is where all the target specific

1613

config variables for a whole site get set. The idea is

1614

that for a centralized testing lab where people have to share a

1615

target between multiple developers. There are settings for both

1616

remote targets and remote hosts. Here's an example of a Master

1617

Config File (also called the Global config file) for a

1618

<emphasis>canadian cross</emphasis>. A canadian cross is when

1619

you build and test a cross compiler on a machine other than the

1620

one it's to be hosted on.</para>

1621

1622

<para>Here we have the config settings for our California

1623

office. Note that all config values are site dependant. Here we

1624

have two sets of values that we use for testing m68k-aout cross

1625

compilers. As both of these target boards has a different

1626

debugging protocol, we test on both of them in sequence.</para>

1627

1628

1629

<title>Global Config file</title>

1630

1631

1632

1633

# Make sure we look in the right place for the board description files.

1634

if ![info exists boards_dir] {

1635

set boards_dir {}

1636

}

1637

lappend boards_dir "/nfs/cygint/s1/cygnus/dejagnu/boards"

1638

1639

verbose "Global Config File: target_triplet is $target_triplet" 2

1640

global target_list

1641

1642

case "$target_triplet" in {

1643

{ "native" } {

1644

set target_list "unix"

1645

}

1646

{ "sparc64-*elf" } {

1647

set target_list "sparc64-sim"

1648

}

1649

{ "mips-*elf" } {

1650

set target_list "mips-sim wilma barney"

1651

}

1652

{ "mips-lsi-elf" } {

1653

set target_list "mips-lsi-sim{,soft-float,el}"

1654

}

1655

{ "sh-*hms" } {

1656

set target_list { "sh-hms-sim" "bloozy" }

1657

}

1658

}

1659

</programlisting>

1660

</example>

1661

1662

<para>In this case, we have support for several cross compilers,

1663

that all run on this host. For testing on operating systems that

1664

don't support Expect, DejaGnu can be run on the local build

1665

machine, and it can connect to the remote host and run all the

1666

tests for this cross compiler on that host. All the remote OS

1667

requires is a working telnetd.</para>

1668

1669

<para>As you can see, all one does is set the variable

1670

<symbol>target_list</symbol> to the list of targets and options to

1671

test. The simple settings, like for

1672

<emphasis>sparc64-elf</emphasis> only require setting the name of

1673

the single board config file. The <emphasis>mips-elf</emphasis>

1674

target is more complicated. Here it sets the list to three target

1675

boards. One is the default mips target, and both

1676

<emphasis>wilma</emphasis> <emphasis>barney</emphasis> are

1677

symbolic names for other mips boards. Symbolic names are covered

1678

in the <xref linkend=addboard> chapter. The more complicated

1679

example is the one for <emphasis>mips-lsi-elf</emphasis>. This one

1680

runs the tests with multiple iterations using all possible

1681

combinations of the <option>--soft-float</option> and the

1682

<option>--el</option> (little endian) option. Needless to say,

1683

this last feature is mostly compiler specific.</para>

1684

1685

</sect1>

1686

1687

1688

<title>Board Config File</title>

1689

1690

<para>The board config file is where board specfic config data

1691

is stored. A board config file contains all the higher-level

1692

configuration settings. There is a rough inheritance scheme, where it is

1693

possible to base a new board description file on an existing one. There

1694

are also collections of custom procedures for common environments. For

1695

more information on adding a new board config file, go to the <xref

1696

linkend=addboard> chapter. </para>

1697

1698

<para>An example board config file for a GNU simulator is as

1699

follows. <function>set_board_info</function> is a procedure that sets the

1700

field name to the specified value. The procedures in square brackets

1701

<emphasis>[]</emphasis> are <emphasis>helper procedures</emphasis>. Thes

1702

are used to find parts of a tool chain required to build an executable

1703

image that may reside in various locations. This is mostly of use for

1704

when the startup code, the standard C lobraries, or the tool chain itself

1705

is part of your build tree.</para>

1706

1707

1708

<title>Board Config File</title>

1709

1710

1711

# This is a list of toolchains that are supported on this board.

1712

set_board_info target_install {sparc64-elf}

1713

1714

# Load the generic configuration for this board. This will define any

1715

# routines needed by the tool to communicate with the board.

1716

load_generic_config "sim"

1717

1718

# We need this for find_gcc and *_include_flags/*_link_flags.

1719

load_base_board_description "basic-sim"

1720

1721

# Use long64 by default.

1722

process_multilib_options "long64"

1723

1724

setup_sim sparc64

1725

1726

# We only support newlib on this target. We assume that all multilib

1727

# options have been specified before we get here.

1728

set_board_info compiler "[find_gcc]"

1729

set_board_info cflags "[libgloss_include_flags] [newlib_include_flags]"

1730

set_board_info ldflags "[libgloss_link_flags] [newlib_link_flags]"

1731

# No linker script.

1732

set_board_info ldscript "";

1733

1734

# Used by a few gcc.c-torture testcases to delimit how large the

1735

# stack can be.

1736

set_board_info gcc,stack_size 16384

1737

# The simulator doesn't return exit statuses and we need to indicate this

1738

# the standard GCC wrapper will work with this target.

1739

set_board_info needs_status_wrapper 1

1740

# We can't pass arguments to programs.

1741

set_board_info noargs 1

1742

</programlisting>

1743

</example>

1744

1745

<para>There are five helper procedures used in this example. The first

1746

one, <function>find gcc</function> looks for a copy of the GNU compiler in

1747

your build tree, or it uses the one in your path. This will also return

1748

the proper transformed name for a cross compiler if you whole build tree

1749

is configured for one. The next helper procedures are

1750

<function>libgloss_include_flags</function> &

1751

<function>libgloss_link_flags</function>. These return the proper flags to

1752

compiler and link an executable image using <xref

1753

linkend=libgloss>, the GNU BSP (Board Support Package). The final

1754

procedures are <function>newlib_include_flag</function> &

1755

<function>newlib_include_flag</function>. These find the Newlib C

1756

library, which is a reentrant standard C library for embedded systems

1757

comprising of non GPL'd code.</para>

1758

1759

</sect1>

1760

1761

1762

<title>Remote Host Testing</title>

1763

1764

<note><para>Thanks to Dj Delorie for the original paper that

1765

this section is based on.</para></note>

1766

1767

<para>DejaGnu also supports running the tests on a remote

1768

host. To set this up, the remote host needs an ftp server, and a

1769

telnet server. Currently foreign operating systems used as

1770

remote hosts are VxWorks, VRTX, DOS/Windows 3.1, MacOS and Windows.</para>

1771

1772

<para>The recommended source for a Windows-based FTP

1773

server is to get IIS (either IIS 1 or Personal Web Server) from

1774

<ulink

1775

URL="http://www.microsoft.com">http://www.microsoft.com</ulink>.

1776

When you install it, make sure you install the FTP server - it's

1777

not selected by default. Go into the IIS manager and change the

1778

FTP server so that it does not allow anonymous FTP. Set the home

1779

directory to the root directory (i.e. c:\) of a suitable

1780

drive. Allow writing via FTP.</para>

1781

1782

<para>It will create an account like IUSR_FOOBAR where foobar is

1783

the name of your machine. Go into the user editor and give that

1784

account a password that you don't mind hanging around in the

1785

clear (i.e. not the same as your admin or personal

1786

passwords). Also, add it to all the various permission groups.</para>

1787

1788

<para>You'll also need a telnet server. For Windows, go

1789

to the <ulink URL="http://ataman.com">Ataman</ulink> web site,

1790

pick up the Ataman Remote Logon Services for Windows, and

1791

install it. You can get started on the eval period anyway. Add

1792

IUSR_FOOBAR to the list of allowed users, set the HOME directory

1793

to be the same as the FTP default directory. Change the Mode

1794

prompt to simple.</para>

1795

1796

<para>Ok, now you need to pick a directory name to do all the

1797

testing in. For the sake of this example, we'll call it piggy

1798

(i.e. c:\piggy). Create this directory.</para>

1799

1800

<para>You'll need a unix machine. Create a directory for the

1801

scripts you'll need. For this example, we'll use

1802

/usr/local/swamp/testing. You'll need to have a source tree

1803

somewhere, say /usr/src/devo. Now, copy some files from

1804

releng's area in SV to your machine:</para>

1805

1806

1807

<title>Remote host setup</title>

1808

1809

1810

cd /usr/local/swamp/testing

1811

mkdir boards

1812

scp darkstar.welcomehome.org:/dejagnu/cst/bin/MkTestDir .

1813

scp darkstar.welcomehome.org:/dejagnu/site.exp .

1814

scp darkstar.welcomehome.org:/dejagnu/boards/useless98r2.exp boards/foobar.exp

1815

export DEJAGNU=/usr/local/swamp/testing/site.exp

1816

1817

</screen>

1818

</example>

1819

1820

<para>You must edit the boards/foobar.exp file to reflect your

1821

machine; change the hostname (foobar.com), username

1822

(iusr_foobar), password, and ftp_directory (c:/piggy) to match

1823

what you selected.</para>

1824

1825

<para>Edit the global <filename> site.exp</filename> to reflect your

1826

boards directory:</para>

1827

1828

1829

<title>Add The Board Directory</title>

1830

1831

1832

lappend boards_dir "/usr/local/swamp/testing/boards"

1833

</programlisting>

1834

</example>

1835

1836

<para>Now run MkTestDir, which is in the contrib

1837

directory. The first parameter is the toolchain prefix, the

1838

second is the location of your devo tree. If you are testing a

1839

cross compiler (ex: you have sh-hms-gcc.exe in your PATH on

1840

the PC), do something like this:</para>

1841

1842

1843

<title>Setup Cross Remote Testing</title>

1844

1845

1846

./MkTestDir sh-hms /usr/dejagnu/src/devo

1847

</programlisting>

1848

</example>

1849

1850

<para>If you are testing a native PC compiler (ex: you have

1851

gcc.exe in your PATH on the PC), do this:</para>

1852

1853

1854

<title>Setup Native Remote Testing</title>

1855

1856

1857

./MkTestDir '' /usr/dejagnu/src/devo

1858

</programlisting>

1859

</example>

1860

1861

<para>To test the setup, <command>ftp</command> to your PC

1862

using the username (iusr_foobar) and password you selected. CD

1863

to the test directory. Upload a file to the PC. Now telnet to

1864

your PC using the same username and password. CD to the test

1865

directory. Make sure the file is there. Type "set" and/or "gcc

1866

-v" (or sh-hms-gcc -v) and make sure the default PATH contains

1867

the installation you want to test.</para>

1868

1869

1870

<title>Run Test Remotely</title>

1871

1872

1873

cd /usr/local/swamp/testing

1874

make -k -w check RUNTESTFLAGS="--host_board foobar --target_board foobar -v -v" > check.out 2>&1

1875

</programlisting>

1876

</example>

1877

1878

<para>To run a specific test, use a command like this (for

1879

this example, you'd run this from the gcc directory that

1880

MkTestDir created):</para>

1881

1882

1883

<title>Run a Test Remotely</title>

1884

1885

1886

make check RUNTESTFLAGS="--host_board sloth --target_board sloth -v compile.exp=921202-1.c"

1887

</programlisting>

1888

</example>

1889

1890

<para>Note: if you are testing a cross-compiler, put in the

1891

correct target board. You'll also have to download more .exp

1892

files and modify them for your local configuration. The -v's

1893

are optional.</para>

1894

1895

</sect1>

1896

1897

1898

<title>Config File Values</title>

1899

1900

<para>DejaGnu uses a named array in Tcl to hold all the info for

1901

each machine. In the case of a canadian cross, this means host

1902

information as well as target information. The named array is

1903

called <symbol>target_info</symbol>, and it has two indices. The

1904

following fields are part of the array.</para>

1905

1906

1907

<title>Command Line Option Variables</title>

1908

1909

<para>In the user editable second section of the <xref

1910

linkend=personal> you can not only override the configuration

1911

variables captured in the first section, but also specify

1912

default values for all on the <command>runtest</command>

1913

command line options. Save for <option>--debug</option>,

1914

<option>--help</option>, and <option>--version</option>, each

1915

command line option has an associated Tcl variable. Use the

1916

Tcl <command>set</command> command to specify a new default

1917

value (as for the configuration variables). The following

1918

table describes the correspondence between command line

1919

options and variables you can set in

1920

<filename>site.exp</filename>. <xref linkend=invoking>, for

1921

explanations of the command-line options.</para>

1922

1923

1924

<title>Tcl Variables For Command Line Options</title>

1925

1926

1927

1928

<entry>runtest</entry><entry>Tcl</entry>

1929

<entry>option</entry><entry>variable</entry><entry>description</entry>

1930

</row></thead>

1931

<tbody>

1932

1933

<row>

1934

1935

1936

<entry>display all test results if set</entry>

1937

</row>

1938

1939

<row>

1940

1941

1942

<entry>set the default baud rate to something other than

1943

9600.</entry>

1944

</row>

1945

1946

<row>

1947

<entry>--connect</entry>

1948

<entry>connectmode</entry>

1949

<entry><command>rlogin</command>,

1950

<command>telnet</command>, <command>rsh</command>,

1951

<command>kermit</command>, <command>tip</command>, or

1952

<command>mondfe</command></entry>

1953

</row>

1954

1955

<row>

1956

<entry>--outdir</entry>

1957

<entry>outdir</entry>

1958

<entry>directory for <filename>tool.sum</filename> and

1959

1960

</row>

1961

1962

<row>

1963

<entry>--objdir</entry>

1964

<entry>objdir</entry>

1965

<entry>directory for pre-compiled binaries</entry>

1966

</row>

1967

1968

<row>

1969

<entry>--reboot</entry>

1970

<entry>reboot</entry>

1971

<entry>reboot the target if set to

1972

<emphasis>"1"</emphasis>; do not reboot if set to

1973

<emphasis>"0"</emphasis> (the default).</entry>

1974

</row>

1975

1976

<row>

1977

<entry>--srcdir</entry>

1978

<entry>srcdir</entry>

1979

<entry>directory of test subdirectories</entry>

1980

</row>

1981

1982

<row>

1983

<entry>--strace</entry>

1984

<entry>tracelevel</entry>

1985

<entry>a number: Tcl trace depth</entry>

1986

</row>

1987

1988

<row>

1989

1990

1991

<entry>name of tool to test; identifies init, test subdir</entry>

1992

</row>

1993

1994

<row>

1995

<entry>--verbose</entry>

1996

<entry>verbose</entry>

1997

<entry>verbosity level. As option, use multiple times; as

1998

variable, set a number, 0 or greater.</entry>

1999

</row>

2000

2001

<row>

2002

<entry>--target</entry>

2003

<entry>target_triplet</entry>

2004

<entry>The canonical configuration string for the target.</entry>

2005

</row>

2006

2007

<row>

2008

2009

<entry>host_triplet</entry>

2010

<entry>The canonical configuration string for the host.</entry>

2011

</row>

2012

2013

<row>

2014

<entry>--build</entry>

2015

<entry>build_triplet</entry>

2016

<entry>The canonical configuration string for the build

2017

host.</entry>

2018

</row>

2019

2020

<row>

2021

2022

<entry>address</entry>

2023

<entry>Email the output log to the specified address.</entry>

2024

</row>

2025

2026

</tbody>

2027

</tgroup>

2028

</table>

2029

</para>

2030

2031

</sect2>

2032

2033

2034

<title>Personal Config File</title>

2035

2036

<para>The personal config file is used to customize

2037

<command>runtest's</command> behaviour for each person. It's

2038

typically used to set the user prefered setting for verbosity,

2039

and any experimental Tcl procedures. My personal

2040

<filename>~/.dejagnurc</filename> file looks like:</para>

2041

2042

2043

<title>Personal Config File</title>

2044

2045

2046

set all_flag 1

2047

set RLOGIN /usr/ucb/rlogin

2048

set RSH /usr/local/sbin/ssh

2049

</programlisting>

2050

</example>

2051

2052

<para>Here I set <symbol>all_flag</symbol> so I see all the test

2053

cases that PASS along with the ones that FAIL. I also set

2054

<symbol>RLOGIN</symbol> to the BSD version. I have

2055

<productname>Kerberos</productname> installed, and when I rlogin

2056

to a target board, it usually isn't supported. So I use the non

2057

secure version rather than the default that's in my path. I also

2058

set <symbol>RSH</symbol> to the <productname>SSH</productname>

2059

secure shell, as rsh is mostly used to test unix

2060

machines within a local network here.</para>

2061

2062

</sect2>

2063

</sect1>

2064

2065

</chapter>

2066

2067

2068

<title>Extending DejaGnu</title>

2069

2070

2071

<title>Adding A New Testsuite</title>

2072

2073

<para>The testsuite for a new tool should always be located in that tools

2074

source directory. DejaGnu require the directory be named

2075

<filename>testsuite</filename>. Under this directory, the test cases go

2076

in a subdirectory whose name begins with the tool name. For example, for

2077

a tool named <emphasis>flubber</emphasis>, each subdirectory containing

2078

testsuites must start with <emphasis>"flubber."</emphasis>.</para>

2079

2080

</sect1>

2081

2082

2083

<title>Adding A New Tool</title>

2084

2085

<para>In general, the best way to learn how to write (code or even prose)

2086

is to read something similar. This principle applies to test cases and

2087

to testsuites. Unfortunately, well-established testsuites have a way

2088

of developing their own conventions: as test writers become more

2089

experienced with DejaGnu and with Tcl, they accumulate more utilities,

2090

and take advantage of more and more features of

2091

<productname>Expect</productname> and <productname>Tcl</productname> in

2092

general.</para>

2093

2094

<para>Inspecting such established testsuites may make the prospect of

2095

creating an entirely new testsuite appear overwhelming. Nevertheless,

2096

it is quite straightforward to get a new testsuite going.</para>

2097

2098

<para>There is one testsuite that is guaranteed not to grow more

2099

elaborate over time: both it and the tool it tests were created expressly

2100

to illustrate what it takes to get started with DejaGnu. The

2101

<filename>example/</filename> directory of the DejaGnu distribution

2102

contains both an interactive tool called <command>calc</command>, and a

2103

testsuite for it. Reading this testsuite, and experimenting with it,

2104

is a good way to supplement the information in this section. (Thanks to

2105

Robert Lupton for creating calc and its testsuite---and also the first

2106

version of this section of the manual!)</para>

2107

2108

<para>To help orient you further in this task, here is an outline of the

2109

steps to begin building a testsuite for a program example.</para>

2110

2111

2112

2113

<listitem><para>Create or select a directory to contain your new

2114

collection of tests. Change into that directory (shown here as

2115

<filename>testsuite</filename>):</para>

2116

2117

<para>Create a <filename>configure.in</filename> file in this directory,

2118

to control configuration-dependent choices for your tests. So far as

2119

DejaGnu is concerned, the important thing is to set a value for the

2120

variable <symbol>target_abbrev</symbol>; this value is the link to the

2121

init file you will write soon. (For simplicity, we assume the

2122

environment is Unix, and use <emphasis>unix</emphasis> as the

2123

value.)</para>

2124

2125

<para>What else is needed in <filename>configure.in</filename> depends on

2126

the requirements of your tool, your intended test environments, and which

2127

configure system you use. This example is a minimal configure.in for use

2128

with <productname>GNU Autoconf</productname>. </para></listitem>

2129

2130

<listitem><para>Create <filename>Makefile.in</filename> (if you are using

2131

Autoconf), or <filename>Makefile.am</filename>(if you are using

2132

Automake), the source file used by configure to build your

2133

<filename>Makefile</filename>. If you are using GNU Automake.just add the

2134

keyword <emphasis>dejagnu</emphasis> to the

2135

<emphasis>AUTOMAKE_OPTIONS</emphasis> variable in your

2136

<filename>Makefile.am</filename> file. This will add all the Makefile

2137

support needed to run DejaGnu, and support the <xref linkend=makecheck>

2138

target.</para>

2139

2140

<para>You also need to include two targets important to DejaGnu:

2141

<emphasis>check</emphasis>, to run the tests, and

2142

<emphasis>site.exp</emphasis>, to set up the Tcl copies of

2143

configuration-dependent values. This is called the <xref linkend=local>

2144

The check target must run the <command>runtest</command> program to

2145

execute the tests.</para>

2146

2147

<para>The <filename>site.exp</filename> target should usually set up

2148

(among other things) the $tool variable for the name of your program. If

2149

the local site.exp file is setup correctly, it is possible to execute the

2150

tests by merely typing <command>runtest</command> on the command

2151

line.</para>

2152

2153

2154

<title>Sample Makefile.in Fragment</title>

2155

2156

2157

# Look for a local version of DejaGnu, otherwise use one in the path

2158

RUNTEST = `if test -f $(top_srcdir)/../dejagnu/runtest; then \

2159

echo $(top_srcdir) ../dejagnu/runtest; \

2160

else \

2161

echo runtest; \

2162

fi`

2163

2164

# The flags to pass to runtest

2165

RUNTESTFLAGS =

2166

2167

# Execute the tests

2168

check: site.exp all

2169

$(RUNTEST) $(RUNTESTFLAGS) \

2170

--tool <symbol>${example}</symbol> --srcdir $(srcdir)

2171

2172

# Make the local config file

2173

site.exp: ./config.status Makefile

2174

@echo "Making a new config file..."

2175

-@rm -f ./tmp?

2176

@touch site.exp

2177

2178

-@mv site.exp site.bak

2179

@echo "## these variables are automatically\

2180

generated by make ##" > ./tmp0

2181

@echo "# Do not edit here. If you wish to\

2182

override these values" >> ./tmp0

2183

@echo "# add them to the last section" >> ./tmp0

2184

@echo "set host_os ${host_os}" >> ./tmp0

2185

@echo "set host_alias ${host_alias}" >> ./tmp0

2186

@echo "set host_cpu ${host_cpu}" >> ./tmp0

2187

@echo "set host_vendor ${host_vendor}" >> ./tmp0

2188

@echo "set target_os ${target_os}" >> ./tmp0

2189

@echo "set target_alias ${target_alias}" >> ./tmp0

2190

@echo "set target_cpu ${target_cpu}" >> ./tmp0

2191

@echo "set target_vendor ${target_vendor}" >> ./tmp0

2192

@echo "set host_triplet ${host_canonical}" >> ./tmp0

2193

@echo "set target_triplet ${target_canonical}">>./tmp0

2194

@echo "set tool binutils" >> ./tmp0

2195

@echo "set srcdir ${srcdir}" >> ./tmp0

2196

@echo "set objdir `pwd`" >> ./tmp0

2197

@echo "set <symbol>${examplename}</symbol> <symbol>${example}</symbol>" >> ./tmp0

2198

@echo "## All variables above are generated by\

2199

configure. Do Not Edit ##" >> ./tmp0

2200

@cat ./tmp0 > site.exp

2201

@sed < site.bak \

2202

-e '1,/^## All variables above are.*##/ d' \

2203

>> site.exp

2204

-@rm -f ./tmp?

2205

2206

</programlisting>

2207

</example>

2208

</listitem>

2209

2210

<listitem><para>Create a directory (in <filename>testsuite</filename>)

2211

called <filename>config</filename>. Make a <emphasis>Tool Init

2212

File</emphasis> in this directory. Its name must start with the

2213

<symbol>target_abbrev</symbol> value, or be named

2214

<filename>default.exp</filename> so call it

2215

<filename>config/unix.exp</filename> for our Unix based example. This

2216

is the file that contains the target-dependent procedures.

2217

Fortunately, on Unix, most of them do not have to do very much in

2218

order for <command>runtest</command> to run.</para>

2219

2220

<para>If the program being tested is not interactive, you can get

2221

away with this minimal <filename>unix.exp</filename> to begin

2222

with:</para>

2223

2224

2225

<title>Simple Batch Program Tool Init File</title>

2226

2227

2228

2229

proc foo_exit {} {}

2230

proc foo_version {} {}

2231

2232

</programlisting>

2233

</example>

2234

2235

<para>If the program being tested is interactive, however, you might

2236

as well define a <emphasis>start</emphasis> routine and invoke it by

2237

using an init file like this:</para>

2238

2239

2240

<title>Simple Interactive Program Tool Init File</title>

2241

2242

2243

2244

proc foo_exit {} {}

2245

proc foo_version {} {}

2246

2247

proc foo_start {} {

2248

global ${examplename}

2249

spawn ${examplename}

2250

expect {

2251

-re "" {}

2252

}

2253

}

2254

2255

# Start the program running we want to test

2256

foo_start

2257

2258

</programlisting>

2259

</example>

2260

</listitem>

2261

2262

<listitem><para>Create a directory whose name begins with your tool's

2263

name, to contain tests. For example, if your tool's name is

2264

<emphasis>gcc</emphasis>, then the directories all need to start with

2265

2266

2267

<listitem><para>Create a sample test file. Its name must end with

2268

<filename>.exp</filename>. You can use

2269

<filename>first-try.exp</filename>. To begin with, just write there a

2270

line of Tcl code to issue a message.</para>

2271

2272

2273

<title>Testing A New Tool Config</title>

2274

2275

2276

2277

send_user "Testing: one, two...\n"

2278

2279

</programlisting>

2280

</example>

2281

</listitem>

2282

2283

<listitem><para>Back in the <filename>testsuite</filename> (top

2284

level) directory, run <command>configure</command>. Typically you do

2285

this while in the build directory. You may have to specify more of a

2286

path, if a suitable configure is not available in your execution

2287

path.</para></listitem>

2288

2289

<listitem><para>e now ready to triumphantly type <command>make

2290

check</command> or <command>runtest</command>. You should see

2291

something like this:</para>

2292

2293

2294

<title>Example Test Case Run</title>

2295

2296

2297

Test Run By rhl on Fri Jan 29 16:25:44 EST 1993

2298

2299

=== example tests ===

2300

2301

Running ./example.0/first-try.exp ...

2302

Testing: one, two...

2303

2304

=== example Summary ===

2305

2306

</screen>

2307

</example>

2308

2309

<para>There is no output in the summary, because so far the example

2310

does not call any of the procedures that establish a test

2311

outcome.</para></listitem>

2312

2313

<listitem><para>Write some real tests. For an interactive tool, you

2314

should probably write a real exit routine in fairly short order. In

2315

any case, you should also write a real version routine

2316

soon. </para></listitem>

2317

2318

</itemizedlist>

2319

2320

</sect1>

2321

2322

2323

<title>Adding A New Target</title>

2324

2325

<para>DejaGnu has some additional requirements for target support, beyond

2326

the general-purpose provisions of configure. DejaGnu must actively

2327

communicate with the target, rather than simply generating or managing

2328

code for the target architecture. Therefore, each tool requires an

2329

initialization module for each target. For new targets, you must supply

2330

a few Tcl procedures to adapt DejaGnu to the target. This permits

2331

DejaGnu itself to remain target independent.</para>

2332

2333

<para>Usually the best way to write a new initialization module is to

2334

edit an existing initialization module; some trial and error will be

2335

required. If necessary, you can use the @samp{--debug} option to see what

2336

is really going on.</para>

2337

2338

<para>When you code an initialization module, be generous in printing

2339

information controlled by the <function>verbose</function>

2340

procedure.</para>

2341

2342

<para>For cross targets, most of the work is in getting the

2343

communications right. Communications code (for several situations

2344

involving IP networks or serial lines) is available in a DejaGnu library

2345

file.</para>

2346

2347

<para>If you suspect a communication problem, try running the connection

2348

interactively from <productname>Expect</productname>. (There are three

2349

ways of running <productname>Expect</productname> as an interactive

2350

interpreter. You can run <productname>Expect</productname> with no

2351

arguments, and control it completely interactively; or you can use

2352

<command>expect -i</command> together with other command-line options and

2353

arguments; or you can run the command <command>interpreter</command> from

2354

any <productname>Expect</productname> procedure. Use

2355

<command>return</command> to get back to the calling procedure (if any),

2356

or <command>return -tcl</command> to make the calling procedure itself

2357

return to its caller; use <command>exi</command>t or end-of-file to leave

2358

Expect altogether.) Run the program whose name is recorded in

2359

<symbol>$connectmode</symbol>, with the arguments in

2360

<symbol>$targetname</symbol>, to establish a connection. You should at

2361

least be able to get a prompt from any target that is physically

2362

connected.</para>

2363

2364

</sect1>

2365

2366

2367

<title>Adding A New Board</title>

2368

2369

<para>Adding a new board consists of creating a new board config

2370

file. Examples are in

2371

<filename>dejagnu/baseboards</filename>. Usually to make a new

2372

board file, it's easiest to copy an existing one. It is also

2373

possible to have your file be based on a

2374

<emphasis>baseboard</emphasis> file with only one or two

2375

changes needed. Typically, this can be as simple as just

2376

changing the linker script. Once the new baseboard file is done,

2377

add it to the <symbol>boards_DATA</symbol> list in the

2378

<filename>dejagnu/baseboards/Makefile.am</filename>, and regenerate the

2379

Makefile.in using automake. Then just rebuild and install DejaGnu. You

2380

can test it by:</para>

2381

2382

<para>There is a crude inheritance scheme going on with board files, so

2383

you can include one board file into another, The two main procedures used

2384

to do this are <function>load_generic_config</function> and

2385

<function>load_base_board_description</function>. The generic config file

2386

contains other procedures used for a certain class of target. The

2387

board description file is where the board specfic settings go. Commonly

2388

there are similar target environments with just different

2389

processors.</para>

2390

2391

2392

<title>Testing a New Board Config File</title>

2393

2394

2395

make check RUNTESTFLAGS="--target_board=<emphasis>newboardfile</emphasis>".

2396

</screen>

2397

</example>

2398

2399

<para>Here's an example of a board config file. There are

2400

several <emphasis>helper procedures</emphasis> used in this

2401

example. A helper procedure is one that look for a tool of files

2402

in commonly installed locations. These are mostly used when

2403

testing in the build tree, because the executables to be tested

2404

are in the same tree as the new dejagnu files. The helper

2405

procedures are the ones in square braces

2406

<emphasis>[]</emphasis>, which is the Tcl execution characters.</para>

2407

2408

2409

<title>Example Board Config File</title>

2410

2411

2412

2413

# Load the generic configuration for this board. This will define a basic

2414

# set of routines needed by the tool to communicate with the board.

2415

load_generic_config "sim"

2416

2417

# basic-sim.exp is a basic description for the standard Cygnus simulator.

2418

load_base_board_description "basic-sim"

2419

2420

# The compiler used to build for this board. This has *nothing* to do

2421

# with what compiler is tested if we're testing gcc.

2422

set_board_info compiler "[find_gcc]"

2423

2424

# We only support newlib on this target.

2425

# However, we include libgloss so we can find the linker scripts.

2426

set_board_info cflags "[newlib_include_flags] [libgloss_include_flags]"

2427

set_board_info ldflags "[newlib_link_flags]"

2428

2429

# No linker script for this board.

2430

set_board_info ldscript "-Tsim.ld";

2431

2432

# The simulator doesn't return exit statuses and we need to indicate this.

2433

set_board_info needs_status_wrapper 1

2434

2435

# Can't pass arguments to this target.

2436

set_board_info noargs 1

2437

2438

# No signals.

2439

set_board_info gdb,nosignals 1

2440

2441

# And it can't call functions.

2442

set_board_info gdb,cannot_call_functions 1

2443

2444

</programlisting>

2445

</example>

2446

2447

</sect1>

2448

2449

2450

<title>Board Config File Values</title>

2451

2452

<para>These fields are all in the <symbol>board_info</symbol> These are

2453

all set by using the <function>set_board_info</function> procedure. The

2454

parameters are the field name, followed by the value to set the field

2455

to.</para>

2456

2457

2458

<title>Common Board Info Fields</title>

2459

2460

2461

2462

<entry>Field</entry>

2463

<entry>Sample Value</entry>

2464

<entry>Description</entry>

2465

</row></thead>

2466

<tbody>

2467

2468

<row>

2469

<entry>compiler</entry>

2470

2471

<entry>The path to the compiler to use.</entry>

2472

</row>

2473

2474

<row>

2475

<entry>cflags</entry>

2476

2477

<entry>Compilation flags for the compiler.</entry>

2478

</row>

2479

2480

<row>

2481

<entry>ldflags</entry>

2482

<entry>"[libgloss_link_flags] [newlib_link_flags]"</entry>

2483

<entry>Linking flags for the compiler.</entry>

2484

</row>

2485

2486

<row>

2487

<entry>ldscript</entry>

2488

2489

<entry>The linker script to use when cross compiling.</entry>

2490

</row>

2491

2492

<row>

2493

2494

2495

<entry>Any additional libraries to link in.</entry>

2496

</row>

2497

2498

<row>

2499

<entry>shell_prompt</entry>

2500

<entry>"cygmon>"</entry>

2501

<entry>The command prompt of the remote shell.</entry>

2502

</row>

2503

2504

<row>

2505

<entry>hex_startaddr</entry>

2506

2507

<entry>The Starting address as a string.</entry>

2508

</row>

2509

2510

<row>

2511

<entry>start_addr</entry>

2512

2513

<entry>The starting address as a value.</entry>

2514

</row>

2515

2516

<row>

2517

<entry>startaddr</entry>

2518

2519

2520

</row>

2521

2522

<row>

2523

<entry>exit_statuses_bad</entry>

2524

2525

<entry>Whether there is an accurate exit status.</entry>

2526

</row>

2527

2528

<row>

2529

<entry>reboot_delay</entry>

2530

2531

<entry>The delay between power off and power on.</entry>

2532

</row>

2533

2534

<row>

2535

<entry>unreliable</entry>

2536

2537

<entry>Whether communication with the board is unreliable.</entry>

2538

</row>

2539

2540

<row>

2541

2542

2543

<entry>The path to the simulator to use.</entry>

2544

</row>

2545

2546

<row>

2547

<entry>objcopy</entry>

2548

<entry>$tempfil</entry>

2549

<entry>The path to the <command>objcopy</command> program.</entry>

2550

</row>

2551

2552

<row>

2553

<entry>support_libs</entry>

2554

<entry>"${prefix_dir}/i386-coff/"</entry>

2555

<entry>Support libraries needed for cross compiling.</entry>

2556

</row>

2557

2558

<row>

2559

<entry>addl_link_flags</entry>

2560

2561

<entry>Additional link flags, rarely used.</entry>

2562

</row>

2563

2564

</tbody>

2565

</tgroup>

2566

</table>

2567

</para>

2568

2569

<para>These fields are used by the GCC and GDB tests, and are mostly

2570

only useful to somewhat trying to debug a new board file for one of

2571

these tools. Many of these are used only by a few testcases, and their

2572

purpose is esoteric. These are listed with sample values as a guide to

2573

better guessing if you need to change any of these.</para>

2574

2575

2576

<title>Board Info Fields For GCC & GDB</title>

2577

2578

2579

2580

<entry>Field</entry>

2581

<entry>Sample Value</entry>

2582

<entry>Description</entry>

2583

</row></thead>

2584

<tbody>

2585

2586

<row>

2587

<entry>strip</entry>

2588

<entry>$tempfile</entry>

2589

<entry>Strip the executable of symbols.</entry>

2590

</row>

2591

2592

<row>

2593

<entry>gdb_load_offset</entry>

2594

2595

</row>

2596

2597

<row>

2598

<entry>gdb_protocol</entry>

2599

<entry>"remote"</entry>

2600

<entry>The GDB debugging protocol to use.</entry>

2601

</row>

2602

2603

<row>

2604

<entry>gdb_sect_offset</entry>

2605

2606

</row>

2607

2608

<row>

2609

<entry>gdb_stub_ldscript</entry>

2610

2611

<entry>The linker script to use with a GDB stub.</entry>

2612

</row>

2613

2614

<row>

2615

<entry>gdb_init_command</entry>

2616

<entry>"set mipsfpu none"</entry>

2617

</row>

2618

2619

<row>

2620

<entry>gdb,cannot_call_functions</entry>

2621

2622

<entry>Whether GDB can call functions on the target,</entry>

2623

</row>

2624

2625

<row>

2626

<entry>gdb,noargs</entry>

2627

2628

<entry>Whether the target can take command line arguments.</entry>

2629

</row>

2630

2631

<row>

2632

<entry>gdb,nosignals</entry>

2633

2634

<entry>Whether there are signals on the target.</entry>

2635

</row>

2636

2637

<row>

2638

<entry>gdb,short_int</entry>

2639

2640

</row>

2641

2642

<row>

2643

<entry>gdb,start_symbol</entry>

2644

<entry>"_start";</entry>

2645

<entry>The starting symbol in the executable.</entry>

2646

</row>

2647

2648

<row>

2649

<entry>gdb,target_sim_options</entry>

2650

<entry>"-sparclite"</entry>

2651

<entry>Special options to pass to the simulator.</entry>

2652

</row>

2653

2654

<row>

2655

<entry>gdb,timeout</entry>

2656

2657

<entry>Timeout value to use for remote communication.</entry>

2658

</row>

2659

2660

<row>

2661

<entry>gdb_init_command</entry>

2662

<entry>"print/x \$fsr = 0x0"</entry>

2663

</row>

2664

2665

<row>

2666

<entry>gdb_load_offset</entry>

2667

2668

</row>

2669

2670

<row>

2671

2672

<entry>"--command gdbinit"</entry>

2673

</row>

2674

2675

<row>

2676

<entry>gdb_prompt</entry>

2677

2678

<entry>The prompt GDB is using.</entry>

2679

</row>

2680

2681

<row>

2682

<entry>gdb_run_command</entry>

2683

<entry>"jump start"</entry>

2684

</row>

2685

2686

<row>

2687

<entry>gdb_stub_offset</entry>

2688

2689

</row>

2690

2691

<row>

2692

2693

2694

<entry>Whether to use a GDB stub.</entry>

2695

</row>

2696

2697

<row>

2698

<entry>use_vma_offset</entry>

2699

2700

</row>

2701

2702

<row>

2703

2704

2705

</row>

2706

2707

<row>

2708

<entry>gcc,no_label_values</entry>

2709

2710

</row>

2711

2712

<row>

2713

<entry>gcc,no_trampolines</entry>

2714

2715

</row>

2716

2717

<row>

2718

<entry>gcc,no_varargs</entry>

2719

2720

</row>

2721

2722

<row>

2723

<entry>gcc,stack_size</entry>

2724

2725

<entry>Stack size to use with some GCC testcases.</entry>

2726

</row>

2727

2728

<row>

2729

<entry>ieee_multilib_flags</entry>

2730

<entry>"-mieee";</entry>

2731

</row>

2732

2733

<row>

2734

<entry>is_simulator</entry>

2735

2736

</row>

2737

2738

<row>

2739

<entry>needs_status_wrapper</entry>

2740

2741

</row>

2742

2743

<row>

2744

<entry>no_double</entry>

2745

2746

</row>

2747

2748

<row>

2749

2750

2751

</row>

2752

2753

<row>

2754

<entry>noargs</entry>

2755

2756

</row>

2757

2758

<row>

2759

<entry>nullstone,lib</entry>

2760

<entry>"mips-clock.c"</entry>

2761

</row>

2762

2763

<row>

2764

<entry>nullstone,ticks_per_sec</entry>

2765

2766

</row>

2767

2768

<row>

2769

<entry>sys_speed_value</entry>

2770

2771

</row>

2772

2773

<row>

2774

<entry>target_install</entry>

2775

2776

</row>

2777

2778

</tbody>

2779

</tgroup>

2780

</table>

2781

</para>

2782

2783

</sect1>

2784

2785

2786

<title>Writing A Test Case</title>

2787

2788

<para>The easiest way to prepare a new test case is to base it

2789

on an existing one for a similar situation. There are two major

2790

categories of tests: batch or interactive. Batch oriented tests

2791

are usually easier to write.</para>

2792

2793

<para>The GCC tests are a good example of batch oriented tests.

2794

All GCC tests consist primarily of a call to a single common

2795

procedure, Since all the tests either have no output, or only

2796

have a few warning messages when successfully compiled. Any

2797

non-warning output is a test failure. All the C code needed is

2798

kept in the test directory. The test driver, written in Tcl,

2799

need only get a listing of all the C files in the directory, and

2800

compile them all using a generic procedure. This procedure and a

2801

few others supporting for these tests are kept in the library

2802

module <filename>lib/c-torture.exp</filename> in the GCC test

2803

suite. Most tests of this kind use very few

2804

<productname>expect</productname> features, and are coded almost

2805

purely in Tcl.</para>

2806

2807

<para>Writing the complete suite of C tests, then, consisted of

2808

these steps:</para>

2809

2810

2811

<listitem><para>Copying all the C code into the test directory.

2812

These tests were based on the C-torture test created by Torbjorn

2813

Granlund (on behalf of the Free Software Foundation) for GCC

2814

development.</para></listitem>

2815

2816

<listitem><para>Writing (and debugging) the generic Tcl procedures for

2817

compilation.</para></listitem>

2818

2819

<listitem><para>Writing the simple test driver: its main task is to

2820

search the directory (using the Tcl procedure

2821

<emphasis>glob</emphasis> for filename expansion with wildcards)

2822

and call a Tcl procedure with each filename. It also checks for

2823

a few errors from the testing procedure.</para></listitem>

2824

</itemizedlist>

2825

2826

<para>Testing interactive programs is intrinsically more

2827

complex. Tests for most interactive programs require some trial

2828

and error before they are complete.</para>

2829

2830

<para>However, some interactive programs can be tested in a

2831

simple fashion reminiscent of batch tests. For example, prior

2832

to the creation of DejaGnu, the GDB distribution already

2833

included a wide-ranging testing procedure. This procedure was

2834

very robust, and had already undergone much more debugging and

2835

error checking than many recent DejaGnu test cases.

2836

Accordingly, the best approach was simply to encapsulate the

2837

existing GDB tests, for reporting purposes. Thereafter, new GDB

2838

tests built up a family of Tcl procedures specialized for GDB

2839

testing.</para>

2840

2841

</sect1>

2842

2843

2844

<title>Debugging A Test Case</title>

2845

2846

<para>These are the kinds of debugging information available

2847

from DejaGnu:</para>

2848

2849

2850

2851

<listitem><para>Output controlled by test scripts themselves,

2852

explicitly allowed for by the test author. This kind of

2853

debugging output appears in the detailed output recorded in the

2854

DejaGnu log file. To do the same for new tests, use the

2855

<command>verbose</command> procedure (which in turn uses the

2856

variable also called <emphasis>verbose</emphasis>) to control

2857

how much output to generate. This will make it easier for other

2858

people running the test to debug it if necessary. Whenever

2859

possible, if <emphasis>$verbose</emphasis> is

2860

<emphasis>0</emphasis>, there should be no output other than the

2861

output from <emphasis>pass</emphasis>,

2862

<emphasis>fail</emphasis>, <emphasis>error</emphasis>, and

2863

<emphasis>warning</emphasis>. Then, to whatever extent is

2864

appropriate for the particular test, allow successively higher

2865

values of <emphasis>$verbose</emphasis> to generate more

2866

information. Be kind to other programmers who use your tests:

2867

provide for a lot of debugging information.</para></listitem>

2868

2869

<listitem><para>Output from the internal debugging functions of

2870

Tcl and <productname>Expect</productname>. There is a command

2871

line options for each; both forms of debugging output are

2872

recorded in the file <filename>dbg.log</filename> in the current

2873

directory.</para>

2874

2875

<para>Use <option>--debug</option> for information from the

2876

expect level; it generates displays of the expect attempts to

2877

match the tool output with the patterns specified. This output

2878

can be very helpful while developing test scripts, since it

2879

shows precisely the characters received. Iterating between the

2880

latest attempt at a new test script and the corresponding

2881

<filename>dbg.log</filename> can allow you to create the final

2882

patterns by ``cut and paste''. This is sometimes the best way

2883

to write a test case.</para></listitem>

2884

2885

<listitem><para>Use <option>--strace</option> to see more

2886

detail at the Tcl level; this shows how Tcl procedure

2887

definitions expand, as they execute. The associated number

2888

controls the depth of definitions expanded.</para></listitem>

2889

2890

<listitem><para>Finally, if the value of

2891

<emphasis>verbose</emphasis> is 3 or greater,DejaGnu turns on

2892

the expect command <command>log_user</command>. This command

2893

prints all expect actions to the expect standard output, to the

2894

detailed log file, and (if <option>--debug</option> is on) to

2895

2896

</itemizedlist>

2897

2898

</sect1>

2899

2900

2901

<title>Adding A Test Case To A Testsuite.</title>

2902

2903

<para>There are two slightly different ways to add a test

2904

case. One is to add the test case to an existing directory. The

2905

other is to create a new directory to hold your test. The

2906

existing test directories represent several styles of testing,

2907

all of which are slightly different; examine the directories for

2908

the tool of interest to see which (if any) is most suitable.</para>

2909

2910

<para>Adding a GCC test can be very simple: just add the C code

2911

to any directory beginning with <filename>gcc</filename>. and it

2912

runs on the next <programlisting>runtest --tool

2913

gcc</programlisting>.</para>

2914

2915

<para>To add a test to GDB, first add any source code you will

2916

need to the test directory. Then you can either create a new

2917

expect file, or add your test to an existing one (any

2918

file with a <emphasis>.exp</emphasis> suffix). Creating a new

2919

.exp file is probably a better idea if the test is significantly

2920

different from existing tests. Adding it as a separate file also

2921

makes upgrading easier. If the C code has to be already compiled

2922

before the test will run, then you'll have to add it to the

2923

<filename>Makefile.in</filename> file for that test directory,

2924

then run <command>configure</command> and

2925

2926

2927

<para>Adding a test by creating a new directory is very

2928

similar:</para>

2929

2930

2931

2932

<listitem><para>Create the new directory. All subdirectory names

2933

begin with the name of the tool to test; e.g. G++ tests might be

2934

in a directory called <filename>g++.other</filename>. There can

2935

be multiple test directories that start with the same tool name

2936

(such as <emphasis>g++</emphasis>).</para></listitem>

2937

2938

<listitem><para>Add the new directory name to the

2939

<symbol>configdirs</symbol> definition in the

2940

<filename>configure.in</filename> file for the testsuite

2941

directory. This way when <command>make</command> and

2942

<command>configure</command> next run, they include the new

2943

directory.</para></listitem>

2944

2945

<listitem><para>Add the new test case to the directory, as

2946

above. </para></listitem>

2947

2948

<listitem><para>To add support in the new directory for

2949

configure and make, you must also create a

2950

<filename>Makefile.in</filename> and a

2951

<filename>configure.in</filename>.</para></listitem>

2952

</itemizedlist>

2953

2954

</sect1>

2955

2956

2957

<title>Hints On Writing A Test Case</title>

2958

2959

<para>It is safest to write patterns that match all the output

2960

generated by the tested program; this is called closure.

2961

If a pattern does not match the entire output, any output that

2962

remains will be examined by the next <command>expect</command>

2963

command. In this situation, the precise boundary that determines

2964

which <command>expect</command> command sees what is very

2965

sensitive to timing between the Expect task and the task running

2966

the tested tool. As a result, the test may sometimes appear to

2967

work, but is likely to have unpredictable results. (This problem

2968

is particularly likely for interactive tools, but can also

2969

affect batch tools---especially for tests that take a long time

2970

to finish.) The best way to ensure closure is to use the

2971

<option>-re</option> option for the <command>expect</command>

2972

command to write the pattern as a full regular expressions; then

2973

you can match the end of output using a <emphasis>$</emphasis>.

2974

It is also a good idea to write patterns that match all

2975

available output by using <emphasis>.*\</emphasis> after the

2976

text of interest; this will also match any intervening blank

2977

lines. Sometimes an alternative is to match end of line using

2978

<emphasis>\r</emphasis> or <emphasis>\n</emphasis>, but this is

2979

usually too dependent on terminal settings.</para>

2980

2981

<para>Always escape punctuation, such as <emphasis>(</emphasis>

2982

or <emphasis>&quot</emphasis>, in your patterns; for example, write

2983

<emphasis>\(</emphasis>. If you forget to escape punctuation,

2984

you will usually see an error message like <programlisting>extra

2985

characters after close-quote.</programlisting></para>

2986

2987

<para>If you have trouble understanding why a pattern does not

2988

match the program output, try using the <option>--debug</option>

2989

option to <command>runtest</command>, and examine the debug log

2990

carefully.</para>

2991

2992

<para>Be careful not to neglect output generated by setup rather

2993

than by the interesting parts of a test case. For example,

2994

while testing GDB, I issue a send <emphasis>set height

2995

0\n</emphasis> command. The purpose is simply to make sure GDB

2996

never calls a paging program. The <emphasis>set

2997

height</emphasis> command in GDB does not generate any

2998

output; but running any command makes GDB issue a new

2999

<emphasis>(gdb) </emphasis> prompt. If there were no

3000

<command>expect</command> command to match this prompt, the

3001

output <emphasis>(gdb) </emphasis> begins the text seen by the

3002

next <command>expect</command> command---which might make that

3003

pattern fail to match.</para>

3004

3005

<para>To preserve basic sanity, I also recommended that no test

3006

ever pass if there was any kind of problem in the test case. To

3007

take an extreme case, tests that pass even when the tool will

3008

not spawn are misleading. Ideally, a test in this sort of

3009

situation should not fail either. Instead, print an error

3010

message by calling one of the DejaGnu procedures

3011

<command>error</command> or <command>warning</command>.</para>

3012

3013

</sect1>

3014

3015

3016

<title>Special variables used by test cases.</title>

3017

3018

<para>There are special variables used by test cases. These contain

3019

other information from DejaGnu. Your test cases can use these variables,

3020

with conventional meanings (as well as the variables saved in

3021

<filename>site.exp</filename>. You can use the value of these variables,

3022

but they should never be changed.</para>

3023

3024

3025

3026

3027

<listitem><para>The tracking system (e.g. GNATS) number identifying

3028

a corresponding bugreport. (<emphasis>0</emphasis>} if you do not

3029

specify it in the test script.)</para></listitem>

3030

</varlistentry>

3031

3032

3033

3034

<listitem><para>An optional bug id; may reflect a bug

3035

identification from another organization. (<emphasis>0</emphasis>

3036

if you do not specify it.)</para></listitem>

3037

</varlistentry>

3038

3039

3040

<term>$subdir</term>

3041

<listitem><para>The subdirectory for the current test

3042

case.</para></listitem>

3043

</varlistentry>

3044

3045

3046

<term>$expect_out(buffer)</term>

3047

<listitem><para>The output from the last command. This is an

3048

internal variable set by Expect. More information can be found in

3049

the Expect manual.</para></listitem>

3050

</varlistentry>

3051

3052

3053

<term>$exec_output</term>

3054

<listitem><para>This is the output from a

3055

<function>${tool}_load</function> command. This only applies to

3056

tools like GCC and GAS which produce an object file that must in

3057

turn be executed to complete a test.</para></listitem>

3058

</varlistentry>

3059

3060

3061

<term>$comp_output</term>

3062

<listitem><para>This is the output from a

3063

<function>${tool}_start</function> command. This is conventionally

3064

used for batch oriented programs, like GCC and GAS, that may

3065

produce interesting output (warnings, errors) without further

3066

interaction.</para></listitem>

3067

</varlistentry>

3068

</variablelist>

3069

3070

</sect1>

3071

3072

</chapter>

3073

3074

3075

<title>Unit Testing</title>

3076

3077

3078

<title>What Is Unit Testing ?</title>

3079

3080

<para>Most regression testing as done by DejaGnu is system

3081

testing. This is the complete application is tested all at

3082

once. Unit testing is for testing single files, or small

3083

libraries. In this case, each file is linked with a test case in

3084

C or C++, and each function or class and method is tested in

3085

series, with the test case having to check private data or

3086

global variables to see if the function or method worked.</para>

3087

3088

<para>This works particularly well for testing APIs and at level

3089

where it is easier to debug them, than by needing to trace through

3090

the entire appication. Also if there is a specification for the

3091

API to be tested, the testcase can also function as a compliance

3092

test.</para>

3093

3094

</sect1>

3095

3096

3097

<title>The dejagnu.h Header File</title>

3098

3099

<para>DejaGnu uses a single header file to assist in unit

3100

testing. As this file also produces it's one test state output,

3101

it can be run standalone, which is very useful for testing on

3102

embedded systems. This header file has a C and C++ API for the

3103

test states, with simple totals, and standardized

3104

output. Because the output has been standardized, DejaGnu can be

3105

made to work with this test case, without writing almost any

3106

Tcl. The library module, dejagnu.exp, will look for the output

3107

messages, and then merge them into DejaGnu's.</para>

3108

3109

</sect1>

3110

3111

</chapter>

3112

3113

<!-- Keep this comment at the end of the file

3114

Local variables:

3115

mode: sgml

3116

sgml-omittag:t

3117

sgml-shorttag:t

3118

sgml-namecase-general:t

3119

sgml-general-insert-case:lower

3120

sgml-minimize-attributes:nil

3121

sgml-always-quote-attributes:t

3122

sgml-indent-step:1

3123

sgml-indent-data:nil

3124

sgml-parent-document:nil

3125

sgml-exposed-tags:nil

3126

sgml-local-catalogs:nil

3127

sgml-local-ecat-files:nil

3128

End:

3129

-->

Older »