~ubuntu-branches/debian/stretch/cmdtest/stretch : revision 11

1

% Yarn manual

2

3

Yarn

4

====

5

6

FIXME: This will become a manual for writing test suites in yarn. It

7

is currently not yet written.

8

9

Mission

10

-------

11

This manual will provide all the information needed by Yarn users to enable

12

them to use Yarn effectively in their development projects.

13

14

The information will be

15

16

* easy to find

17

* easy to navigate

18

* easy to use

19

20

The information will include details of

21

22

* how to perform certain tasks

23

* why things are done in particular ways

24

25

Document Status

26

---------------

27

28

### What's Done

29

30

* Outline

31

* Mission

32

* this Document Status section

33

34

### What's New

35

36

* Writing Scenarios

37

* Test Language Specification

38

* Introduction

39

* Skeleton

40

* What is `yarn`?

41

42

43

### What's Next

44

45

* Introduction

46

* Remaining sections

47

* `yarn`'s command line

48

* How to embed `yarn` in Markdown

49

50

Introduction

51

------------

52

53

### What is `yarn`?

54

55

`yarn` is a scenario testing tool: you write a scenario describing how a

56

user uses your software and what should happen, and express, using

57

very lightweight syntax, the scenario in such a way that it can be tested

58

automatically. The scenario has a simple, but strict structure:

59

60

SCENARIO name of scenario

61

GIVEN some setup for the test

62

WHEN thing that is to be tested happens

63

THEN the post-conditions must be true

64

65

As an example, consider a very short test scenario for verifying that

66

a backup program works, at least for one simple case.

67

68

SCENARIO basic backup and restore

69

GIVEN some live data in a directory

70

AND an empty backup repository

71

WHEN a backup is made

72

THEN the data can be restored

73

74

(Note the addition of AND: you can have multiple GIVEN, WHEN, and

75

THEN statements. The AND keyword makes the text be more readable.)

76

77

78

### Who is `yarn` for?

79

80

### Who are the test suites written in `yarn` for?

81

82

### What kinds of testing is `yarn` for?

83

84

### Why `yarn` instead of other tools?

85

86

### Why not cmdtest?

87

88

Writing Scenarios

89

-----------------

90

91

Scenarios are meant to be written in mostly human readable language.

92

However, they are not free form text. In addition to the GIVEN/WHEN/THEN

93

structure, the text for each of the steps needs a computer-executable

94

implementation. This is done by using IMPLEMENTS. The backup scenario

95

from above might be implemented as follows:

96

97

IMPLEMENTS GIVEN some live data in a directory

98

rm -rf "$DATADIR/data"

99

mkdir "$DATADIR/data"

100

echo foo > "$DATADIR/data/foo"

101

102

IMPLEMENTS GIVEN an empty backup repository

103

rm -rf "$DATADIR/repo"

104

mkdir "$DATADIR/repo"

105

106

IMPLEMENTS WHEN a backup is made

107

backup-program -r "$DATADIR/repo" "$DATADIR/data"

108

109

IMPLEMENTS THEN the data can be restored

110

mkdir "$DATADIR/restored"

111

restore-program -r "$DATADIR/repo" "$DATADIR/restored"

112

diff -rq "$DATADIR/data" "$DATADIR/restored"

113

114

Each "IMPLEMENT GIVEN" (or WHEN, THEN) is followed by a regular

115

expression on the same line, and then a shell script that gets executed

116

to implement any step that matches the regular expression. The

117

implementation can extract data from the match as well: for example,

118

the regular expression might allow a file size to be specified.

119

120

The above example seems a bit silly, of course: why go to the effort

121

to obfuscate the various steps? The answer is that the various steps,

122

implemented using IMPLEMENTS, can be combined in many ways, to test

123

different aspects of the program being tested. In effect, the IMPLEMENTS

124

sections provide a vocabulary which the scenario writer can use to

125

express a variety of usefully different scenarios, which together

126

test all the aspects of the software that need to be tested.

127

128

Moreover, by making the step descriptions be human language

129

text, matched by regular expressions, most of the test can

130

hopefully be written, and understood, by non-programmers. Someone

131

who understands what a program should do, could write tests

132

to verify its behaviour. The implementations of the various

133

steps need to be implemented by a programmer, but given a

134

well-designed set of steps, with enough flexibility in their

135

implementation, that quite a good test suite can be written.

136

137

### Test Language Specification

138

139

A test document is written in [Markdown][markdown], with block

140

quoted code blocks being interpreted specially. Each block

141

must follow the syntax defined here.

142

143

* Every step in a scenario is one line, and starts with a keyword.

144

145

* Each implementation (IMPLEMENTS) starts as a new block, and

146

continues until there is a block that starts with another

147

keyword.

148

149

The following keywords are defined.

150

151

* **SCENARIO** starts a new scenario. The rest of the line is the name of

152

the scenario. The name is used for documentation and reporting

153

purposes only and has no semantic meaning. SCENARIO MUST be the

154

first keyword in a scenario, with the exception of IMPLEMENTS.

155

The set of documents passed in a test run may define any number of

156

scenarios between them, but there must be at least one or it is a

157

test failure. The IMPLEMENTS sections are shared between the

158

documents and scenarios.

159

160

* **ASSUMING** defines a condition for the scenario. The rest of the

161

line is "matched text", which gets implemented by an

162

IMPLEMENTS section. If the code executed by the implementation

163

fails, the scenario is skipped.

164

165

* **GIVEN** prepares the world for the test to run. If

166

the implementation fails, the scenario fails.

167

168

* **WHEN** makes the change to the world that is to be tested.

169

If the code fails, the scenario fails.

170

171

* **THEN** verifies that the changes made by the GIVEN steps

172

did the right thing. If the code fails, the scenario fails.

173

174

* **FINALLY** specifies how to clean up after a scenario. If the code

175

fails, the scenario fails. All FINALLY blocks get run either when

176

encountered in the scenario flow, or at the end of the scenario,

177

regardless of whether the scenario is failing or not.

178

179

* **AND** acts as ASSUMING, GIVEN, WHEN, THEN, or FINALLY: whichever

180

was used last. It must not be used unless the previous step was

181

one of those, or another AND.

182

183

* **IMPLEMENTS** is followed by one of ASSUMING, GIVEN, WHEN, or THEN,

184

and a PCRE regular expression, all on one line, and then further

185

lines of shell commands until the end of the block quoted code

186

block. Markdown is unclear whether an empty line (no characters,

187

not even whitespace) between two block quoted code blocks starts a

188

new one or not, so we resolve the ambiguity by specifiying that a

189

code block directly following a code block is a continuation unless

190

it starts with one of the scenario testing keywords.

191

192

The shell commands get parenthesised parts of the match of the

193

regular expression as environment variables (`$MATCH_1` etc). For

194

example, if the regexp is "a (\d+) byte file", then `$MATCH_1` gets

195

set to the number matched by `\d+`.

196

197

The test runner creates a temporary directory, whose name is

198

given to the shell code in the `DATADIR` environment variable.

199

200

The test runner sets the `SRCDIR` environment variable to the

201

path to the directory it was invoked from (by convention, the

202

root of the source tree of the project).

203

204

The test runner removes all other environment variables, except

205

`TERM`, `USER`, `USERNAME`, `LOGNAME`, `HOME`, and `PATH`. It also

206

forces `SHELL` set to `/bin/sh`, and `LC_ALL` set to `C`, in order

207

to have as clean an environment as possible for tests to run in.

208

209

The shell commands get invoked with `/bin/sh -eu`, and need to

210

be written accordingly. Be careful about commands that return a

211

non-zero exit code. There will eventually be a library of shell

212

functions supplied which allow handling the testing of non-zero

213

exit codes cleanly. In addition functions for handling stdout and

214

stderr will be provided.

215

216

The code block of an IMPLEMENTS block fails if the shell

217

invocation exits with a non-zero exit code. Output to stderr is

218

not an indication of failure. Any output to stdout or stderr may

219

or may not be shown to the user.

220

221

Semantics:

222

223

* The name of each scenario (given with SCENARIO) must be unique.

224

* All names of scenarios and steps will be normalised before use

225

(whitespace collapse, leading and trailing whitespace

226

* Every ASSUMING, GIVEN, WHEN, THEN, FINALLY must be matched by

227

exactly one IMPLEMENTS. The test runner checks this before running

228

any code.

229

* Every IMPLEMENTS may match any number of ASSUMING, GIVEN, WHEN,

230

THEN, or FINALLY. The test runner may warn if an IMPLEMENTS is unused.

231

* If ASSUMING fails, that scenario is skipped, and any FINALLY steps

232

are not run.

233

234

235

236

Outline

237

-------

238

239

* Introduction

240

- what is yarn?

241

- who is yarn for?

242

- who are the test suites written in yarn for?

243

- what kinds of testing is yarn for?

244

- why yarn instead of other tools?

245

- why not cmdtest?

246

- NOT installation instructions

247

* Examples

248

- a test suite for "hello world"

249

- make the files available so people can try things for themselves

250

- a few simple scenarios

251

* The yarn testing language

252

- Markdown with blockquotes for the executable code

253

- SCENARIO + the step-wise keywords

254

- IMPLEMENTS sections

255

* Running yarn

256

- command line syntax

257

- examples of various ways to run yarn in different scenarios:

258

- how to run just one scenario

259

- how to run yarn under cron or jenkins

260

- formatting a test suite in yarn with pandoc

261

* Best practices

262

- this chapter will describe best practices for writing test suites

263

with yarn

264

- how to structure the files: what to put in each *.yarn file, e.g.,

265

where should IMPLEMENTS go

266

- how to write test suites that make it easy to debug things when a

267

test case fails

268

- good phrasing guidelines for yarn scenario names and step names

269

- what things are good to keep visible to the reader, what are

270

better hidden inside impementations of steps, with examples from

271

real projects using yarn

272

- guidelines for well-defined steps that are easy to understand and

273

easy to implement

274

- anti-patterns: things that are good to avoid

275

- make tests fast

276

- make test code be obviously correct; make test code be the best

277

code

278

- when is it OK to skip scenarios?

279

* Case studies

280

- this chapter will discuss ways to use yarn in things that are not

281

just "run this program and examine the output"

282

- start a daemon in the background, kill it at the end of a scenario

283

- how to use a really heavy-weight thing in test suites (e.g., start

284

a database server for all scenarios to share)