2701.3.1
by Tim Penhey
initial commit |
1 |
How to write tests |
2 |
================== |
|
3 |
||
4 |
On the whole, new or updated code will not pass review unless there are tests |
|
5 |
associated with the code. For code additions, the tests should cover as much |
|
6 |
of the new code as practical, and for code changes, either the tests should be |
|
7 |
updated, or at least the tests that already exist that cover the refactored |
|
8 |
code should be identified when requesting a review to show that there is already |
|
9 |
test coverage, and that the refactoring didn't break anything. |
|
10 |
||
2701.3.2
by Tim Penhey
More text. |
11 |
|
2701.3.1
by Tim Penhey
initial commit |
12 |
go test and gocheck |
13 |
------------------- |
|
14 |
||
15 |
The `go test` command is used to run the tests. Juju uses the `gocheck` package |
|
16 |
("launchpad.net/gocheck") to provide a checkers and assert methods for the test |
|
17 |
writers. The use of gocheck replaces the standard `testing` library. |
|
18 |
||
2701.3.2
by Tim Penhey
More text. |
19 |
Across all of the tests in juju-core, the gocheck package is imported |
20 |
with a shorter alias, because it is used a lot. |
|
21 |
||
22 |
```go |
|
23 |
import ( |
|
24 |
// system packages |
|
25 |
||
26 |
gc "launchpad.net/gocheck" |
|
27 |
||
28 |
// juju packages |
|
29 |
) |
|
30 |
``` |
|
31 |
||
2701.3.1
by Tim Penhey
initial commit |
32 |
|
33 |
setting up tests for new packages |
|
34 |
--------------------------------- |
|
35 |
||
36 |
Lets say we are creating a new provider for "magic" cloud, and we have a package |
|
37 |
called "magic" that lives at "launchpad.net/juju-core/provider/magic". The |
|
38 |
general approach for testing in juju is to have the tests in a separate package. |
|
39 |
Continuing with this example the tests would be in a package called "magic_test". |
|
40 |
||
41 |
A common idiom that has occurred in juju is to setup to gocheck hooks in a special |
|
42 |
file called `package_test.go` that would look like this: |
|
43 |
||
44 |
||
45 |
```go |
|
46 |
// Copyright 2014 Canonical Ltd. |
|
47 |
// Licensed under the AGPLv3, see LICENCE file for details. |
|
48 |
||
49 |
package magic_test |
|
50 |
||
51 |
import ( |
|
52 |
"testing" |
|
53 |
||
54 |
gc "launchpad.net/gocheck" |
|
55 |
) |
|
56 |
||
57 |
func Test(t *testing.T) { |
|
58 |
gc.TestingT(t) |
|
59 |
} |
|
60 |
``` |
|
61 |
||
62 |
or |
|
63 |
||
64 |
```go |
|
65 |
// Copyright 2014 Canonical Ltd. |
|
66 |
// Licensed under the AGPLv3, see LICENCE file for details. |
|
67 |
||
68 |
package magic_test |
|
69 |
||
70 |
import ( |
|
71 |
stdtesting "testing" |
|
72 |
||
73 |
"launchpad.net/juju-core/testing" |
|
74 |
) |
|
75 |
||
76 |
func Test(t *stdtesting.T) { |
|
77 |
testing.MgoTestPackage(t) |
|
78 |
} |
|
79 |
``` |
|
80 |
||
81 |
The key difference here is that the first one just hooks up `gocheck` |
|
82 |
so it looks for the `gocheck` suites in the package. The second makes |
|
83 |
sure that there is a mongo available for the duration of the package tests. |
|
84 |
||
2701.3.2
by Tim Penhey
More text. |
85 |
A general rule is not to setup mongo for a package unless you really |
86 |
need to as it is extra overhead. |
|
87 |
||
88 |
||
89 |
writing the test files |
|
90 |
---------------------- |
|
91 |
||
92 |
Normally there will be a test file for each file with code in the package. |
|
93 |
For a file called `config.go` there should be a test file called `config_test.go`. |
|
94 |
||
95 |
The package should in most cases be the same as the normal files with a "_test" suffix. |
|
96 |
In this way, the tests are testing the same interface as any normal user of the |
|
97 |
package. It is reasonably common to want to modify some internal aspect of the package |
|
98 |
under test for the tests. This is normally handled by a file called `export_test.go`. |
|
99 |
Even though the file ends with `_test.go`, the package definition is the same as the |
|
100 |
normal source files. In this way, for the tests and only the tests, additional |
|
101 |
public symbols can be defined for the package and used in the tests. |
|
102 |
||
103 |
Here is an annotated extract from `provider/local/export_test.go` |
|
104 |
||
105 |
```go |
|
106 |
// The package is the "local" so it has access to the package symbols |
|
107 |
// and not just the public ones. |
|
108 |
package local |
|
109 |
||
110 |
import ( |
|
111 |
"github.com/juju/testing" |
|
112 |
gc "launchpad.net/gocheck" |
|
113 |
||
114 |
"launchpad.net/juju-core/environs/config" |
|
115 |
) |
|
116 |
||
117 |
var ( |
|
2701.3.4
by Tim Penhey
Fix formatting, and add comment about the juju conn suite. |
118 |
// checkIfRoot is a variable of type `func() bool`, so CheckIfRoot is |
119 |
// a pointer to that variable so we can patch it in the tests. |
|
2701.3.2
by Tim Penhey
More text. |
120 |
CheckIfRoot = &checkIfRoot |
121 |
// providerInstance is a pointer to an instance of a private structure. |
|
122 |
// Provider points to the same instance, so public methods on that instance |
|
123 |
// are available in the tests. |
|
124 |
Provider = providerInstance |
|
125 |
) |
|
126 |
||
127 |
// ConfigNamespace is a helper function for the test that steps through a |
|
128 |
// number of private methods or variables, and is an alternative mechanism |
|
129 |
// to provide functionality for the tests. |
|
130 |
func ConfigNamespace(cfg *config.Config) string { |
|
131 |
env, _ := providerInstance.Open(cfg) |
|
132 |
return env.(*localEnviron).config.namespace() |
|
133 |
} |
|
134 |
``` |
|
135 |
||
136 |
Suites and Juju base suites |
|
137 |
--------------------------- |
|
138 |
||
139 |
With gocheck tests are grouped into Suites. Each suite has distinct |
|
140 |
set-up and tear-down logic. Suites are often composed of other suites |
|
141 |
that provide specific set-up and tear-down behaviour. |
|
142 |
||
2715.17.3
by Ian Booth
Factor out BaseSuite and separate FakeHome suites |
143 |
There are four main suites: |
2701.3.2
by Tim Penhey
More text. |
144 |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
145 |
* /testing.BaseSuite (testing/base.go) |
2715.17.3
by Ian Booth
Factor out BaseSuite and separate FakeHome suites |
146 |
* /testing.FakeHomeSuite (testing/environ.go) |
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
147 |
* /testing.FakeJujuHomeSuite (testing/environ.go) |
2701.3.2
by Tim Penhey
More text. |
148 |
* /juju/testing.JujuConnSuite (juju/testing/conn.go) |
149 |
||
2715.17.3
by Ian Booth
Factor out BaseSuite and separate FakeHome suites |
150 |
The last three have the BaseSuite functionality included through |
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
151 |
composition. The BaseSuite isolates a user's home directory from accidental |
2715.17.3
by Ian Booth
Factor out BaseSuite and separate FakeHome suites |
152 |
modification (by setting $HOME to "") and errors if there is an attempt to do |
153 |
outgoing http access. It also clears the relevant $JUJU_* environment variables. |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
154 |
The BaseSuite is also composed of the core LoggingSuite, and also LoggingSuite |
155 |
from github.com/juju/testing, which brings in the CleanupSuite from the same. |
|
156 |
The CleanupSuite has the functionality around patching environment variables |
|
157 |
and normal variables for the duration of a test. It also provides a clean-up |
|
158 |
stack that gets called when the test teardown happens. |
|
2701.3.2
by Tim Penhey
More text. |
159 |
|
2715.17.3
by Ian Booth
Factor out BaseSuite and separate FakeHome suites |
160 |
All test suites should embedd BaseSuite. Those that need the extra functionality |
161 |
can instead embedd one of the fake home suites: |
|
162 |
||
163 |
* FakeHomeSuite: creates a fake home directory with ~/.ssh and fake ssh keys. |
|
164 |
* FakeJujuHomeSuite: as above but also sets up a ~/.juju with a fake environment. |
|
2701.3.2
by Tim Penhey
More text. |
165 |
|
2701.3.4
by Tim Penhey
Fix formatting, and add comment about the juju conn suite. |
166 |
The JujuConnSuite does this and more. It also sets up a state server and api |
167 |
server. This is one problem with the JujuConnSuite, it almost always does a |
|
168 |
lot more than you actually want or need. This should really be broken into |
|
169 |
smaller component parts that make more sense. If you can get away with not |
|
170 |
using the JujuConnSuite, you should try. |
|
2701.3.2
by Tim Penhey
More text. |
171 |
|
2701.3.3
by Tim Penhey
Section on checkers, and other review tweaks. |
172 |
To create a new suite composed of one or more of the suites above, you can do |
173 |
something like: |
|
2701.3.2
by Tim Penhey
More text. |
174 |
|
175 |
```go |
|
176 |
type ToolsSuite struct { |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
177 |
testing.BaseSuite |
2701.3.2
by Tim Penhey
More text. |
178 |
dataDir string |
179 |
} |
|
180 |
||
181 |
var _ = gc.Suite(&ToolsSuite{}) |
|
182 |
||
183 |
``` |
|
184 |
||
185 |
If there is no extra setup needed, then you don't need to specify any |
|
186 |
set-up or tear-down methods as the LoggingSuite has them, and they are |
|
187 |
called by default. |
|
188 |
||
189 |
If you did want to do something, say, create a directory and save it in |
|
190 |
the dataDir, you would do something like this: |
|
191 |
||
192 |
```go |
|
193 |
func (t *ToolsSuite) SetUpTest(c *gc.C) { |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
194 |
t.BaseSuite.SetUpTest(c) |
2701.3.2
by Tim Penhey
More text. |
195 |
t.dataDir = c.MkDir() |
196 |
} |
|
197 |
``` |
|
198 |
||
199 |
If the test suite has multiple contained suites, please call them in the |
|
200 |
order that they are defined, and make sure something that is composed from |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
201 |
the BaseSuite is first. They should be torn down in the reverse order. |
2701.3.2
by Tim Penhey
More text. |
202 |
|
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
203 |
Even if the code that is being tested currently has no logging or outbound |
204 |
network access in it, it is a good idea to use the BaseSuite as a base: |
|
205 |
* it isolates the user's home directory against accidental modification |
|
206 |
* if someone does add outbound network access later, it will be caught |
|
2701.3.2
by Tim Penhey
More text. |
207 |
* it brings in something composed of the CleanupSuite |
2715.17.2
by Ian Booth
Introduce new BaseSuite for tests |
208 |
* if someone does add logging later, it is captured and doesn't pollute |
2701.3.2
by Tim Penhey
More text. |
209 |
the logging output |
210 |
||
211 |
||
212 |
Patching variables and the environment |
|
213 |
-------------------------------------- |
|
214 |
||
215 |
Inside a test, and assuming that the Suite has a CleanupSuite somewhere |
|
2701.3.3
by Tim Penhey
Section on checkers, and other review tweaks. |
216 |
in the composition tree, there are a few very helpful functions. |
2701.3.2
by Tim Penhey
More text. |
217 |
|
218 |
```go |
|
219 |
||
220 |
var foo int |
|
221 |
||
222 |
func (s *someTest) TestFubar(c *gc.C) { |
|
223 |
// The TEST_OMG environment value will have "new value" for the duration |
|
224 |
// of the test. |
|
225 |
s.PatchEnvironment("TEST_OMG", "new value") |
|
226 |
||
227 |
// foo is set to the value 42 for the duration of the test |
|
228 |
s.PatchValue(&foo, 42) |
|
229 |
} |
|
230 |
``` |
|
231 |
||
232 |
PatchValue works with any matching type. This includes function variables. |
|
233 |
||
234 |
||
2701.3.3
by Tim Penhey
Section on checkers, and other review tweaks. |
235 |
Checkers |
236 |
-------- |
|
237 |
||
238 |
Checkers are a core concept of `gocheck` and will feel familiar to anyone |
|
239 |
who has used the python testtools. Assertions are made on the gocheck.C |
|
240 |
methods. |
|
241 |
||
242 |
```go |
|
243 |
c.Check(err, gc.IsNil) |
|
244 |
c.Assert(something, gc.Equals, somethingElse) |
|
245 |
``` |
|
246 |
||
247 |
The `Check` method will cause the test to fail if the checker returns |
|
248 |
false, but it will continue immediately cause the test to fail and will |
|
249 |
continue with the test. `Assert` if it fails will cause the test to |
|
250 |
immediately stop. |
|
251 |
||
252 |
For the purpose of further discussion, we have the following parts: |
|
253 |
||
254 |
`c.Assert(observed, checker, args...)` |
|
255 |
||
256 |
The key checkers in the `gocheck` module that juju uses most frequently are: |
|
257 |
||
258 |
* `IsNil` - the observed value must be `nil` |
|
259 |
* `NotNil` - the observed value must not be `nil` |
|
260 |
* `Equals` - the observed value must be the same type and value as the arg, |
|
261 |
which is the expected value |
|
262 |
* `DeepEquals` - checks for equality for more complex types like slices, |
|
263 |
maps, or structures. This is DEPRECATED in favour of the DeepEquals from |
|
264 |
the `github.com/juju/testing/checkers` covered below |
|
265 |
* `ErrorMatches` - the observed value is expected to be an `error`, and |
|
266 |
the arg is a string that is a regular expression, and used to match the |
|
267 |
error string |
|
268 |
* `Matches` - a regular expression match where the observed value is a string |
|
269 |
* `HasLen` - the expected value is an integer, and works happily on nil |
|
270 |
slices or maps |
|
271 |
||
272 |
||
273 |
Over time in the juju project there were repeated patterns of testing that |
|
274 |
were then encoded into new and more complicated checkers. These are found |
|
275 |
in `github.com/juju/testing/checkers`, and are normally imported with the |
|
276 |
alias `jc`. |
|
277 |
||
278 |
The matchers there include (not an exclusive list): |
|
279 |
||
280 |
* `IsTrue` - just an easier way to say `gc.Equals, true` |
|
281 |
* `IsFalse` - observed value must be false |
|
282 |
* `GreaterThan` - for integer or float types |
|
283 |
* `LessThan` - for integer or float types |
|
284 |
* `HasPrefix` - obtained is expected to be a string or a `Stringer`, and |
|
285 |
the string (or string value) must have the arg as start of the string |
|
286 |
* `HasSuffix` - the same as `HasPrefix` but checks the end of the string |
|
287 |
* `Contains` - obtained is a string or `Stringer` and expected needs to be |
|
288 |
a string. The checker passes if the expected string is a substring of the |
|
289 |
obtained value. |
|
290 |
* `DeepEquals` - works the same way as the `gocheck.DeepEquals` except |
|
291 |
gives better errors when the values do not match |
|
292 |
* `SameContents` - obtained and expected are slices of the same type, |
|
293 |
the checker makes sure that the values in one are in the other. They do |
|
294 |
not have the be in the same order. |
|
295 |
* `Satisfies` - the arg is expected to be `func(observed) bool` |
|
296 |
often used for error type checks |
|
297 |
* `IsNonEmptyFile` - obtained is a string or `Stringer` and refers to a |
|
298 |
path. The checker passes if the file exists, is a file, and is not empty |
|
299 |
* `IsDirectory` - works in a similar way to `IsNonEmptyFile` but passes if |
|
300 |
the path element is a directory |
|
301 |
* `DoesNotExist` - also works with a string or `Stringer`, and passes if |
|
302 |
the path element does not exist |
|
303 |
||
304 |
||
2701.3.2
by Tim Penhey
More text. |
305 |
|
306 |
Good tests |
|
307 |
---------- |
|
308 |
||
309 |
Good tests should be: |
|
310 |
* small and obviously correct |
|
311 |
* isolated from any system or environment values that may impact the test |