~ubuntu-archive/britney/britney2 : revision 787

1

#! TITLE: Britney migration documentation

2

#! SUBTITLE: Understanding britney's workflow

3

4

# Migrations

5

6

This is a technical introduction to how britney

7

handles migrations. Being an introduction, it deliberately

8

oversimplifies certain things at the expense of accuracy.

9

It also covers common migration issues and how to fix

10

them.

11

12

The document is primarily aimed at contributors for

13

distributions that want to understand the basics of

14

britney and its migration rules.

15

16

The documentation also aspires to be a general purpose document

17

for britney that is applicable for multiple distributions.

18

However, it does reference distribution-specific practises in

19

some examples to prevent the documentation from becoming too

20

abstract. Furthermore, the document assumes familiarity with

21

Debian-based distribution practises and terminology (such as

22

"suites" and "source package").

23

24

## A high level overview of britney and migrations

25

26

The purpose of britney is to (semi-)automatically select

27

a number of migration items from a series of source suites

28

(e.g. Debian unstable) that are ready to migrate to

29

the target suite (e.g. Debian testing).

30

31

The definition of "ready" can be summarized as satisfying all

32

of the following points:

33

34

1. The migration items pass a number of policies for the target

35

suite. Most of these policies are basically that the

36

migration items do not regress on selected QA checks.

37

* An item satisfying this part is called a `valid candiate`.

38

1. Installability will not regress as a result of

39

migrating the migration items.

40

* An item that (also) satisfies this part will be selected

41

for migration.

42

43

The keyword in both points being *regress*. If a package has an

44

existing issue in the target suite, the item including a new version

45

of that package is generally allowed to migrate if it has the same

46

issue (as it is not a regression).

47

48

This only leaves the definition of a migration items. They come

49

in several variants defined in the next section.

50

51

## Migration items

52

53

Internally, britney groups packages into migration items based on a

54

few rules. There are several kinds of migration items and this

55

document will only describe the source migration item.

56

57

> A source migration item is one upload of a source package, with

58

> associated binary packages once built.

59

60

Once a new version of a source package appears in the source suite,

61

britney will create track it with a source migration item. As the

62

binary packages are built and uploaded, they will be included into the

63

migration item and various QA checks/policies will be applied to the

64

item.

65

66

Once britney deems the item ready, it will attempt to

67

migrate the item (i.e. source with its binaries) to the

68

target suite.

69

70

71

As implied earlier, there are several other migration types. But they

72

are not covered in this document. They deal with cases like removals,

73

rebuilds of existing binaries, etc.

74

75

## Migration phase 1: Policies / Excuses

76

77

To begin with, britney will apply a number of policies to

78

all migration items. Each policy will rate each migration

79

item and the combined results will be added into one of

80

britney's output documents known as the "excuses" (exists in

81

an HTML and a YAML variant). A migration item that passes all

82

applicable policies will be labelled as a `valid candidate` in

83

the excuses and continue to the next phase.

84

85

86

The policies gives exactly one verdict to each item, some of

87

these verdicts are:

88

89

* The item passes the policy.

90

* The policy is waiting for test suites before providing a

91

pass/fail result (temporary failure).

92

* The item fails the policy and the failure is believed to

93

be "permanent" (given no external changes).

94

* The item does not pass the policy, but britney has

95

insufficient information to determine if the failure is

96

persistent or not.

97

98

It is important to note that all verdicts are based on the current

99

data that britney has access to. This mean that without any change

100

to the items themselves:

101

102

1. Items that passed originally may fail in a later britney run.

103

1. Likewise, items may go from a "permanent failure" to a pass.

104

105

This can be seen in the following example case:

106

107

1. A new version of package is uploaded.

108

* Britney processes the package and concludes that there no blocking bugs,

109

so the package passes the bug policy.

110

1. Then before it migrates, someone files a blocking bug against

111

the new version.

112

* Britney reprocesses the package and now concludes it has a regression in

113

the bug policy (i.e. the policy verdict goes from "pass" to "permanent fail").

114

1. The bug is examined and it is determined that the bug also affects the

115

version in the target suite. The bug tracker is updated to reflect this.

116

* Britney reprocesses the package again and now concludes there is a blocking

117

bug, but it is not a regression (since it also affects the target suite).

118

This means the policy verdict now go from "fail" to "pass".

119

120

This is also applicable to e.g. the piuparts policy, where if the test is

121

rescheduled on the piuparts side and the result changes as a result of that.

122

123

Finally, the people running the britney instance can overrule any

124

policy by applying a [britney hint](hints.html), if they deem it

125

necessary. One caveat here is that not all policies can be overridden

126

directly and some will require the "ignore all policies"-hint (known

127

as the `force`-hint).

128

129

Since most policies are defined based on regressions,

130

a hinted migration generally implies that the problem will not

131

prevent future migrations for newer versions of the same source

132

package (assuming that the problem is deterministic).

133

134

## Migration phase 2: Installability regression testing

135

136

For the migration items that pass the previous phase, britney

137

will do a test migration to see if anything becomes uninstallable.

138

This is a more expensive test to ensure the migration does not cause

139

installability regressions.

140

141

The status of this phase is *not* included in the excuses. To debug

142

problems here, the britney log file has to be examined. This requires

143

a bit more technical insight as it has not been polished as much as

144

the excuses.

145

146

### Confirming a migration

147

148

To start with; if a migration is accepted and "committed" (i.e. it will not

149

be rolled back), britney will include in a line starting with `final:` like

150

in this example:

151

152

Apparently successful

153

final: -cwltool,-libtest-redisserver-perl,-pinfo,-webdis,hol88

154

start: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

155

orig: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

156

end: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

157

SUCCESS (182/177)

158

159

The above example is a regular migration run where 4 source removal migration

160

items and one source migration item where accepted (those listed on the

161

`final:` line). The rest of the information are various statistical counters

162

which are useful for other purposes beyond the scope of this document.

163

164

When debugging a migration for an item that passed the previous phase, if the

165

item appears on a `final:` line like that, then it is migrated. That is, the

166

problem is most likely that the britney run crashes later or the britney's

167

output is not committed to the archive (for reasons outside britney's control).

168

169

On the flip side, if the migration item of interest does *not* appear in a

170

final line, then the migration was rejected (or rolled back).

171

172

Reminder: Migration items generally use the name of the source package. There

173

are exceptions to that "rule" (but they are not common cases covered by this

174

document).

175

176

### Debugging failed migration attempts

177

178

Start by confirming that the migration item was not accepted (as described

179

in the above section). If the migration item does not appear on a `final:` line,

180

then we need to debug the actual migration attempts. Migration attempts look

181

something like this:

182

183

trying: -webdis

184

accepted: -webdis

185

ori: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

186

pre: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

187

now: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

188

all: -pinfo -webdis

189

[...]

190

trying: libaws

191

skipped: libaws (0, 165, 11)

192

got: 45+0: a-4:i-27:a-5:a-1:a-1:m-0:m-3:m-1:p-1:s-2

193

* arm64: libaws-bin, libaws17.2.2017, libaws3.3.2.2-dev, liblog4ada3-dev

194

[...]

195

Trying easy from autohinter: asis/2017-1 dh-ada-library/6.12 [...]

196

start: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

197

orig: 41+0: a-4:i-27:a-1:a-1:a-1:m-0:m-3:m-1:p-1:s-2

198

easy: 261+0: a-26:i-49:a-23:a-23:a-23:m-22:m-25:m-23:p-23:s-24

199

* amd64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

200

* i386: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

201

* arm64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

202

* armel: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

203

[...]

204

FAILED

205

206

This example has one succeeding migration (`-webdis`) and one failing

207

(`libaws`) plus finally a failed `easy`-hint with several packages.

208

Both of the two first are "single item" migrations (i.e. the attempt only

209

includes a single item in isolation). However, Britney can do multi-item

210

migrations (even outside hints).

211

212

Please keep in mind that items can attempted multiple times and accepted in a

213

later attempt. It is not always immediately obvious, which attempt is better

214

for debugging. When in doubt, it is *usually* easiest to look at the attempt

215

with the least amount of new uninstallable packages.

216

217

In the libaws example, a total of 4 binary packages become uninstallable on the

218

architecture `arm64`. Here is the output again with this information high lighted:

219

220

migration item(s) being attemped

221

vvvvvv

222

trying: libaws

223

skipped: libaws (0, 165, 11)

224

got: 45+0: a-4:i-27:a-5:a-1:a-1:m-0:m-3:m-1:p-1:s-2

225

* arm64: libaws-bin, libaws17.2.2017, libaws3.3.2.2-dev, liblog4ada3-dev

226

^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

227

||||| The binary packages becoming uninstallable (here 4)

228

Affected architecture (here "arm64")

229

230

Please note that britney is lazy and will often reject an item after proving

231

that there is a regression on a single architecture. So in the above example,

232

we are not actually sure whether this problem is architecture specific. For

233

`easy`-hints, the information is presented slightly different.

234

235

Trying easy from autohinter: asis/2017-1 dh-ada-library/6.12 [...]

236

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

237

migration item(s) being attemped

238

239

[... several lines of statistics from start, before and after ...]

240

* amd64: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

241

^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

242

||||| The binary packages becoming uninstallable on amd64

243

Affected architecture (here "amd64")

244

245

* i386: asis-programs, libasis2017, libasis2017-dev, libaws-bin, [...]

246

^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

247

||||| The binary packages becoming uninstallable on i386

248

Affected architecture (here "i386")

249

[... more architectures with binary packages becoming uninstallable ...]

250

251

252

While this tells us what britney tried to migrate and what would break (become

253

uninstallable) as a result, it is not very helpful at explaining *why*

254

things break. If there are few broken packages, it is often a question of

255

looking for `Breaks`-relations or `Depends`-relations with upper bounds on

256

versions / on old packages being removed. Alternatively, there are also tools

257

like `dose-debcheck`, which attempts to analyse and explain problems like this.

258

259

# Common issues with policy decisions

260

261

## Britney complains about a fixed bug in the source suite (bug policy)

262

263

All decisions about bugs are related to data set extracted

264

from the bug tracker. If britney says that the new version

265

introduces a bug, then it is because the data set from the bug

266

tracker lists that bug for *a* version in the source suite and

267

without it appearing for the version(s) in the target suite.

268

269

Please note that these data sets do not include versions, so

270

britney is unable to tell exactly which versions are affected.

271

The only thing, it can tell, is what suite the bug affects.

272

273

There is a number of common cases, where this is observed:

274

275

* The metadata on the bug is wrong. A known example is the

276

Debian BTS, where if a bug has a `fixed` version equal to

277

a `found` version, the bug is considered unfixed.

278

279

* The bug is fixed, but the old version is still around in

280

the source suite. In this case, britney will generally

281

mention a "missing build" or "old binaries".

282

283

If the metadata is wrong, the solution is to fix it in the bug

284

tracker and wait until britney receives a new data set. In

285

the other case, the recommendation is to see the sections on

286

"missing builds" and "old binaries" below. As long as they

287

are present, the package may be blocked by bugs in the older

288

versions of the binaries.

289

290

## Britney complains about "missing builds"

291

292

A "missing build" happens when britney detects that the binaries

293

for a given architecture are missing or is not up to date. This

294

is detected by checking the "Packages" files in the archive, so

295

britney have no knowledge of *why* the build is missing.

296

Accordingly, this kind of issue is flagged as a "possibly permanent"

297

issue.

298

299

If the omission is deliberate (e.g. the new version no longer

300

supports that architecture), then please have the old binaries

301

for that architecture removed from the *source* suite. Once

302

those are removed, britney will no longer see that as a problem.

303

304

Otherwise, please check the build services for any issues with

305

building or uploading the package to the archive.

306

307

**Common misconceptions**: If the architecture is no longer

308

supported, the removal of the old binaries should happen in

309

the *source* suite (e.g. Debian unstable). However, many

310

people mistakenly request a removal from the *target* suite

311

(e.g. Debian testing). Unfortunately, this is not the proper

312

solution (and, britney does not support architecture

313

specific removals so it may be difficult to do anyhow).

314

315

## Britney complains about "old binaries"

316

317

Depending on the configuration of the britney instance, this may

318

or may not be a blocker. If the distribution has chosen to enable

319

the "ignore_cruft" option, this is merely a warning/note. That

320

said, even in this mode it can block a package from migration.

321

322

This appears when britney detects that there are older versions of

323

the binary packages around, which was built by (an older version of)

324

the same source package.

325

326

This is common with distributions where their archive management

327

software is capable of keeping old binaries as long as something

328

depends on them (e.g. DAK as used by Debian). Therefore, the

329

most common solution is to ensure all reverse dependencies are

330

updated to use the new binaries and then have the old ones

331

removed (the latter commonly known as "decrufting"). Technically,

332

this is also solvable by "decrufting" without updating/rebuilding

333

other packages. Though whether this is an acceptable practise

334

depends on the distribution.

335

336

Alternatively, if the distribution uses the "ignore_cruft" option,

337

this (in itself) is not a blocker. However, it commonly triggers

338

non-obvious issues:

339

340

* If the bugs policy is enabled, an bug in the old binaries that

341

is fixed in the new version will still be a blocker. Here, the

342

best solution is to get rid of the old binaries.

343

344

(Note: the bugs data is not versioned so britney cannot tell which

345

versions the bug applies to. Just which suite they affect)

346

347

* Even if the migration item is a valid candidate (i.e. all policy

348

checked have passed), it may cause installability regressions as

349

britney will also attempt to keep the old binaries around as long

350

as they are used. The most often cause of this when the old

351

binaries are not co-installable with the new ones.

352

353

(Note: Britney generally only works with the highest version of a

354

given binary. If you have libfoo1 depends on libfoo-data v1 and

355

then libfoo2 depends on libfoo-data v2, then libfoo1 will become

356

uninstallable as libfoo-data v2 will "shadow" libfoo-data v1)

357