4
\section*{Automatic Volume Recycling}
5
\label{_ChapterStart22}
6
\index[general]{Recycling!Automatic Volume }
7
\index[general]{Automatic Volume Recycling }
8
\addcontentsline{toc}{section}{Automatic Volume Recycling}
10
By default, once Bacula starts writing a Volume, it can
11
append to the volume, but it will not overwrite the existing
12
data thus destroying it.
13
However when Bacula {\bf recycles} a Volume, the Volume becomes
14
available for being reused, and Bacula can at some later time
15
over write the previous contents of that Volume.
16
Thus all previous data will be lost. If the Volume is a tape,
17
the tape will be rewritten from the beginning. If the Volume is
18
a disk file, the file will be truncated before being rewritten.
20
You may not want Bacula to automatically recycle (reuse) tapes. This would
21
require a large number of tapes though, and in such a case, it is possible
22
to manually recycle tapes. For more on manual recycling, see the section
23
entitled \ilink{ Manually Recycling Volumes}{manualrecycling} below in this
26
Most people prefer to have a Pool of tapes that are used for daily backups and
27
recycled once a week, another Pool of tapes that are used for Full backups
28
once a week and recycled monthly, and finally a Pool of tapes that are used
29
once a month and recycled after a year or two. With a scheme like this, the
30
number of tapes in your pool or pools remains constant.
32
By properly defining your Volume Pools with appropriate Retention periods,
33
Bacula can manage the recycling (such as defined above) automatically.
35
Automatic recycling of Volumes is controlled by three records in the {\bf
36
Pool} resource definition in the Director's configuration file. These three
41
\item VolumeRetention = \lt{}time\gt{}
45
Automatic recycling of Volumes is performed by Bacula only when it wants a
46
new Volume and no appendable Volumes are available in the Pool. It will then
47
search the Pool for any Volumes with the {\bf Recycle} flag set and whose
48
Volume Status is {\bf Full}. At that point, the recycling occurs in two steps.
49
The first is that the Catalog for a Volume must be purged of all Jobs and
50
Files contained on that Volume, and the second step is the actual recycling of
51
the Volume. The Volume will be purged if the VolumeRetention period has
52
expired. When a Volume is marked as Purged, it means that no Catalog records
53
reference that Volume, and the Volume can be recycled. Until recycling
54
actually occurs, the Volume data remains intact. If no Volumes can be found
55
for recycling for any of the reasons stated above, Bacula will request
56
operator intervention (i.e. it will ask you to label a new volume).
58
A key point mentioned above, that can be a source of frustration, is that Bacula
59
will only recycle purged Volumes if there is no other appendable Volume
60
available, otherwise, it will always write to an appendable Volume before
61
recycling even if there are Volume marked as Purged. This preserves your data
62
as long as possible. So, if you wish to "force" Bacula to use a purged
63
Volume, you must first ensure that no other Volume in the Pool is marked {\bf
64
Append}. If necessary, you can manually set a volume to {\bf Full}. The reason
65
for this is that Bacula wants to preserve the data on your old tapes (even
66
though purged from the catalog) as long as absolutely possible before
70
\subsection*{Automatic Pruning}
71
\index[general]{Automatic Pruning }
72
\index[general]{Pruning!Automatic }
73
\addcontentsline{toc}{subsection}{Automatic Pruning}
75
As Bacula writes files to tape, it keeps a list of files, jobs, and volumes in
76
a database called the catalog. Among other things, the database helps Bacula
77
to decide which files to back up in an incremental or differential backup, and
78
helps you locate files on past backups when you want to restore something.
79
However, the catalog will grow larger and larger as time goes on, and
80
eventually it can become unacceptably large.
82
Bacula's process for removing entries from the catalog is called Pruning. The
83
default is Automatic Pruning, which means that once an entry reaches a certain
84
age (e.g. 30 days old) it is removed from the catalog. Once a job has been
85
pruned, you can still restore it from the backup tape, but one additional step
86
is required: scanning the volume with bscan. The alternative to Automatic
87
Pruning is Manual Pruning, in which you explicitly tell Bacula to erase the
88
catalog entries for a volume. You'd usually do this when you want to reuse a
89
Bacula volume, because there's no point in keeping a list of files that USED
90
TO BE on a tape. Or, if the catalog is starting to get too big, you could
91
prune the oldest jobs to save space. Manual pruning is done with the
92
\ilink{ prune command}{ManualPruning} in the console. (thanks to
93
Bryce Denney for the above explanation).
95
\subsection*{Prunning Directives}
96
\index[general]{Prunning Directives }
97
\index[general]{Directives!Prunning }
98
\addcontentsline{toc}{subsection}{Prunning Directives}
100
There are three pruning durations. All apply to catalog database records and
101
not to the actual data in a Volume. The pruning (or retention) durations are
102
for: Volumes (Media records), Jobs (Job records), and Files (File records).
103
The durations inter-depend a bit because if Bacula prunes a Volume, it
104
automatically removes all the Job records, and all the File records. Also when
105
a Job record is pruned, all the File records for that Job are also pruned
106
(deleted) from the catalog.
108
Having the File records in the database means that you can examine all the
109
files backed up for a particular Job. They take the most space in the catalog
110
(probably 90-95\% of the total). When the File records are pruned, the Job
111
records can remain, and you can still examine what Jobs ran, but not the
112
details of the Files backed up. In addition, without the File records, you
113
cannot use the Console restore command to restore the files.
115
When a Job record is pruned, the Volume (Media record) for that Job can still
116
remain in the database, and if you do a "list volumes", you will see the
117
volume information, but the Job records (and its File records) will no longer
120
In each case, pruning removes information about where older files are, but it
121
also prevents the catalog from growing to be too large. You choose the
122
retention periods in function of how many files you are backing up and the
123
time periods you want to keep those records online, and the size of the
124
database. You can always re-insert the records (with 98\% of the original data)
125
by using "bscan" to scan in a whole Volume or any part of the volume that
128
By setting {\bf AutoPrune} to {\bf yes} you will permit {\bf Bacula} to
129
automatically prune all Volumes in the Pool when a Job needs another Volume.
130
Volume pruning means removing records from the catalog. It does not shrink the
131
size of the Volume or affect the Volume data until the Volume gets
132
overwritten. When a Job requests another volume and there are no Volumes with
133
Volume Status {\bf Append} available, Bacula will begin volume pruning. This
134
means that all Jobs that are older than the {\bf VolumeRetention} period will
135
be pruned from every Volume that has Volume Status {\bf Full} or {\bf Used}
136
and has Recycle set to {\bf yes}. Pruning consists of deleting the
137
corresponding Job, File, and JobMedia records from the catalog database. No
138
change to the physical data on the Volume occurs during the pruning process.
139
When all files are pruned from a Volume (i.e. no records in the catalog), the
140
Volume will be marked as {\bf Purged} implying that no Jobs remain on the
141
volume. The Pool records that control the pruning are described below.
145
\item [AutoPrune = \lt{}yes|no\gt{}]
146
\index[console]{AutoPrune }
147
If AutoPrune is set to {\bf yes} (default), Bacula
148
will automatically apply the Volume retention period when running a Job and
149
it needs a new Volume but no appendable volumes are available. At that point,
150
Bacula will prune all Volumes that can be pruned (i.e. AutoPrune set) in an
151
attempt to find a usable volume. If during the autoprune, all files are
152
pruned from the Volume, it will be marked with VolStatus {\bf Purged}. The
153
default is {\bf yes}. Note, that although the File and Job records may be
154
pruned from the catalog, a Volume will be marked Purged (and hence
155
ready for recycling) if the Volume status is Append, Full, Used, or Error.
156
If the Volume has another status, such as Archive, Read-Only, Disabled,
157
Busy, or Cleaning, the Volume status will not be changed to Purged.
159
\item [Volume Retention = \lt{}time-period-specification\gt{}]
160
\index[console]{Volume Retention }
161
The Volume Retention record defines the length of time that Bacula will
162
guarantee that the Volume is not reused counting from the time the last job
163
stored on the Volume terminated.
165
When this time period expires, and if {\bf AutoPrune} is set to {\bf yes},
166
and a new Volume is needed, but no appendable Volume is available, Bacula
167
will prune (remove) Job records that are older than the specified Volume
170
The Volume Retention period takes precedence over any Job Retention period
171
you have specified in the Client resource. It should also be noted, that the
172
Volume Retention period is obtained by reading the Catalog Database Media
173
record rather than the Pool resource record. This means that if you change
174
the VolumeRetention in the Pool resource record, you must ensure that the
175
corresponding change is made in the catalog by using the {\bf update pool}
176
command. Doing so will insure that any new Volumes will be created with the
177
changed Volume Retention period. Any existing Volumes will have their own
178
copy of the Volume Retention period that can only be changed on a Volume by
179
Volume basis using the {\bf update volume} command.
181
When all file catalog entries are removed from the volume, its VolStatus is
182
set to {\bf Purged}. The files remain physically on the Volume until the
183
volume is overwritten.
185
Retention periods are specified in seconds, minutes, hours, days, weeks,
186
months, quarters, or years on the record. See the
187
\ilink{Configuration chapter}{Time} of this manual for
188
additional details of time specification.
190
The default is 1 year.
192
\item [Recycle = \lt{}yes|no\gt{}]
194
This statement tells Bacula whether or not the particular Volume can be
195
recycled (i.e. rewritten). If Recycle is set to {\bf no} (the default), then
196
even if Bacula prunes all the Jobs on the volume and it is marked {\bf
197
Purged}, it will not consider the tape for recycling. If Recycle is set to
198
{\bf yes} and all Jobs have been pruned, the volume status will be set to
199
{\bf Purged} and the volume may then be reused when another volume is needed.
200
If the volume is reused, it is relabeled with the same Volume Name, however
201
all previous data will be lost.
204
It is also possible to "force" pruning of all Volumes in the Pool
205
associated with a Job by adding {\bf Prune Files = yes} to the Job resource.
208
\subsection*{Recycling Algorithm}
209
\index[general]{Algorithm!Recycling }
210
\index[general]{Recycling Algorithm }
211
\addcontentsline{toc}{subsection}{Recycling Algorithm}
213
After all Volumes of a Pool have been pruned (as mentioned above, this happens
214
when a Job needs a new Volume and no appendable Volumes are available), Bacula
215
will look for the oldest Volume that is Purged (all Jobs and Files expired),
216
and if the {\bf Recycle} flag is on (Recycle=yes) for that Volume, Bacula will
217
relabel it and write new data on it.
219
The full algorithm that Bacula uses when it needs a new Volume is:
222
\item Search the Pool for a Volume with VolStatus=Append (if there is more
223
than one, the Volume with the oldest date last written is chosen. If
224
two have the same date then the one with the lowest MediaId is chosen).
225
\item Search the Pool for a Volume with VolStatus=Recycle and the InChanger
226
flag is set true (if there is more than one, the Volume with the oldest
227
date last written is chosen. If two have the same date then the one
228
with the lowest MediaId is chosen).
229
\item Try recycling any purged Volumes.
230
\item Prune volumes applying Volume retention period (Volumes with VolStatus
231
Full, Used, or Append are pruned).
232
\item Search the Pool for a Volume with VolStatus=Purged
233
\item If InChanger was set, go back to the first step above, but
234
this second time, ignore the InChanger flag in step 2.
235
\item Attempt to create a new Volume if automatic labeling enabled
236
If Python is enabled, a Python NewVolume even is generated before
237
the Label Format check is used.
238
\item If a Pool named "Scratch" exists, search for a Volume and if found
239
move it to the current Pool for the Job and use it.
240
\item Prune the oldest Volume if RecycleOldestVolume=yes (the Volume with the
241
oldest LastWritten date and VolStatus equal to Full, Recycle, Purged, Used,
242
or Append is chosen). This record ensures that all retention periods are
244
\item Purge the oldest Volume if PurgeOldestVolume=yes (the Volume with the
245
oldest LastWritten date and VolStatus equal to Full, Recycle, Purged, Used,
246
or Append is chosen). We strongly recommend against the use of {\bf
247
PurgeOldestVolume} as it can quite easily lead to loss of current backup
249
\item Give up and ask operator.
252
The above occurs when Bacula has finished writing a Volume or when no Volume
253
is present in the drive.
255
On the other hand, if you have inserted a different Volume after the last job,
256
and Bacula recognizes the Volume as valid, it will request authorization from
257
the Director to use this Volume. In this case, if you have set {\bf Recycle
258
Current Volume = yes} and the Volume is marked as Used or Full, Bacula will
259
prune the volume and if all jobs were removed during the pruning (respecting
260
the retention periods), the Volume will be recycled and used.
261
The recycling algorithm in this case is:
264
\item If the VolStatus is {\bf Append} or {\bf Recycle} and {\bf Accept Any
265
Volume} is set, the volume will be used.
266
\item If {\bf Recycle Current Volume} is set and the volume is marked {\bf
267
Full} or {\bf Used}, Bacula will prune the volume (applying the retention
268
period). If all Jobs are pruned from the volume, it will be recycled.
271
This permits users to manually change the Volume every day and load tapes in
272
an order different from what is in the catalog, and if the volume does not
273
contain a current copy of your backup data, it will be used.
275
\subsection*{Recycle Status}
276
\index[general]{Status!Recycle }
277
\index[general]{Recycle Status }
278
\addcontentsline{toc}{subsection}{Recycle Status}
280
Each Volume inherits the Recycle status (yes or no) from the Pool resource
281
record when the Media record is created (normally when the Volume is labeled).
282
This Recycle status is stored in the Media record of the Catalog. Using
283
the Console program, you may subsequently change the Recycle status for each
284
Volume. For example in the following output from {\bf list volumes}:
288
+----------+-------+--------+---------+------------+--------+-----+
289
| VolumeNa | Media | VolSta | VolByte | LastWritte | VolRet | Rec |
290
+----------+-------+--------+---------+------------+--------+-----+
291
| File0001 | File | Full | 4190055 | 2002-05-25 | 14400 | 1 |
292
| File0002 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
293
| File0003 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
294
| File0004 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
295
| File0005 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
296
| File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
297
| File0007 | File | Purged | 1896466 | 2002-05-26 | 14400 | 1 |
298
+----------+-------+--------+---------+------------+--------+-----+
302
all the volumes are marked as recyclable, and the last Volume, {\bf File0007}
303
has been purged, so it may be immediately recycled. The other volumes are all
304
marked recyclable and when their Volume Retention period (14400 seconds or 4
305
hours) expires, they will be eligible for pruning, and possibly recycling.
306
Even though Volume {\bf File0007} has been purged, all the data on the Volume
307
is still recoverable. A purged Volume simply means that there are no entries
308
in the Catalog. Even if the Volume Status is changed to {\bf Recycle}, the
309
data on the Volume will be recoverable. The data is lost only when the Volume
310
is re-labeled and re-written.
312
To modify Volume {\bf File0001} so that it cannot be recycled, you use the
313
{\bf update volume pool=File} command in the console program, or simply {\bf
314
update} and Bacula will prompt you for the information.
318
+----------+------+-------+---------+-------------+-------+-----+
319
| VolumeNa | Media| VolSta| VolByte | LastWritten | VolRet| Rec |
320
+----------+------+-------+---------+-------------+-------+-----+
321
| File0001 | File | Full | 4190055 | 2002-05-25 | 14400 | 0 |
322
| File0002 | File | Full | 1897236 | 2002-05-26 | 14400 | 1 |
323
| File0003 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
324
| File0004 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
325
| File0005 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
326
| File0006 | File | Full | 1896460 | 2002-05-26 | 14400 | 1 |
327
| File0007 | File | Purged| 1896466 | 2002-05-26 | 14400 | 1 |
328
+----------+------+-------+---------+-------------+-------+-----+
332
In this case, {\bf File0001} will never be automatically recycled. The same
333
effect can be achieved by setting the Volume Status to Read-Only.
335
\subsection*{Making Bacula Use a Single Tape}
337
\index[general]{Tape!Making Bacula Use a Single }
338
\index[general]{Making Bacula Use a Single Tape }
339
\addcontentsline{toc}{subsection}{Making Bacula Use a Single Tape}
341
Most people will want Bacula to fill a tape and when it is full, a new tape
342
will be mounted, and so on. However, as an extreme example, it is possible for
343
Bacula to write on a single tape, and every night to rewrite it. To get this
344
to work, you must do two things: first, set the VolumeRetention to less than
345
your save period (one day), and the second item is to make Bacula mark the
346
tape as full after using it once. This is done using {\bf UseVolumeOnce =
347
yes}. If this latter record is not used and the tape is not full after the
348
first time it is written, Bacula will simply append to the tape and eventually
349
request another volume. Using the tape only once, forces the tape to be marked
350
{\bf Full} after each use, and the next time {\bf Bacula} runs, it will
353
An example Pool resource that does this is:
359
Use Volume Once = yes
362
VolumeRetention = 12h # expire after 12 hours
368
\subsection*{A Daily, Weekly, Monthly Tape Usage Example}
370
\index[general]{Daily, Weekly, Monthly Tape Usage Example }
371
\index[general]{Example!Daily Weekly Monthly Tape Usage }
372
\addcontentsline{toc}{subsection}{Daily, Weekly, Monthly Tape Usage Example}
374
This example is meant to show you how one could define a fixed set of volumes
375
that Bacula will rotate through on a regular schedule. There are an infinite
376
number of such schemes, all of which have various advantages and
379
We start with the following assumptions:
382
\item A single tape has more than enough capacity to do a full save.
383
\item There are 10 tapes that are used on a daily basis for incremental
384
backups. They are prelabeled Daily1 ... Daily10.
385
\item There are 4 tapes that are used on a weekly basis for full backups.
386
They are labeled Week1 ... Week4.
387
\item There are 12 tapes that are used on a monthly basis for full backups.
388
They are numbered Month1 ... Month12
389
\item A full backup is done every Saturday evening (tape inserted Friday
390
evening before leaving work).
391
\item No backups are done over the weekend (this is easy to change).
392
\item The first Friday of each month, a Monthly tape is used for the Full
394
\item Incremental backups are done Monday - Friday (actually Tue-Fri
398
We start the system by doing a Full save to one of the weekly volumes or one
399
of the monthly volumes. The next morning, we remove the tape and insert a
400
Daily tape. Friday evening, we remove the Daily tape and insert the next tape
401
in the Weekly series. Monday, we remove the Weekly tape and re-insert the
402
Daily tape. On the first Friday of the next month, we insert the next Monthly
403
tape in the series rather than a Weekly tape, then continue. When a Daily tape
404
finally fills up, {\bf Bacula} will request the next one in the series, and
405
the next day when you notice the email message, you will mount it and {\bf
406
Bacula} will finish the unfinished incremental backup.
408
What does this give? Well, at any point, you will have the last complete
409
Full save plus several Incremental saves. For any given file you want to
410
recover (or your whole system), you will have a copy of that file every day
411
for at least the last 14 days. For older versions, you will have at least 3
412
and probably 4 Friday full saves of that file, and going back further, you
413
will have a copy of that file made on the beginning of the month for at least
416
So you have copies of any file (or your whole system) for at least a year, but
417
as you go back in time, the time between copies increases from daily to weekly
420
What would the Bacula configuration look like to implement such a scheme?
426
Run = Level=Full Pool=Monthly 1st sat at 03:05
427
Run = Level=Full Pool=Weekly 2nd-5th sat at 03:05
428
Run = Level=Incremental Pool=Daily tue-fri at 03:05
434
Client = LocalMachine
439
Schedule = "NightlySave"
441
# Definition of file storage device
446
Password = XXXXXXXXXXXXX
452
Include = signature=MD5 {
461
VolumeRetention = 10d # recycle in 10 days
467
Use Volume Once = yes
470
VolumeRetention = 30d # recycle in 30 days (default)
475
Use Volume Once = yes
478
VolumeRetention = 365d # recycle in 1 year
484
\subsection*{ Automatic Pruning and Recycling Example}
485
\label{PruningExample}
486
\index[general]{Automatic Pruning and Recycling Example }
487
\index[general]{Example!Automatic Pruning and Recycling }
488
\addcontentsline{toc}{subsection}{Automatic Pruning and Recycling Example}
490
Perhaps the best way to understand the various resource records that come into
491
play during automatic pruning and recycling is to run a Job that goes through
492
the whole cycle. If you add the following resources to your Director's
498
Name = "30 minute cycle"
499
Run = Level=Full Pool=File Messages=Standard Storage=File
501
Run = Level=Full Pool=File Messages=Standard Storage=File
513
Schedule = "30 minute cycle"
515
# Definition of file storage device
518
Address = XXXXXXXXXXX
520
Password = XXXXXXXXXXXXX
526
Include = signature=MD5 {
533
Use Volume Once = yes
544
Where you will need to replace the {\bf ffffffffff}'s by the appropriate files
545
to be saved for your configuration. For the FileSet Include, choose a
546
directory that has one or two megabytes maximum since there will probably be
547
approximately 8 copies of the directory that {\bf Bacula} will cycle through.
549
In addition, you will need to add the following to your Storage daemon's
557
Archive Device = /tmp
560
AutomaticMount = yes;
567
With the above resources, Bacula will start a Job every half hour that saves a
568
copy of the directory you chose to /tmp/File0001 ... /tmp/File0012. After 4
569
hours, Bacula will start recycling the backup Volumes (/tmp/File0001 ...). You
570
should see this happening in the output produced. Bacula will automatically
571
create the Volumes (Files) the first time it uses them.
573
To turn it off, either delete all the resources you've added, or simply
574
comment out the {\bf Schedule} record in the {\bf Job} resource.
576
\subsection*{Manually Recycling Volumes}
577
\label{manualrecycling}
578
\index[general]{Volumes!Manually Recycling }
579
\index[general]{Manually Recycling Volumes }
580
\addcontentsline{toc}{subsection}{Manually Recycling Volumes}
582
Although automatic recycling of Volumes is implemented in version 1.20 and
584
\ilink{Automatic Recycling of Volumes}{_ChapterStart22} chapter of
585
this manual), you may want to manually force reuse (recycling) of a Volume.
587
Assuming that you want to keep the Volume name, but you simply want to write
588
new data on the tape, the steps to take are:
591
\item Use the {\bf update volume} command in the Console to ensure that the
592
{\bf Recycle} field is set to {\bf 1}
593
\item Use the {\bf purge jobs volume} command in the Console to mark the
594
Volume as {\bf Purged}. Check by using {\bf list volumes}.
597
Once the Volume is marked Purged, it will be recycled the next time a Volume
600
If you wish to reuse the tape by giving it a new name, follow the following
604
\item Use the {\bf purge jobs volume} command in the Console to mark the
605
Volume as {\bf Purged}. Check by using {\bf list volumes}.
606
\item In Bacula version 1.30 or greater, use the Console {\bf relabel}
607
command to relabel the Volume.
610
Please note that the relabel command applies only to tape Volumes.
612
For Bacula versions prior to 1.30 or to manually relabel the Volume, use the
616
\item Use the {\bf delete volume} command in the Console to delete the Volume
618
\item If a different tape is mounted, use the {\bf unmount} command,
619
remove the tape, and insert the tape to be renamed.
620
\item Write an EOF mark in the tape using the following commands:
624
mt -f /dev/nst0 rewind
629
where you replace {\bf /dev/nst0} with the appropriate device name on your
631
\item Use the {\bf label} command to write a new label to the tape and to
632
enter it in the catalog.
635
Please be aware that the {\bf delete} command can be dangerous. Once it is
636
done, to recover the File records, you must either restore your database as it
637
was before the {\bf delete} command, or use the {\bf bscan} utility program to
638
scan the tape and recreate the database entries.
added
removed
manual-fr/recycling.tex
