Easy Backup

NOTE: This document is under construction. It is not any where near complete and it may be full of typos and wrong grammar. If you spot one, please let me know.

Starting from the desperation of loosing some works to a need of a backup solution. After looking around trying different existing solution without much satisfaction (well they are good just don't suit me). I sat down and made one. One of the big problem I had with those existing solutions is how can files/folders are set to be included or excluded from the backup. Most of them use pattern matching to do that and that is very hard to maintain. Other problems are multiple backup policies and a reminder to mount the backup device before the backup.

A few weekends later, EasyBackup is created. EasyBackup allows user to select the included and excluded files using Gnome emblems. Which make it very easy as you can see with Nautilus what included and what's not. It also make use of Nautilus script to provide quick way to revert and recover the files.

I don't see the program to be popular or anything near that. It is easy to use because it integrates well with Nautilus but the setup is not every user friendly and it programmed in PHP. As you know, Php is not every popular for desktop applications which make it less appeal to other programmer (I think). Due to the fact that I am not yet every familiar with how things are done in Python (its libraries) and I am quite a busy man right now. I don't see myself joining those projects. But I think (and I will be every happy to see) the idea can be ported to other language and platform (like KDE) or included into a better backup solution like TimeVault, SimpleHomeBackup or FlyBack. See design and implementation note below.

As I said, this is supposed to be for personal user for many reasons but after using it for sometimes, I think it can be useful for many people so here is it.

Features

Basic Backup & Restore operations

Backup
Scheduled backup
Revert files/folders
Recover files/folders

Integrate well with Gnome

Add includes/excludes using Gnome emblems
Revert and Recover files/folders with Nautilus Script

Multiple settings for different backup policy

Backup can be done by root or/and regular user
Allow backup to open terminal to see the backup progress (in case the connection is slow)
Allow difference backup target, includes/excludes, time to backup and other setting

Use rdiff-backup as the backend

The liability is good
Advance options of rdiff-backup is allowed
Display rdiff-backup command (for checking or tweak)

Automatic backup

Flexible way to set crobtab job
Reminder for preparing the backup target (such as mounting)
Expandable with Pre&Post backup script (can add mysql or svn backup or auto mount)

Pre-backup and Post-backup script

Call shell but can use php directly with it

Free and Open :D

Limitations

No automatic installation
No GUI tools for other task or setting

Configuration via configuration file editing
Scheduling the automatic backup need command line

To revert or recover from different setting, the config file must be swap
More ... :p

Future plan

An easier way to install
GUI for configuration
If people use it, I will put it on a project hosting (may be Google)
Show nautilus note of the file while recovering
Easy and intuitive way to swap the default config file

Usages

Before you can use EasyBackup, the setup and configuration must be done first (see the installation and configuration below). I put this part here because I want you to have a general picture about the program and you may see why it worth trying.

Backup

Ensure 'BackupDIR' entry in the default.ini is set to the backup directory (where the backup files will be) you want and a folder with the same name as your user name is created inside that BackupDIR with the permission setting that you can read and write from that folder.
Mark any files/folders you want them to be included in the backup with the "ToBackup" emblem.
Mark any files/folders inside those included that you want them to be excluded in the backup with the "ToNotBackup" emblem.
Run the backup. You can do one of this

Open the terminal and run

.EasyBackup/EasyBackup.php
Right click anywhere on your desktop and select Scripts->Easy Backup

Alternatively you can run ".EasyBackup/EasyBackup.php -l" in terminal. The '-l' option will tell EasyBackup to produce a log file that you can take a look. The log file will be in the folder ~/EasyBackup/logs

Revert

Open Nautilus and browse to the file/folder you want to revert to the older version.
Right click the file/folder and select Scripts->Easy Revert.
A dialog will ask the version of files you want to revert. select the one you want
Another dialog will ask for the path and file name of the reverted file (you can simply press enter to save the revert file over the current one)

Recover

Open Nautilus and browse to the folder in which the file you want to recover use to be.
Right-click anywhere and select Scripts->Easy Recover.
A dialog will should a list of files/folders that used to be in this folder. You now can select the desired files
Easy Revert will be run on that selected files and you can recover the file as you revert to the older version of the file.

Schedule backup

At the moment, the only way to set up the automatic backup is to run EasyBackup in command line (no GUI, sorry).

Open terminal and the following command:

.EasyBackup/EasyBackup.php -I 10min
Well that's it :D

If you want to use a different configuration, simple add the option "--ConfigFile '<the config file name>'". The configuration file must be in ~/.EasyBackup folder.

Run EasyBackup with '--help' to see to specify the time for the scheduling. Notice that there are two option here '-i' and '-I'.

'-i' is to set an exact time. If you miss it (like the computer was off or you are using rdiff-backup at the time), EasyBackup will skip that schedule.
'-I' is to set 'after that time' time. So if you miss it, EasyBackup will try at the first possible minute after that time. This option is suggested especially for a less-frequency backup.

Installation

Prerequisite

You need to have the following package installed to use this (based on Ubuntu 8.04)

Php Command line (php5-cli)
Rdiff-Backup (rdiff-backup)

If you don't know how ....
In case you are in Ubuntu 8.04, simply click the links above.
If you are in Ubuntu 8.04, go here http://simplyubuntu.wordpress.com/2006/06/27/a-beginners-guide-to-installing-programs-in-ubuntu/.

Getting the program

Download the program from: here.

Setting up

Extract the file to the target folder: /home/<username>/.EasyBackup/

tar xvfz EasyBackup.tar.gz
cp -Rf EasyBackup/source/ .EasyBackup
cp -Rf EasyBackup/confs/default.ini .EasyBackup
Create a log folder at ~/.EasyBackup/logs and mark it when ToNotBackup emblems (there is no need to backup the log)

mkdir ~/.EasyBackup/logs
Set appropriate permission. EasyBackup.php, EasyRecover.php and EasyRevert.php a permission to execute but not write (for security)

chmod 555 .EasyBackup/*.php

Create link to the nautilus script folder EasyBackup.php, EasyRecover.php and EasyRevert.php to /home/<username>/.gnome2/nautilus-script/ Easy Backup, Easy Recover, Easy Revert.

mkdir -p ~/.gnome2/nautilus-scripts
cd ~/.gnome2/nautilus-scripts
ln -s ~/.EasyBackup/EasyBackup.php "Easy Backup"
ln -s ~/.EasyBackup/EasyRecover.php "Easy Recover"
ln -s ~/.EasyBackup/EasyRevert.php "Easy Revert"

Install Emblems - This have to be done manually in GUI. I really don't know how. BE CAREFUL, you CANNOT change it once you added (at least I don't know how :p).

Use Nautilus open to /home/<username>/EasyBackup/resource.
On the side bar, change from Places to Emblems
Drag backup.png in to the emblem area
Enter the name "ToBackup" NO SPACE, NO DOUBLE QUOTE.
Repeat the process for no_backup.png (name it as "ToNotBackup").
Repeat the process for system_backup.png (name it as "SystemBackup").
If you want to add more emblem of your own, you can BUT name it with all alphabets (no numbers, spaces or symbols) unless you know how to see its name that Gnome give.

Installation Script
For those who does are lazy to do it one by one step here is the installation script. This script just do part of the job, you still need to install emblems by yourself.

#!/bin/sh

# Prerequisite
sudo apt-get install php5-cli rdiff-backup

# Getting the program
wget ...

# Extracting the file
tar xvfz EasyBackup.tar.gz
cp -Rf EasyBackup/source/ .EasyBackup
cp -Rf EasyBackup/confs/default.ini .EasyBackup

# Create a log folder
mkdir ~/.EasyBackup/logs

# Set appropriate permission
chmod 555 EasyBackup.php EasyRecover.php EasyRevert.php

# Create nautilus script menu
mkdir -p ~/.gnome2/nautilus-scripts
cd ~/.gnome2/nautilus-scripts
ln -s ~/.EasyBackup/EasyBackup.php "Easy Backup"
ln -s ~/.EasyBackup/EasyRecover.php "Easy Recover"
ln -s ~/.EasyBackup/EasyRevert.php "Easy Revert"

After this you have to install an emblem by yourself. See Setting up: Step 5 above.

Configurations

Basic setup

For a basic user data (as oppose to administrative) backup, there is only one configuration item that is needed to be set. Then if you want to schedule an automatic backup, you need to run one command and then you are ready to go :D.

Step 1: Set the backup directory

Open the default configuration file '/home/<username>/.EasyBackup/default.ini'
Look for an item named 'BackupDIR' (on the line # 9) and change the path to where you want the backup file will go.
Create a folder with the same name of the user immediately inside the 'BackupDIR' and make it writable to the you

For example, if your username is 'user' and BackupDIR is set to "/media/Extra/Backup/", create a folder name 'user' inside it using

sudo mkdir -p "/media/Extra/Backup/user"

then set the permission

sudo chown user:user "/media/Extra/Backup/user"
sudo chmod 755 "/media/Extra/Backup/user"

Step 2: Setup automatic backup

Open terminal and run

.EasyBackup/EasyBackup.php -I 10min

This will set an automatic backup for every 10 minutes. You can set 1 day by using "-I 1day" or two weeks for "-I 2week". (accept min, hour, day, week, month).

That's it. :D

Advance setup

There are number of options you can set in the configuration files. I provide here a brief description here and more detail may be added later.

BackupDIR (path)
The path (with tail slashes) where you want the backup file to be. This path must be exclude from the backup either with emblems or exclude config item (see below).
BackupAsRoot (boolean)
Whether or not the backup is run as root.
OpenTerminal (boolean)
Whether or not a terminal windows will be opened when an automatic backup is invoked. (so that you can see its output)
RemoveOlderThan (time period)
The age of snapshots that should be deleted. For example, "2W" means that every snapshots older than 2 weeks will be deleted.
EmplemsToIncluded[] (array of emblem names)
An array of emblem names you use to flag included files/folders.
EmplemsToExcluded[] (array of emblem names)
An array of emblem names you use to flag excluded files/folders.
PreBackupCommand (shell command)
The shell command to be executed just before the backup is executed. If the command return a string starting with "EVALME: " the string will be executed (using php command eval). The building program, will remind you if the backup is not mounted.
PostBackupCommand (shell command)
The shell command to be executed just before the backup is executed. If the command return a string starting with "EVALME: " the string will be executed (using php command eval). The building program, will remind you if the backup is not mounted.
EnableCronJobEmail (boolean)
Whether or not the notification email will be send (by cron) when an automatic backup is run.
Parameters[] (array of rdiff-backup parameter)
The rdiff-backup parameter that is not includes or excludes (well you can put those here too if you want). These parameters will be added into the rdiff-backup command before includes and excludes so they are high-priority ones.
Includes[] (array of rdiff-backup --include item)
Here is where you add the includes that cannot be specified by emblems such as *.avi or cache folder. It use shell pattern matching.
Excludes[] (array of rdiff-backup --exclude item)
Here is where you add the includes that cannot be specified by emblems such as *.avi or cache folder. It use shell pattern matching.

NOTE: the emblem names that EasyBackup extract from gnome are not necessary the same as the one displayed in nautilus so not all emblem names can be use directly with EmplemsToIncluded[] and EmplemsToExcluded[] especially the built-in emblems. There is a way to check that but I have not had time to write a program to check that automatically. Therefore, for the moment, it is recommended that only the given emblems or the one created and added by you are to be used and the emblems name must be all alphabets (no number, no space and no symbol).

Example backup policy

I am currently using two backup policies. The more frequency and the less sequence.

The more frequency backup

The first one is now running every 10 minutes. I can run it more often like every minutes but I think that may be too much :p. I backup every 10 minutes because I have a bad habit of using Shift-Delete to delete files (I try to quit but it keep coming back). Here is what I did:

Set BackupDIR in default.ini to be a path of a folder in my internal hard drive (i.e. '/media/Extra/Backup/')
Set BackupAsRoot to be false. So that I don't need to type password every 10 minutes
Set OpenTerminal to be false. Because I don't need to see it every time.
Set RemoveOlderThan to 2W. This because my less frequency backup is scheduled to every 1 week so delete it after 2 weeks allow some fall back.
Set EmplemsToIncluded to "ToBackup" and EmplemsToExcluded to "ToNotBackup"
Only use ToBackup and ToNotBackup to files in my home folders (because it is not running as root)
Comment out PreBackupCommand and PostBackupCommand as the backup folder is always mounted.
The rest of the option is left alone.
Run '.EasyBackup/EasyBackup -I 10m' in terminal
DONE!!

Having files backup every 10 minutes allows me to be stupid :p deleting files I should not delete or save over files I should not do. Well, It give me some safety net. In theory, the more often you backup the less time it take to do each backup but in practice, there is some overhead activity rdiff-backup needs to do on each backup. I found that 10 minutes is an appropriate time.

The less frequency backup

This backup is set to be run every week using -I so it can't miss. I backup files in /etc and /var (my web page and database are there). The following steps are what I did:

Copy default.ini to another file name weekly.ini
Set BackupDIR in weekly.ini to a path on a removable media in my case I use a cheap (but slow) 8GB USB drive (i.e. '/media/SD8GBBackup/Backup/')
Set BackupAsRoot to be true. So that the backup can include the files that I cannot normally read.
Set OpenTerminal to be true. Because it only run once a week and it very slow (my slow USB drive) I need to see if the backup is success.
Set RemoveOlderThan to 6M (for 6 month). From more than a month of using, I think the 8GM drive can take my backup for much longer but I think having too much backup will overwhelm me when I try to recover it (just my two cents).
Set EmplemsToIncluded to "ToBackup" and "SystemBackup",and EmplemsToExcluded to "ToNotBackup"
Only use ToBackup and ToNotBackup to files in my home folders and use SystemBackup to everything that I want to backup weekly but not every 10 minutes which are /etc and /var.
Use PreBackupCommand and PostBackupCommand so that there will be a dialog warning me to plug in my USB drive for the backup and remind me to remove it when done.
The rest of the option is left alone.
Run '.EasyBackup/EasyBackup --ConfigFile weekly.ini -I 1W' in terminal
DONE!!

Both backup policies can be used at the same time. Since the weekly one is set using -I, it will try again a minute later if the backup is run at the same time with another backup (the 10 minute one) so you don't need to worry about missing a backup.

Contributions and Supports

If you use it and have any suggestion or even patchs. Please feel free to let me know.
If you want translate to other language or port it to other platform (like python KDE), Go for it!! :D
If you want to want to include it in to other project, just simple let me know. :D
If you use it and like it and want to support the project, consider donate ... please. I am a poor student and your donation will go to my tuition. : ( . Contact me.

Development Notes

Aim and Target

Well, to backup my files.
To show that there is a simple but yet intuitive and user friendly way to backup solution in Linux (without fancy 3D like on Mac).

Tools

I use PHP as it is one of my favorite language and the language which I have quite an experience on.
I use Zenity a lot because GUI with PHP is not very easy to set up for the moment.
I use Nautilus script because it is very intuitive and require very less learning curve.

Design and Implementation

I intentionally make PHP calls shell command for many parsing activities (thinks to grep) rather doing it with PHP so that it will be quite easy to port to other language (like python or ruby). This results in some reliability and performance problems. But most of them are avoidable and does not cause major problems.
I tried my best (as time allows) to make the program easy to read and study like lots of comments and separating functions (those are quite general) and sub-routines (specific for EasyBackup) so people can take a look. Functions are those in Lib_...php files and Sub-routine are those in the application files.
EasyBackup treat symbolic link as a file meaning that if you include or exclude a symbolic link. You include or exclude the file the link is point to.

Backup

To get the list of files with emblems, I 'cat' all files in ~/.nautilus/metafiles/. Those files named followed directory of the files with emblems and other info (like Note). Then, I use grep to parse file with the entry <file name="..."><keyword name="">...</file>. Which give as a list of the meta file name (which is the folder name), the entry <file name=".."> which contain the file name and the keyword (name of the marked emblem). After a few more string manipulation in PHP, I collect the file list and use it in includes and excludes. (See detail in the function getListsFromSystemEmblems(...) in Lib_Gnome.inc).
The list of includes and excludes are combine and collapse. For example if you include /home/user/Documents and include /home/user/Documents/Works, the program will remove /home/user/Documents/Works since it already been included by the inclusion of /home/user/Documents. This is needed because rdiff-backup will scream if you don't. This is done by looking for the closest file folder name in the list and see if the closest folder has the same Include/Exclude with the file/folder in question. If they do have the same status, delete the file/folder from the list. See more detail in combineExcludesIncludes(...) of EasyBackup.php.
After that, I inversedly sort the file name to make sure that the inner folder comes before the outer ones (higher priority comes first). Then I add the prefix "--include" and "--exclude" accordingly. If root folder is included, I remove it from the list as that is the default. See more detail in combineExcludesIncludes(...) and getCommandPrefixFromConfig() of EasyBackup.php.
Finally, I combine parameters, the includes/excludes from the configurations with the includes/exclude from emblems to create a rdiff-backup command. The parameters are put first, then the I/E from the config and lastly the I/E from the emblems in order to ORDER the appropriate priority of all the parameters. The command should look like this:

rdiff-backup <params from config> <includes/excludes from config> <includes/excludes from emblems> / <BackupDIR>/<username>

See more detail in getCommandPrefixFromConfig() of EasyBackup.php.
The next step is to execute the pre-backup script. I want a flexible way to run shell script and php. The result is that the pre-backup script will be a shell script. but if the return of the execution of the shell script starts with "EVALME: ", the sub-sequencing string will be eval as if it is a php code.
Before the backup is actually begin, EasyBackup check for the last time (it checks once already at the beginning) if another instance of rdiff-backup is running. If not so, the backup process begin.
If the backup is to be run as root, the "device" environment parameter will be set to :0.0 and gksudo is run. This is to ensure that the GUI of gksudo can run even if this program is run via cron.

Logging
Schedule Backup
Recover
Revert

Know Problems

I am not seems to be able to use --max-file-size or --min-file-size for some unknown reason. I'll explain later.
Nautilus sometime (I am sure there is a reason but I just don't know it) give a wrong path to nautilus script when the file is on the desktop (not folder Desktop on file browser) so Revert and Recover for desktop may not work properly.
Once in a while, rdiff-backup will fail to backup from crontab. After investigating the issue, I found that if rdiff-backup stop running before it complete the backup (like when the computer is turned off because it done), broken pipe error will be thrown and every sub-sequence backup will fail with the same error. I can't seem to find a way to prevent this but there is something we can do after this happened. Simply run the backup manually, the problem will go away.
Because of this, I made EasyBackup to copy the log file to your desktop if it detect an error in the log file. Once you see log files start appearing on your desktop, you can check what happened and if you see the broken pipe. Just run the backup again.
There is an inconsistency in recognizing if a file has been deleted. Sometime, just delete the file is enough to make EasyBackup recognize (when you run EasyRecover), however, sometime, a backup need to be run before the file is recognized as deleted and listed in EasyRecover list. I will investigate this problem in the near future.