The script starts by creating a copy of all of the important information in a staging area.  Keeping the staging area intact will help speed up the backup after it has been run once since rsync is used for most of the copying.  After the staging area is updated, we use rdiff-backup as our incremental backup system.  The rdiff-backup repository can then be copied to other local and offsite locations to increase reduncancy in the backup.  The output is sent by email every time the backup runs, so we are automatically updated on its status, and can correct any errors that may have taken place.

Relevant Tools

Following is a description of the tools that are used to back up our data, as well as some of the tools that are holding out data.  Different sets of data must be backed up in different was in order to maximize efficiency and reliability.

Hardware

We use a mixture of standard desktop hardware, server hardware, and virtual dedicated servers for our infrastructure.  Services requiring reliability are all run on the virtual dedicated servers providing uptime guarantees.  All of our backups are gathered and stored on two local computers, each with a raid 1 array, and copied to an offsite computer.

Bash

http://www.gnu.org/software/bash/manual/bashref.html

Bash is an interactive shell and a scripting language that is installed by default on Linux, and can be configured on many other operating system.  We use it for the backup system because it is easy to call other programs that are already designed to do a lot of the work that we need to do for the backup.  This gets us a full backup in less than 100 lines of code.  The small amount of code makes it easy to proof read and test for errors.

Rsync

http://samba.anu.edu.au/rsync/

Rsync is a very common program for copying data from one location to another when bandwidth can be limited, or when large amounts of data must be copied.  It first checks if the file exists in the destination, and, if it does exist, makes sure that it is that same file.  If the file already exists, and is up to date, the file is not copied.  For our backup, this saves a lot of time since there is generally only a small amount of files that change.

Rdiff-Backup

http://rdiff-backup.nongnu.org/

After the staging area has been updated, we use rdiff-backup to keep track of 10 days of history.  rdiff-backup copies files from the source to an rdiff-backup repository.  Instead of deleting files that have been removed and overwriting files that have changed, it stores a backup of any changes.  It is possible to go into the repository later and restore any version that has not been removed.

OpenVZ

http://wiki.openvz.org/

OpenVZ allows us to quickly and easily set up new virtualized environments, known as containers.  OpenVZ containers are not fully virtualized, so they do not require as many resources as traditional virtual machines.  Our OpenVZ containers are used to quickly and easily set up testing environments, and to provide some of our services within our office that do not have the uptime requirements of the services running on the offsite virtual dedicated servers.

Logical Volume Management

http://tldp.org/HOWTO/LVM-HOWTO/

Logical Volume Management (LVM) is an abstraction layer between the hard drive and the partitions and filesystems on that hard drive.  One or more hard drives or partitions, known as physical volumes in LVM language, can be combined into a volume group.  The volume group can then be divided into multiple logical volumes, which behave like partitions and can be formatted like a normal partition would be.  There are many features in LVM that provide an advantage over normal partition management including resizing partitions and volume groups while the filesystems are mounted, moving the logical volumes from one physical drive to another, also while the filesystems are mounted, and many other management tasks.  Our backup script takes advantage of the ability to create a snapshot of a filesystem.  A snapshot creates another logical volume that is an image of the first logical volume frozen in time.  Any changes that are made to the files on the original logical volume will be stored in the extra space at the end of the volume group that is dedicated to the snapshot when the snapshot is created.  Business can continue as normal as we create a backup from a consistent set of files where we no longer need to worry about someone writing to the files while we are backing them up.

MySQL

http://www.mysql.com/

MySQL is a database that comes with it’s own set of backup tools to ensure that you get a consistent copy of the database.  Mysqldump locks the database when necessary to make sure that it does not make a copy of the database when some other program is in the middle of making a change to the database.

Subversion

http://subversion.apache.org/

Subversion is a version control system.  We use it for all of our development projects to ensure that each team of developers is working with the most recent version of the code.  Because all of our projects use subversion, we have built up a large collection of code that would take a long time to fully back up each day.  We use subversion tools along with some bash scripting to compare the most recent version of the code in version control with the version that we last backed up, and only back up the changes.

Grep, Sed, and Others

http://linuxmanpages.com/

There are many other tools used throughout the script.  The two biggest examples are grep, a regular expression program, and sed, a stream editor which takes some input and changes it as specified.  If you want to know more about any of these tools, you can look them up in the Linux manual pages, or you can find more information on them using your favorite search engine.

Variables

Following are the variables that are used in this script.  Some of the values have been changed for security reasons.

# space seperated list of servers that have openvz and lvm set up
CONTAINER_SERVERS="fox walrus"
# server to put a second copy of the backup on, located in MN
SECONDARY_BACKUP="walrus"
# offsite server
OFFSITE_BACKUP="offsite.server.com"
# Location on the openvz servers where the lvm snapshot of the openvz partition is mounted
SRC="/vzsnap/"
# Location on the main backup server to collect all files
STAGING="/backup/staging/"
# Location of the rdiff-backup repository
RECENT="/backup/recent_backup/"
# Location on the second server for a copy of the rdiff-backup repository
SECOND_RECENT="/backup/recent_backup/"
# Location on the offsite server for a copy of the rdiff-backup repository
OFFSITE_RECENT="/home/backup/recent_backup/"
# Command locations
LVCREATE=/sbin/lvcreate
LVREMOVE=/sbin/lvremove
VZLIST=/usr/sbin/vzlist
VZCTL=/usr/sbin/vzctl

OpenVZ Containers

# containers
echo "Staging Containers"
for host in $CONTAINER_SERVERS
do
   containerlist=`ssh $host $VZLIST -aH -o veid`
   for container in $containerlist
   do
      echo "Backing up $container from $host"
      ssh $host $LVCREATE -L2G -s -n vzsnap /dev/filestore/vz > /dev/null

      # check if the container is running
      run="`ssh $host vzlist | grep --only-matching $container`"
      if [ "$run" == "$container" ]
      then
         # check if mysql server is running inside the container
         if [ -f /vz/private/$container/etc/init.d/mysql ]
         then
            ssh $host "vzctl exec $container 'if [ -f /var/run/mysqld/mysqld.pid ]; then echo mysql is running; echo /var/run/mysqld/mysqld.pid > /mysqld.pid; fi'"
            if [ -f /vz/private/$container/mysqld.pid ]
            then
               pass="`ssh $host cat /vz/private/$container/etc/mysql/debian.cnf | grep --max-count=1 password | sed 's/^password = //'`"
               user="debian-sys-maint"
               ssh $host vzctl exec $container mysqldump --user=$user -p$pass --all-databases > /backup/staging/$container.dump
               ssh $host vzctl exec $container rm /mysqld.pid
            fi
         fi
      fi

      ssh $host mkdir -p $SRC
      ssh $host mount /dev/filestore/vzsnap $SRC
      rsync -a --delete root@$host:$SRC/private/$container $STAGING/
      rsync -a root@$host:$SRC/etc/vz/conf/$container.conf $STAGING/
      ssh $host umount $SRC
      ssh $host $LVREMOVE /dev/filestore/vzsnap --force > /dev/null
      ssh $host rmdir $SRC
   done
done

For each server that is running OpenVZ containers, we want to make a full backup of each of the containers.

1. We use LVM to create a snapshot of the filesystem.  This gives us a version of the files that we can copy that is consistent with itself at a given point in time.
2. In the case that MySQL is running in the container, we want to make sure that we do not get a corrupt copy of the database just in case it was in the middle of a commit when the LVM snapshot was taken.  For this, we will use the standard MySQL backup tool mysqldump.  This will also allow us to easily examine an old version of the database without fully restoring the entire container.
3. Finally, we copy the configuration for the container, and all of the files within the container.

This will give us enough to restore any of our containers to the point at which the backup was taken.

Version Control (SVN)

echo "Staging Source Control"
REPOS=`ssh scm.mentormate.com ls /files/repo/svn/`
echo $REPOS
for repo in $REPOS
do
   echo "processing svn repo $repo"
   head=`svn info svn+ssh://scm.mentormate.com/files/repo/svn/$repo/ | grep "^Revision: .*" | sed 's/^Revision: //'`
   curfile=$STAGING/svn.$repo.lastversion
   # check if there is currently a backup of the repo
   if [ -f $curfile ]
   then
      # if yes, read the last saved file
      current=`cat $curfile`
   else
      # if no, start from the beginning
      current=0
   fi
   if [ $current -lt $head ]
   then
      if [ $current -ne 0 ]
      then
         current=$(( $current + 1 ))
      fi
      padcurrent=`printf "%05d" $current`
      padhead=`printf "%05d" $head`
      # take incremental backup
      ssh scm.mentormate.com svnadmin dump -r$current:$head --incremental /files/repo/svn/$repo | gzip --best > $STAGING/svn.$repo.$padcurrent-$padhead.dump.gz
      echo "$head" > $curfile
   else
      echo "$repo does not have any changes"
   fi
done

We use subversion for version control.   It’s very easy to dump a full backup of a subversion repository, but when the repositories get large, this can start to take a lot of time and bandwidth.  It is better to only back up the new revisions.

1. Get a list of all subversion repositories
2. The head is the version that was most recently committed to the repository.  This can be obtained by checking the svn info.
3. The head that we backed up the last time that we made a backup was saved to a file.  This will give us a range for the last backup that we made to the current head.
4. An incremental backup is taken of only the new revisions, and the current head is written to a file.  The filename will tell us which revisions were backed up.

When restoring this, the files can be combined and loaded into a new repository, or they can be loaded into a new repository one after the other.  If you decide that there are too many files for one repository, you can simply delete all of the files for that repository, including the last revision file, from the staging area, and a full backup will be taken the next time the backup script runs.

Backup Router Configuration

wget -q https://192.168.1.1/diag_backup.php --post-data 'Submit=Download' --no-check-certificate --http-user=admin --http-password="**********" --output-document=$STAGING/pfsense.xml

We use pfSense for our router software.  This allows us great flexibility at a very low cost.  pfSense has a web page with a button for downloading the configuration file.  Wget can be set to send the appropriate information to request the configuration file.  In the event that the router hardware fails, we can quickly install pfSense on new hardware, and load the configuration.

Crontab from the Backup Server

crontab -l > $STAGING/crontab.`hostname`

If our main backup server goes down, we want to make sure that we know everything that it was doing.  All of our important backups jobs run as a single user, so we only need to get one crontab here.

Staging to Incremental Backup

rdiff-backup $STAGING/ $RECENT/
rdiff-backup --remove-older-than 10B $RECENT/

Now that we have everything in the staging area, we want to copy it to a more long term location.  We will do this using rdiff-backup, which will store multiple revisions of our files without making a full copy of each one.  We then need to remove old revisions of the files so that the repository does not get too large.  In this case, we keep 10 versions of the files.

Offsite Backup

rsync -azq --delete $RECENT/ $SECONDARY_BACKUP:$SECOND_RECENT/
rsync -azq --delete --bwlimit=10 $RECENT/ user@$OFFSITE_BACKUP:$OFFSITE_RECENT/

The final step is to make sure that we have multiple copies of our big backup.  The rdiff-backup repository is copied to a second onsite server, for easy restore in the event of hardware failure on the backup machine, and an offsite server.

Conclusion

By gathering all of our information into one place, then taking an incremental backup and sending that to remote servers, we have easily created a backup script that stores incremental versions of all of our important data in one place.

The latest stable version of Chrome browser at the time this article was written was 3.0.195.33. We will compare it to the current version of IE, which is 8.0.6001.18865. We started with Google’s own benchmark that is hosted at their site and is called V8 Benchmark Suite. The result differed from one machine to another, but the score of the Google browser was always about 38 times higher than the one of IE – very close to the numbers that Google is claiming. Of course, as much we believe in the accuracy of the tests, we wanted to use some independent benchmarking sites, too. After some research, we found some of the major ones are SunSpider JavaScript Benchmark and JSBenchmark. Their tests are a complex mixture of different tasks, including drawing, code decompression, encrypting and operations with arrays, dates and handling exceptions. As we expected, the difference between the two browsers was much smaller compared to Google’s own test, but it was again pretty impressive. The results differed from platform to platform, but overall the “SunSpider” test showed that Chrome is ten times faster, while the “JSBenchmark” reported that it is “only” seven times faster.

JavaScript Speed Test

In order to conclude that Internet Explorer is much slower than Google Chrome in JavaScript performance though, we still needed more facts. As an alternative to the complex and heavy testing suites, we created a very short example with few array operations. We wanted it to be fairly simple and pretty close to real life examples. Basically, this piece of code assigned random numbers to an array of 20,000 items. Then we reversed the order of the elements. In the next two steps we sorted it in ascending and descending order. The script outputs the milliseconds needed for its completion. We did not expect a lot of difference in doing something so basic. Here is the code:

JavaScript Speed Test Source Code

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
		<title>MentorMate JavaScript simple speed test</title>
		<script type="text/javascript">
			<!--
			function testStart() {
				var mySampleNumbers = new Array();
				var now;
				var start_time_mill;
				var end_time_mill;

				//gets the start date
				now = new Date();
				start_time_mill = now.getTime();

				//assigning random numbers to an array of 20 thousand items
				for (i=0;i<=20000;i++) {
					mySampleNumbers[i] = Math.random();
				}

				//reverse the order of the elements in an array
				mySampleNumbers.reverse();

				//sort the numbers (ascending)
				mySampleNumbers.sort(sortAsc);

				//sort the numbers (descending)
				mySampleNumbers.sort(sortDesc);

				//gets the end date
				now = new Date();
				end_time_mill = now.getTime();
				displayResult(end_time_mill - start_time_mill);
			}
			function sortAsc(m, n) {
				if (m > n) {
					return 1;
				}
				else if (m < n) {
					return -1;
				}
				else {
					return 0;
				}
			}
			function sortDesc(m, n) {
				if (m > n) {
					return -1;
				}
				else if (m < n) {
					return 1;
				}
				else {
					return 0;
				}
			}
			function displayResult(result) {
				alert("Result: " + result + " ms.");
			}
			// -->
		</script>
	</head>
	<body>
		<form action="">
			<p>
				<input type="button" value="Start" onclick="testStart()" />
			</p>
		</form>
	</body>
</html>

JavaScript Speed Test Results

There were differences from PC to PC as with the other tests, but the tendency stayed the same. The time of completion of the tasks in Chrome was about six times faster than IE. This was the smallest advantage the Google browser had compared to the tests that were run so far, but it sounds pretty prestige for Chrome to have such a boost in performance for a really small task.

In conclusion, we can say that Google Chrome has set new standards for speed as far as JavaScript is concerned. Although the advantage the browser has might not be as big as Google claim in their advertising campaigns, it is something very impressive and gives the users additional speed for simple everyday tasks as we saw in the example above. We are looking forward to see the ninth version of Internet Explorer soon, as in the early discussions the developers show ambition to reach the level of performance that Chrome already has. The acceleration of the JavaScript engines for all the browsers can only bring benefits to users and contribute to the creation of richer, more complex Internet applications.

• Hardware Failure, hard drives and other computer components fail regularly.
• Destruction of your site, fire or flooding.
• Accidental data loss, somebody unintentionally deletes one or more files.
• Software error, a software error could corrupt one or more of your files.

The difference between backup and redundancy

Most modern servers use RAID for redundancy.  This usually means that when you put something on the server, it gets written to two hard drives.  In the event that one hard drive goes down, you can replace it with no disruption to service.  This is an example of redundancy, and will help protect you against a very specific type of failure.  Some systems, like Google, will have multiple servers, allowing an entire server to go down without disruption to service.  It is important to have redundancy so that services to your company or to your clients are not disrupted, but it is not a replacement for backups.

Backups involve creating a second copy of your data.  Often times, multiple backups from different points in time will be maintained.  Restoring a backup takes time, and can result in disruption of service.  Backups help protect from user errors like accidental deletion and software errors that could corrupt data.  Having more than one copy of your files is very important for recovery from these errors.

What to backup

Email

If you host your own email, this is probably one of the most important things to backup.  If your email is hosted, see if you can backup your email, or see what your provider is doing to backup your email.

Document Management System

If you have a document management system or some other type of server that you store your files on, this is also a very important group of files to back up.

Desktops

If you have a lot of people, it can be very difficult to back up each desktop that you have.  A better solution is to make sure that all important files are stored on a server so you only need to back up a single server.  You should have a plan in place for what happens when a desktop goes down in order to reduce the impact on the productivity of the person who was using it.

Web Application Data

If your website collects data, or if you have any web applications that collect data, make sure that you back up the databases and other information stores.  SQL databases have backup programs that can be run.  You can set your operating system to run the backup regularly.  Also, if there is space where users can upload files, make sure to back up the files as well.

Custom Software

Custom software can be expensive to develop.  Make sure that you have a backup of any custom software that you have had built so that you can easily deploy it again if you need to.

Information in “The Cloud”

You may have documents or other information that is hosted on the internet.  If at all possible, you should take your own backups.  If you cannot take your own backups, how reliable is the service provider?  Think carefully about each of the services that you are subscribed to.  These could be Salesforce, Google Docs, email providers, Online stores (do you need your amazon.

Any other products that are generated

Each company produces different types of products.  Make sure that you go through the list of products that you have and ask yourself what would happen and how you would recover if you lost some data.  Make sure to think about digital components that are created to support physical products.

How

The number of backup programs that exist and techniques that are available are too numerous to count, so I will only cover a few concepts.

Important Concepts

Local and Offsite

It is a good idea to have a copy of your data locally and a copy of your data offsite.  The offsite backup is important in the event that your office is destroyed.  The location of the offsite backup could also be destroyed, so having an onsite backup will help protect you against that.  Another reason to keep an onsite backup is that if you need to restore an item, the backup is immediately available.

Automated Vs. Manual

Human memory is flawed, people take vacations and sick time, and staffing changes.  If your backup is automated, these events will have a smaller impact on your backup system.  An automated backup system can grab all of your data every night, and push it to an offsite location over the internet or a T1 pipe.  There may be parts of the backup that must be done manually, like changing media, or a few parts of the backup that are difficult to automate, but the more automatic it is, the better.  Still, make sure to check it periodically to make sure that it is still taking the backup and you are not getting errors.

Backup Media

Depending on how much you need to back up, you may want different types of media.  Tape drives are popular for large quantities of data.  CD or DVD media is a cheap way to back up small amounts of data, but make sure that you get high quality media, and check it from time to time to make sure that it is not deteriorating.  Hard drives are very convenient since they are rewritable.  Keep in mind that hard drives fail.  Using RAID for redundancy is a good way to help protect yourself from hard drive failure.

Ensure it works

After you have created your backup system and are making sure that it runs, make sure to test it by restoring data from the backup.  If you cannot restore a backup, it is not really a backup.

Security

If you are backing up confidential or sensitive data, make sure that the backup is at least as secure as the data that you are backing up.  A compromised backup is just as bad as the data being compromised since you have a copy of everything.  If the backup is encrypted, make sure that enough people have the key, and that the key is backed up somewhere.  If you encrypt the data and lose the key, you lose the backup.

Organization

Make sure that you always know what backup is current, what backup is old, and when you’ve taken the backups.  Your software may take care of this for you.  If it does not, putting dates on the folders, or a file with the information within the folder can be good ways to keep organized.  You also need to know where it is going and how to restore it.  Make sure that this is documented in case the person who built the backup is not available when a restore is needed.  You can also streamline the backup process by making sure that the files that need backing up are organized in as few places as possible.

Human Redundancy

Make sure that multiple people know where the backup is and how to restore it.  It is a good idea to have some documentation on the backup system, how the backups are created, any manual steps and when they need to be performed, and how to restore each component of the backup, either fully or partially.

Software

Virtual Machines

Virtual machines typically have a way to make a copy of the entire machine.  This can be taken advantage of to backup an entire system from all of the software that is in stalled to the data files that are on the machine.

rsync and rdiff-backup

rsync is used to copy files from one place to another, optionally over the network or internet.  All data transfers are encrypted.  When files are copied, only the files that have changed are copied, reducing the amount of bandwidth required for most cases.  rdiff-backup is similar, but it will keep incremental backups making it so that you can revert to a previous version of your backup.  To save space, files are only saved twice if they have changed.

dd

dd can make a copy of a hard drive.  This can be used to make an image of your operating system hard drive from time to time.  If you ever need to restore, you can reload the image to the drive, and all of your software will be ready to go.

Others

There is enough backup software out there that I could not hope to cover all of it.  Look at your needs and find the group of software that fills your needs.

Conclusion

Make sure you think carefully about why you are backing up your data.  Your reasons for backing up your data will have an impact on how you back it up.  What would happen to you or your business if you lost some or all of your files?  Who is going to be able to restore your files?