A Backup Scheme Described

I wrote a bit about my backup routine in my rant about TrueImage. I wanted to set out in more detail the scheme I have set up because it also incorporates some mklinking referred to in my previous post.

My previous example of mklinking dealt directly with steam applications. There is another type of data that you do not want to have clogging up and SSD. That is any large file that you only require to access sequentially. Or in non-computer science lingo, music, videos and photos. These can be large files (a modern SLR camera will happily spit out 10Mb image files), but you do not need the super fast access time of an SSD to use them properly.

Think about it, you could happily watch a movie from a DVD player. That has a PATHETIC data rate and seek time compared to an SSD, but it works fine for movie file. Movie files are also LARGE. My Sanyo camcorder mentioned elsewhere spits out files up to 20mins/Gb. They take up a lot of space. If a 64Gb SSD (with usable space of say 60Gb) was used exclusively for these video files you would fit no more than 20 hours of video on it. For £60. That's £3/hour for storage of video. That makes no sense. Not when at the time of writing you can pick up a 2Tb classic drive for the same £60. OK, that 2Tb drive is only actually about 1.8 real Terabytes. Still, that's 620 hours of video for £60. That's £0.09/hour.

So for large data files we want the 9p slow and steady storage, not the £3 super fast storage. What I put on the classic disk is basically the contents of My Videos, My Music, My Pictures, and My Documents. To do this I create a directory on the classic disk with appropriately named folders in it. If your OS is on the SSD, then your classic disk is probably mounted at d:. You can do this with the mouse and gui, but the dos prompt commands (windows key + r, type cmd, hit enter) look like this:

cd d:\
mkdir Data
cd Data
mkdir Videos
mkdir Music
mkdir Pictures
mkdir Documents

For the benefit of the hard of thinking, those commands [c]hange to the [\] root [d]irectory of the [:] disk assigned the letter [d]. They next [m]a[k]e a new [dir]ectory, [c]hange into that new [d]irectory, and [m]ake some other [dir]ectories.

You then fill up those folders with the data of choice. Copy this from your non-SSD old system drive, or from your up to date back up. Then, if using windows 7, you need to delete the Videos, Music, Pictures, Documents from your user folder. There should be nothing in those folders on a fresh install - but check anyway, and if there is then move it to the folders on the classic disk we just made.

cd C:\Users\[your username here]
xcopy Videos D:\Data\Videos /E /H /K
rmdir Videos /S
xcopy Music D:\Data\Music /E /H /K
rmdir Music /S
xcopy Pictures D:\Data\Pictures /E /H /K
rmdir Pictures /S
xcopy Documents D:\Data\Documents /E /H /K
rmdir Documents /S

[xcopy] is a special dos command that is used to copy large collections of files at once. It is basically more flexible and powerful than the standard dos [copy] command. For our purposes it is exactly the same as Control+c'ing a folder and Control+v'ing that folder somewhere else. The [/E] flag tells it to copy all subfolders even if they are empty, the [/H] flag makes sure it copies hidden or system files, and the [/K] flag copies the attributes of the files being copied (read only etc) as well. The other command [r]e[m]oves the specified [dir]ectory and all its [/S]ubdirectories.

Finally you want to set up the mklinks:

mklink /J "C:\Users\[your username here]\Documents" "D:\Data\Documents"
mklink /J "C:\Users\[your username here]\Videos" "D:\Data\Video"
mklink /J "C:\Users\[your username here]\Pictures" "D:\Data\Pictures"
mklink /J "C:\Users\[your username here]\Music" "D:\Data\Music"

You should now be up and running. If you go to the start button and click on Music, the folder that should open is the one on the classic disk. The benefit to using mklink is that it appears to the OS and all software running on it (such as iTunes) that all this data is in your user folder on the SSD, so you do not need to specially configure any other software. It should just work.

Now you have set that up, backups are a doddle. Buy a large external 3.5" disk that comes with it's own power supply. Yes, it is bulky. Yes it is a pain to have to carry around a data and power cable. BUT it is much cheaper per Mb than a 2.5" USB powered external disk. And all it is going to do is sit in the vicinity of your main machine ready to be used as a backup device at a moments notice. You then just need Allway Sync, or a similar folder syncing application, to sync the d:\data folder onto your external drive. Make sure you sync that ONE folder every so often (i.e. when you have bought new iTunes music, written a new document, or uploaded a camera card to your desktop) and you have a backup copy of ALL of your data.

I find the advantages of this system are:

Simplicity. The backup is just a copy of all your data files. You can plug the external disk into another machine running virtually any desktop OS and your have instant access to your files.
Cost. Folder Synchronisation software is inexpensive (Allway Sync is nagware).
Speed. All I do is open Allway Sync, choose the "Data" profile I have set up, run an analysis just to check, and then run a sync. It does not take a long time unless you have added a crap load of stuff.
Flexibility. Running dual OS's? Simply link Ubuntu's similarly named data folders to the ones on the windows disk. Hey presto, same data, two different OS's.

There are disadvantages to this system:
Versioning. The backup is a simple mirror of your files. It does not keep multiple copies of your files, so you cannot undo a change, or deletion, once you have synced. You could implement this, but you would need more expensive dedicated backup software. As far as I am concerned, I do not want to lose family media. I am not overly exercised about restoring last Tuesday's copy of a 1Gb video file rather than last Wednesdays. Generally the data in these folders is NOT going to change, it is going to be added to or deleted, but not altered.
Onsite. The backup disk is in the same room as the desktop with the original data, which is not going to help if the equipment is stolen, or the building burns down. For that protection, you need to pay for a online backup solution - or get an ISP who provides one.

I prefer the advantages to the disadvantages. I do see a need for a versioning backup system, which is why I bought TrueImage Home. But I use that pretty exclusively for the OS disk. With that you absolutely want to be able to go back to last Tuesday, or "whenever I didn't have this bastard virus".

You may also want to move some other folders from your user folder on the SSD to the classic disk. For me a priority was the Downloads folder. You delete the original folder as normal and then:

mkdir d:\Data\Downloads
mklink /J "C:\Users\[your username here]\Downloads" "D:\Data\Downloads"

You probably do not want to backup the downloads folder - I use mine as a scratch area for downloading loads of rubbish. If I want to keep stuff I download, I shift it to an appropriate data folder. So you will now want to edit the profile you have set up in Allway Sync to EXCLUDE the Downloads folder from the backup. This saves space on your external disk.

You can also get cute with the exclusion filters INSIDE the folders you are syncing. So if you have a Temp scratch area in Music or Videos, you can exclude that from the backup as well.

mklink

One of the least intuitive things I found about Linux, especially when moving over from a history of Microsoft OS's, was the idea of 'mounting' a disk to a point in the file system.

It is something that does take a bit of getting used to. Once you do, though, a number of other opportunities present themselves. One interesting thing is the ability to link files and other folders to other parts of the file system. This is different to the Window's idea of a shortcut. This is something that actually happens at file system level, with the result that no matter what programs you use, they will always be able to access the material in the same way.

Well, finally Microsoft have cottoned on to the fact that this might be a good idea. Hence in Vista, and now in Windows 7, there is the command

mklink

This [m]a[k]es a [link] between a folder in one place and a placeholder in another place on your filesystem.

Why would you want to bother with this? I will give you one, frankly excellent, example. Steam on SSD.

See, if you jump on the hype bandwagon and get yourself a solid state disk, you will rapidly discover 2 things.

1. It wasn't just hype, they really are excellent bits of kit.
2. There just is not enough space on the drive for all the stuff you want to put on it.

The second issue arises because SSD's are many times more expensive per gigabyte than traditional spinny magnetic disks. Realistically, in early 2011, 64Gb disk drives are reasonably priced, and 120Gb drives are the economic limit for most users.

If you are using 64Gb, this means that once you have stuck the bloated behemoth that is Windows on the disk, you'll be left with, realistically, 40Gb. Games these days can take up north of 15Gb (Dragon Age Origins I am looking at you). Clearly, space is going to be an issue. The main problem here is that Steam will only install games to one location. It will not let you say this game this disk, this game the other disk.

Game files are stored in the folder [Steam]/steamapps/common/[name of game folder]. Once you have installed the games you can move the [name of game folder] to another, slower, disk and run the command:

mklink /j "[Steam Folder]/steamapps/common/[name of game folder]" "[Other disk and folder]/[name of game folder]"

This will establish a hard link to the game folder on another disk. As far as Steam is concerned the game is still where it started, and it will run fine.

What this needs is some enterprising type to make an application that keeps track of your steam folders and moves them on and off the SSD for you managing the links all the time. Also, a profile system per game would be excellent, so you could just move off the movie files for instance, leaving the rest of the game on the SSD. Said enterprising type could then flog the software to Valve who should just incorporate the facility in Steam, thereby making my life just a little bit easier.

This person, for instance.

Downloading Flash Videos

OK, so is your ISP are bandwidth throttling you, getiplayer sorts out patchy video from the Beeb, but what about youtube et al? There is actually a simple way to store flash video using ubuntu without the need to purchase downloadsaverproextreme. You just grab the downloaded file from Ubuntu's temp space.

First of all you want to be sure that firefox knows it has enough cache space to download whatever it is we are viewing. Enter this into the address bar of firefox:

about:config

Then search for the following settings and change them as suggested:

browser.cache.disk.capacity = 500000
media.cache_size = 500000

Incidentally, none of that may be necessary, but the next step certainly is. Change to the temporary folder:

cd /tmp

And view the contents of the folder:

ls -alhSr

That lists [a]ll files with a[l]l information, with file sizes in [h]uman readable format, [r]everse ordered by [S]ize so that the largest is last. I prefer it in the easier to remember sequence:

ls -Sharl

Now, go to firefox, open the webpage that the video is on, and start the video playing. Pause it right away so that the stutter of slowly downloading video doesn't drive you mental. Run the [ls] command above from time to time and you can watch the video file increase in size. It is probably going to end up as the largest file in the /tmp folder - and therefore at the bottom of the list.

Copy this file to ~/Videos, while the firefox tab is still open. This is important, if you navigate away from the page the temporary file is deleted immediately. If the video is from youtube it should just be a x264 video and aac audio in a flv wrapper. You can change wrappers to .mp4 to make the video file compatible with your video player of choice with ffmpeg, while keeping the actual video and audio data unchanged with this command:

ffmpeg -i [your filename here].flv -vcodec copy -acodec copy [output name].mp4

Booting VirtualBox from USB

If you have a Live OS installed onto a USB Key, you may want to test it by booting it from VirtualBox. It turns out that this is a real pain in the arse. There is apparently no way to do this natively, but there is a work around. What you have to do is use a command line (yay!) utility to make a fake hard disk image which points to the USB Key. So VirtualBox THINKS it is booting from a hard disk image but is ACTUALLY booting from the USB Key.

Anyway. The appropriate command in Ubuntu looks like this:

sudo VBoxManage internalcommands createrawvmdk -filename ~/test_usb.vmdk -rawdisk /dev/sdc

That works if you want the new fake disk image to be called [test_usb] and to be stored in [~/] home. Also it links to a USB disk mounted at [/dev/sdc]. The [.vmdk] bit is just the file type for VirtualBox disks.

If you want to do this on Windows, you run a very similar command which looks like this:

VBoxManage internalcommands createrawvmdk -filename "%USERPROFILE%\test_usb.vmdk" -rawdisk \\.\PhysicalDrive1

Apparently [%USERPROFILE%] is the Windows equivalent of [~]. The tricky bit is going to be [PhysicalDrive1]. You are going to have to work out what the number that goes on the end of that is, just as you would have to find out where a USB Key had been mounted in Linux. To do this in Windows run the command line [diskpart] program and then the command [list disk] and it should pop out with a list of the disks attached to the machine with their numbers. It looks like this on my virtual XP:

Microsoft DiskPart version 5.1.3565

Copyright (C) 1999-2003 Microsoft Corporation.
On computer: SB-VIRTUAL

DISKPART> list disk

  Disk ###  Status      Size     Free     Dyn  Gpt
  --------  ----------  -------  -------  ---  ---
  Disk 0    Online        10 GB      0 B

DISKPART>

So I would want to use PhysicalDrive0 in that case. That would be monumentally stupid though, because I would mount the disk I was using to run my OS from in another machine AT THE SAME TIME. I do not want to imagine what the crash would look like.

You will also need to navigate to the install directory for VirtualBox to run the command (the VBoxManage command, not the diskpart command). Linux sticks the program file in the path, but Windows doesn't.

If it didn't go without saying all you do next is just add the .vmdk file as the principal hard disk to your virtual machine.