Booting VirtualBox from USB

If you have a Live OS installed onto a USB Key, you may want to test it by booting it from VirtualBox. It turns out that this is a real pain in the arse. There is apparently no way to do this natively, but there is a work around. What you have to do is use a command line (yay!) utility to make a fake hard disk image which points to the USB Key. So VirtualBox THINKS it is booting from a hard disk image but is ACTUALLY booting from the USB Key.

Anyway. The appropriate command in Ubuntu looks like this:

sudo VBoxManage internalcommands createrawvmdk -filename ~/test_usb.vmdk -rawdisk /dev/sdc

That works if you want the new fake disk image to be called [test_usb] and to be stored in [~/] home. Also it links to a USB disk mounted at [/dev/sdc]. The [.vmdk] bit is just the file type for VirtualBox disks.

If you want to do this on Windows, you run a very similar command which looks like this:

VBoxManage internalcommands createrawvmdk -filename "%USERPROFILE%\test_usb.vmdk" -rawdisk \\.\PhysicalDrive1

Apparently [%USERPROFILE%] is the Windows equivalent of [~]. The tricky bit is going to be [PhysicalDrive1]. You are going to have to work out what the number that goes on the end of that is, just as you would have to find out where a USB Key had been mounted in Linux. To do this in Windows run the command line [diskpart] program and then the command [list disk] and it should pop out with a list of the disks attached to the machine with their numbers. It looks like this on my virtual XP:

Microsoft DiskPart version 5.1.3565

Copyright (C) 1999-2003 Microsoft Corporation.
On computer: SB-VIRTUAL

DISKPART> list disk

  Disk ###  Status      Size     Free     Dyn  Gpt
  --------  ----------  -------  -------  ---  ---
  Disk 0    Online        10 GB      0 B

DISKPART>

So I would want to use PhysicalDrive0 in that case. That would be monumentally stupid though, because I would mount the disk I was using to run my OS from in another machine AT THE SAME TIME. I do not want to imagine what the crash would look like.

You will also need to navigate to the install directory for VirtualBox to run the command (the VBoxManage command, not the diskpart command). Linux sticks the program file in the path, but Windows doesn't.

If it didn't go without saying all you do next is just add the .vmdk file as the principal hard disk to your virtual machine.

Cleaning Up a Virtualbox XP Install

(Or frankly just cleaning up an XP install. But then XP natively will have a big chunky disc to spread all of its insanity into, whereas a Virtual XP install will probably be quite restricted.)

I have started using VirtualBox to run XP on my Ubuntu workstation just in case I have any compatibility problems with native Ubuntu programs. Like, for instance, the Exchange Server connection in Evolution. This part seems to have been programmed with all the precision and attention to detail of a Jackson Pollock painting. Want to move a large email to a public folder? "The connection to the Exchange Server has been lost". OK maybe that was a tad ambitious. How about just opening a sodding email? "No chance mate, you're getting the raw HTML of the Outlook Web Access login screen instead".

So an XP guest OS with Outlook installed seems like a good idea to cover the rare occasions when Evolution acts like a total dick. Like Tuesdays. Or Thursdays.

I am no expert when it comes to the arcana of Microsoft licensing, but my case has an official sticker on it with a product key, so I think I am safe running XP on it using that key, albeit as a Virtual Machine. Same goes for the Office suite that came with the machine as well. It isn't being used anywhere else since I wiped the disk and stuck Ubuntu on a couple of years ago.

Or Wednesdays, come to think of it.

Well, happily the installation and activation of both XP and Office went swimmingly. My installation CD was only SP1. I installed SP3 over the top of it, but now I have a large collection of crap in the Windows folder to be used in the unlikely event that I ever want to NOT have SP3 installed. Space in general is a bit of an issue. I have a 40Gb SSD drive in the machine, which is more than fine for Ubuntu, but is slightly pushing it for Ubuntu + XP, even as a VM. I gave the virtual machine a 10Gb HDD, which I think is about the bare minimum for an XP install. After installation of Office and a bit of Proprietary Management Software, it is down to less than 3Gb free. Which is not ideal.

So lets fire up Tree Size Free and have a look at where all the space is going.

Biggest Culprit is the Windows folder with 3.5Gb
Next up is the root folder of C: with 1.5Gb
Next is the User folder with about 1Gb in it.
Lastly there is about half a Gb in Program Files.

Yes that adds up to about 6.5Gb. Where has the remaining half gigabyte gone? Two words. System Restore. Great idea in general, no use on a space limited task dedicated machine if you do regular back ups. To remove this, Start -> Right Click on -> My Computer -> Left Click on -> Properties -> System Restore Tab -> Turn Off System Restore Check Box. It is not that hard is it?

OK, first proper clean up - I had left the installation files for the proprietary software lying on the desktop. Delete = 300Mb back. The rest of the User folder was an archive of emails amounting to a couple of hundred Mb which I uploaded to the server, and the local copy of my exchange emails. Can't do anything about them, or indeed the Program Files. Need those to actual run the applications. Still, we've cleared half a Gb. Lets look at the root of C: next.

It has one big file in it: pagefile.sys. It is a whole 1.5Gb itself. This is the famous windows pagefile which the system uses if windows runs out of RAM. Lets run a basic test by opening all the applications installed. Word, Excel, Outlook and the Proprietary software. Done that, and the pagefile usage barely his 250Mb. So, let's trim back the 1.5Gb pagefile to a more sensible 512Mb to give us some headroom.

To do this we go to Start -> Right Click on -> My Computer -> Left Click on -> Properties -> Advanced Tab -> Performance Settings Button -> Advanced (Again) -> Virtual Memory Change Button. I chose a custom size, starting at 512Mb and limited to 1024Mb. Much more sensible for a machine with 1Gb of ram to start with.

Lastly we have the Windows folder. Most of this we DO NOT WANT TO FUCK WITH. However, there are a number of temporary files that we can get rid of. The reason is that XP keeps the files that were replaced by each update or Service Pack install just in case you want to undo the update. As long as it works, I don't. And, as with the System Restore rational, I will be keeping a backup of the install so I don't need the install to back itself up. This is dead easy with virtual disks - you just make a copy of the disk image somewhere. For instance with this command:

sudo apt-get install p7zip-full
7za a XP_10Gb.vdi.lzma XP_10Gb.vdi

So, what can we get rid of in Windows? There are four main folders that I looked at:

C:\Windows\$NtUninstall[blahblah] <- lots of these
C:\Windows\Driver Cache\I386
C:\Windows\ServicePackFiles\I386
C:\Windows\SoftwareDistribution\Download

The first type are just lots of hidden (Explorer Window -> Tools Menu -> Folder Options -> View Tab -> Show Hidden Files and Folders Radio Button & Uncheck Hide Protected Operating System Files) folders storing uninstall data for most windows updates. It is Completely messy of XP to just dump them in the Windows folder.

The next three folders all store downloaded files that have been installed. You MAY need these again if you are installing any new software. What I did was copy all of the foregoing directories into a backup folder, and then stuck that on my NAS [Network Attached Storage]. Bear in mind that you need to install any outstanding updates before moving these files otherwise the update will break and need to be downloaded again.

Also in Windows is a folder called Installer. Do not mess with this. But if you do want to mess with this, google for msicuu2.exe and use that rather than doing it manually.

Anyway, with that clear out, the Windows folder was reduced from 3.5Gb to 1.7Gb, which is ideal. I now have just under 7Gb free, representing a space recovery of 4Gb on a 10Gb drive. Good Stuff.

There is also a temp folder has appeared in the root of C: called MSOCache. This is apparently a copy of the Office install files - presumably in case I want to install any optional items, or parts I chose not to install to begin with. It is only 100Mb or so, and it also appears in the Disk Cleanup program (Start -> My Computer -> Right Click -> C: -> Properties -> Disk Cleanup Button) so I can clean this up easily at a later date if I want.

Why oh fucking why are the stupid Uninstall, Driver Cache, Service Pack, and Downloaded files not also in the Disk Cleanup program? As Advanced options perhaps? As it stands I will have to go through this lengthy process every so often to clean out these unnecessary files. Pain.

get_iplayer 2011

[This Update has now been Updated]

Time, I think, for an update on the get_iplayer situation. For the last few months I have found that my previous instructions work just fine for downloading SD content. They were borked though, for HD content. Which has been a crying shame, because that's the whole sodding point of get_iplayer as far as I am concerned. Want to watch BBC HD content but have an ISP comprised of a larger collection of cocks than the mid-west's bi-annual prettiest rooster award? Tough, your bandwidth will never be enough to watch streaming HD. Unless, of course, you set your alarm for 3 a.m.

So, time for an update, especially when I noticed that my previous and well thought out instructions were being described as out of date. The shame.

I find it is best to start with a vanilla version of Ubuntu when working out how to do stuff like this, otherwise you never know which package you installed six months ago and forgot about is actually essential to the whole enterprise. You then write down a careful list of instructions for someone else to follow who has NOT installed the critical package, and it doesn't sodding work, and you end up looking like a half-wit.

So, best to use the latest Ubuntu LiveCD in a virtual machine for testing purposes. [You can then make sure that everything actually works. If it doesn't the problem is somewhere at your end. If it does work you can go on to try to install the software on your distribution of choice, and enjoy some dependency hell, but knowing all the time that the theory is sound. None of this is necessary, if you want to you can just skip over all the virtualisation stuff and get straight to the software, go ahead.]

A perfectly sensible virtual machine is VirtualBox. You will find the installer package here. Grab the one for your architecture (intel/amd basically). You can even install the Windows version and then run Ubuntu inside the VirtualBox, and follow along with these instructions if you so desire.

If you are running it from Ubuntu, you also need to install the following package to get a screen resolution more than 800x600. This is optional if tiny resolutions do not bother you, or if you remember the time when 640x480 was standard and 800x600 was High-End.

sudo apt-get install virtualbox-ose-guest-x11

Now reboot your host machine (the physical machine you are working on).

When it comes back on, you may need to run the following command to get VirtualBox to work - I need to using Lucid:

sudo modprobe -r kvm_intel

Welcome back Windows people. Next step is to grab the Ubuntu CD from here. 32bit is fine. [I have also tested this with the latest (10th February 2011) daily build of Natty, and it still works fine - in fact the VirtualBox extensions aren't needed to get a decent resolution.] Once that has downloaded, fire up VirtualBox and create a new machine. I gave mine 1024Mb RAM. For our current purposes we will not actually be installing the OS, just testing get_iplayer and friends, so do not bother with a Hard Disk. In my version, that is all that is needed to create the machine, which will of course NOT WORK, because there is no boot media.

So, next click on the machine, and hit the settings button. I then went to the Storage option on the left hand side of the window which pops up. I clicked on the shiny CD with 'empty' next to it, and then in the drop down menu under Attributes on the right hand side I changed it to Primary Master. I then clicked the small CD button next to the drop down menu and choose to use a virtual CD/DVD file. I then just navigated to where ever I downloaded the Ubuntu LiveCD.

Next, to optionally get a better screen resolution, I went back to the middle panel, and clicked on the IDE Controller line and clicked the CD image with the green plus on it to add a new CD drive. I opted to choose a disk, and navigated to

/usr/share/virtualbox/VBoxGuestAdditions.iso

I am not sure where that file will be located on a Windows install, but you can presumably search for it. [I tested this on Windows 7 and it worked fine. The .iso I refer to in stored in the Program Files folder where VirtualBox is installed.] It is not essential for what comes next though - it just make using VirtualBox a nicer experience.

I then fiddled about with some other settings. I activated USB 2.0, and I wanted to us 2D acceleration but that only works on Windows machines. Having done all that I just clicked OK at the bottom right of the window. I then just made sure my Machine was still selected at the left hand side of the window and hit the start button. I chose to try Ubuntu rather than install.

OK. We should now be at the Ubuntu desktop. It is a bit cramped at 800x600. If you are fine with that, just scroll down. If not, mount the Additions CD (just click on it in the places menu) pop open a terminal window and run:

sudo /media/VBOXADDITIONS_4.0.2_69518/VBoxLinuxAdditions.run

It took a few seconds to install and then invited me to restart the machine. That would be DAFT on a live cd, so instead just shut down X:

sudo /etc/init.d/gdm stop

And when the dust has settled, you need to switch to a text terminal screen. You would ordinarily do this with Control + Alt + F1, but if you are using an Ubuntu host, that will just change you to the terminal screen on your host. Instead use the RIGHT Control Key, NO Alt, and hit F1. Now restart X:

startx

Bingo, better screen resolution.

Now, onto get_iplayer. Welcome to people who could not be bother with VirtualBox.

First open a terminal, make sure you have the latest package information and install some software we will be needing:

sudo apt-get update
sudo apt-get install git-core build-essential subversion libssl-dev ffmpeg

If you are playing along at home, you should now be able to paste commands straight into the Virtual machine. [So what is this extra software? 'git-core' and 'subversion' do essentially the same thing. These are programs that let you download software, including source code, from internet sites. The software maintainers are able to keep track of different versions of their software and you can interact with it. All we will be doing is downloading the latest version. 'build-essential' is a collection of tools that let us compile software from source code into executable files and libraries. 'libssl-dev' contains software that needs to be incorporated in the programs we are downloading when they compile. SSL is a secure communication system for the internet, and it is already installed on your machine. However, we need the 'dev'elopment files for it, so that it can get built into the new software. All you have at the moment is the end product. Finally 'ffmpeg' is a powerful command line video converter which is going to be used to convert the downloaded video from flash format to the easier to use .mp4 format.]

Once that software has downloaded, make sure your terminal prompt shows you are in a sensible place. Your home directory is just fine:

ubuntu@ubuntu:~$

If not, change to your home directory by running:

cd ~

Now grab the latest version of get_iplayer by running this command (it will make a directory called get_iplayer in directory you run the command from, which is why we made sure you were in home):

git clone git://git.infradead.org/get_iplayer.git

Move into the directory that was just made and download the latest rtmpdump (which will be downloaded into its own folder much like get_iplayer):

cd get_iplayer
svn co svn://svn.mplayerhq.hu/rtmpdump/trunk rtmpdump

Then move in to the rtmpdump folder and build the software:

cd rtmpdump
make SYS=posix

The [SYS=posix] tells it to build the Linux version, as opposed to Windows or whatever else is possible. I then test ran the rtmpdump program that had been created and got this:

./rtmpdump

./rtmpdump: error while loading shared libraries: librtmp.so.0: cannot open shared object file: No such file or directory

This got no better when I [make install]ed the rtmpdump software. I would prefer to keep everything in [~] home if I can. The error was complaining that it could not find a library. The library was right there in a subfolder, so I just made a link to the Ubuntu library folder:

sudo ln -sv ~/get_iplayer/rtmpdump/librtmp/librtmp.so.0 /lib

I then tested again, and got:

RTMPDump v2.3(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPLERROR: You must specify a hostname (--host) or url (-r "rtmp://host[:port]/playpath") containing a hostname

Which was much better. Now to set up get_iplayer, I dropped back down to the get_iplayer directory, and ran the program once to update its plugins from the net:

cd ..
./get_iplayer

And then I setup the usual preferences I use:

./get_iplayer --prefs-add --modes=flashhd,flashvhigh,flashhigh,flashnormal,iphone
./get_iplayer --prefs-add --flvstreamer "/home/ubuntu/get_iplayer/rtmpdump/rtmpdump"

It was quite happy being told to use rtmpdump under the flvstreamer preference. Job is, as they say, a good'un.