Listening to BBC Radio Shows on an MP3 player

In the name of bastard fuck, this was far fucking harder than it had any right to be.

You have Windows Media Centre installed on your PC. Either XP, or Vista or 7. Whatever. Fine. You record DVB-T1 broadcast TV programs and watch them quite happily. You record the odd radio program (also broadcast over DVB-T1), and you can play it back on the TV. But who wants to sit around the TV listening to radio programs, what do you think this is, the 1930's?

You have recorded an audio file. You want to listen to the audio on another device. How fucking difficult do you think this is going to be? Oh, boy you are in for a fucking treat.

So, how do we do this? I should interject here and say that there is a fairly straightforwards, albeit time-consuming, way to deal with this. You fire up audacity, change the input source to be the sound card audio out, then hit record and play whatever it is you want to record. The problem with this is that you are recording a compressed track so you are going to lose quality. In my view the BBC are the experts when it comes to digital audio compression and trasmission, and it is going to spoil whatever they broadcast if I recompress it. Its like sitting in the back row of the cinema with your digital video camera. You are not exactly going to walk out with a blue ray quality version of the film are you?

First thing. If using Vista or Windows 7, media centre spits out files with the extension .wtv. These are unreadable by anything but windows, so we need to convert them into a more common format. Helpfully if you right click on the file you will find an option to convert it to dvr-ms file.

What I would then ordinarily do is fire the file up in Video Redo. This is an excellent product for taking in a dvr-ms file, cutting out all the bits you do not want (ads and so forth) and then spitting out the original video and audio streams in an mpg file. You can then happily load that mpg file into avidemux, or your editor of choice and convert it to whatever format you desire.

However, Video Redo shits itself when it is given a dvr-ms file without any video. It just cannot handle it. The more recent versions of Video Redo may be different, but I am not paying for an upgrade for something this simple.

So we want to extract the audio in some other fashion. When something is difficult, then make it harder by doing it on the command line. If we want robust command line editing tools then we want Linux. I have tried ffmpeg under ubuntu with a command that looks a bit like this:

ffmpeg -i filename.dvr-ms -acodec copy -o filename.mp2

Doesn't fucking work. Apparently ffmpeg also cannot handle the earth shattering reality of a dvr-ms file without a fucking video stream.

How about:

mplayer -dumpaudio -dumpfile audio.mp2 filename.dvr-ms

No fucking joy. Cannot ever find an audio stream. It turns out that vlc is practically the only fucking thing on the planet that will spit the untouched audio out of a BBC Digital TV Broadcast Radio Recording. The command is this gem:

cvlc "[filename].dvr-ms" vlc://quit --sout '#transcode{vcodec=none}:duplicate{dst=std{access=file,mux=raw,dst="[filename].mp2"}}'

That's a fucking little beauty isn't it. Far more than you could ever realistically want to know about those options can be found here and here.

How to Post Maths in Blogspot: MathJax & QuickLatex

Edit your template and add the following before the closing "head" tag:

<script type=\"text/x-mathjax-config\">
MathJax.Hub.Config({
  MMLorHTML: {
    prefer: {Firefox: \"HTML\"}
  }
});
</script>
<script type=\"text/javascript\" src=\"http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full\"></script>

Then you just stick the TeX stuff in backslash square brackets for standalone formulae or backslash rounded brackets for in text symbols.

To get graphics you need to use Quicklatex. You need to drop down the choose options box on that page and then insert this text to activate the Tikz graphics package, with the tikz library that lets you calculate co-ordinates using algebra:

\usepackage{tikz}
\usetikzlibrary{calc}

You then get the URL of an image which you can stick in your blog.

Can I Exchange this for something that works?

OK, so they you are running Ubuntu on your work PC. You can connect to the office Exchange server using Evolution so you can get all your email. You are doing all of this using open source software. You are most likely feeling pretty smug.

Eventually you will want to leave the office, so you want to fire on the out of office notifications in Evolution. You read about how to do so. You find out you need to follow this chain of menus and buttons:

Edit -> Preferences -> Edit -> Account Editor -> Exchange Settings -> I am out of the office.

However, at the second "Edit" you get a "Evolution Error" window informing you that the lovely open source program you were oh so smug about "could not read out-of-office state"

Bastard. The problem is that if you google for help on this error, the search terms "state" and "evolution" tend to pollute the results with results about the particular subset of Americans who could politely be called reality-challenged. They could also impolitely be described as the type of fucking cretins who would happily watch their friends and family combusting in agonising pain if their sacred text of choice said that humans were impervious to fire.

Even searching in the Ubuntu Forums, I cannot find an answer to this. Or, come to it, why even though I have selected Global Contact List for Auto Completion in the settings, it doesn't fucking auto complete email addresses from the Global Contacts List.

A Backup Scheme Described

I wrote a bit about my backup routine in my rant about TrueImage. I wanted to set out in more detail the scheme I have set up because it also incorporates some mklinking referred to in my previous post.

My previous example of mklinking dealt directly with steam applications. There is another type of data that you do not want to have clogging up and SSD. That is any large file that you only require to access sequentially. Or in non-computer science lingo, music, videos and photos. These can be large files (a modern SLR camera will happily spit out 10Mb image files), but you do not need the super fast access time of an SSD to use them properly.

Think about it, you could happily watch a movie from a DVD player. That has a PATHETIC data rate and seek time compared to an SSD, but it works fine for movie file. Movie files are also LARGE. My Sanyo camcorder mentioned elsewhere spits out files up to 20mins/Gb. They take up a lot of space. If a 64Gb SSD (with usable space of say 60Gb) was used exclusively for these video files you would fit no more than 20 hours of video on it. For £60. That's £3/hour for storage of video. That makes no sense. Not when at the time of writing you can pick up a 2Tb classic drive for the same £60. OK, that 2Tb drive is only actually about 1.8 real Terabytes. Still, that's 620 hours of video for £60. That's £0.09/hour.

So for large data files we want the 9p slow and steady storage, not the £3 super fast storage. What I put on the classic disk is basically the contents of My Videos, My Music, My Pictures, and My Documents. To do this I create a directory on the classic disk with appropriately named folders in it. If your OS is on the SSD, then your classic disk is probably mounted at d:. You can do this with the mouse and gui, but the dos prompt commands (windows key + r, type cmd, hit enter) look like this:

cd d:\
mkdir Data
cd Data
mkdir Videos
mkdir Music
mkdir Pictures
mkdir Documents

For the benefit of the hard of thinking, those commands [c]hange to the [\] root [d]irectory of the [:] disk assigned the letter [d]. They next [m]a[k]e a new [dir]ectory, [c]hange into that new [d]irectory, and [m]ake some other [dir]ectories.

You then fill up those folders with the data of choice. Copy this from your non-SSD old system drive, or from your up to date back up. Then, if using windows 7, you need to delete the Videos, Music, Pictures, Documents from your user folder. There should be nothing in those folders on a fresh install - but check anyway, and if there is then move it to the folders on the classic disk we just made.

cd C:\Users\[your username here]
xcopy Videos D:\Data\Videos /E /H /K
rmdir Videos /S
xcopy Music D:\Data\Music /E /H /K
rmdir Music /S
xcopy Pictures D:\Data\Pictures /E /H /K
rmdir Pictures /S
xcopy Documents D:\Data\Documents /E /H /K
rmdir Documents /S

[xcopy] is a special dos command that is used to copy large collections of files at once. It is basically more flexible and powerful than the standard dos [copy] command. For our purposes it is exactly the same as Control+c'ing a folder and Control+v'ing that folder somewhere else. The [/E] flag tells it to copy all subfolders even if they are empty, the [/H] flag makes sure it copies hidden or system files, and the [/K] flag copies the attributes of the files being copied (read only etc) as well. The other command [r]e[m]oves the specified [dir]ectory and all its [/S]ubdirectories.

Finally you want to set up the mklinks:

mklink /J "C:\Users\[your username here]\Documents" "D:\Data\Documents"
mklink /J "C:\Users\[your username here]\Videos" "D:\Data\Video"
mklink /J "C:\Users\[your username here]\Pictures" "D:\Data\Pictures"
mklink /J "C:\Users\[your username here]\Music" "D:\Data\Music"

You should now be up and running. If you go to the start button and click on Music, the folder that should open is the one on the classic disk. The benefit to using mklink is that it appears to the OS and all software running on it (such as iTunes) that all this data is in your user folder on the SSD, so you do not need to specially configure any other software. It should just work.

Now you have set that up, backups are a doddle. Buy a large external 3.5" disk that comes with it's own power supply. Yes, it is bulky. Yes it is a pain to have to carry around a data and power cable. BUT it is much cheaper per Mb than a 2.5" USB powered external disk. And all it is going to do is sit in the vicinity of your main machine ready to be used as a backup device at a moments notice. You then just need Allway Sync, or a similar folder syncing application, to sync the d:\data folder onto your external drive. Make sure you sync that ONE folder every so often (i.e. when you have bought new iTunes music, written a new document, or uploaded a camera card to your desktop) and you have a backup copy of ALL of your data.

I find the advantages of this system are:

Simplicity. The backup is just a copy of all your data files. You can plug the external disk into another machine running virtually any desktop OS and your have instant access to your files.
Cost. Folder Synchronisation software is inexpensive (Allway Sync is nagware).
Speed. All I do is open Allway Sync, choose the "Data" profile I have set up, run an analysis just to check, and then run a sync. It does not take a long time unless you have added a crap load of stuff.
Flexibility. Running dual OS's? Simply link Ubuntu's similarly named data folders to the ones on the windows disk. Hey presto, same data, two different OS's.

There are disadvantages to this system:
Versioning. The backup is a simple mirror of your files. It does not keep multiple copies of your files, so you cannot undo a change, or deletion, once you have synced. You could implement this, but you would need more expensive dedicated backup software. As far as I am concerned, I do not want to lose family media. I am not overly exercised about restoring last Tuesday's copy of a 1Gb video file rather than last Wednesdays. Generally the data in these folders is NOT going to change, it is going to be added to or deleted, but not altered.
Onsite. The backup disk is in the same room as the desktop with the original data, which is not going to help if the equipment is stolen, or the building burns down. For that protection, you need to pay for a online backup solution - or get an ISP who provides one.

I prefer the advantages to the disadvantages. I do see a need for a versioning backup system, which is why I bought TrueImage Home. But I use that pretty exclusively for the OS disk. With that you absolutely want to be able to go back to last Tuesday, or "whenever I didn't have this bastard virus".

You may also want to move some other folders from your user folder on the SSD to the classic disk. For me a priority was the Downloads folder. You delete the original folder as normal and then:

mkdir d:\Data\Downloads
mklink /J "C:\Users\[your username here]\Downloads" "D:\Data\Downloads"

You probably do not want to backup the downloads folder - I use mine as a scratch area for downloading loads of rubbish. If I want to keep stuff I download, I shift it to an appropriate data folder. So you will now want to edit the profile you have set up in Allway Sync to EXCLUDE the Downloads folder from the backup. This saves space on your external disk.

You can also get cute with the exclusion filters INSIDE the folders you are syncing. So if you have a Temp scratch area in Music or Videos, you can exclude that from the backup as well.

mklink

One of the least intuitive things I found about Linux, especially when moving over from a history of Microsoft OS's, was the idea of 'mounting' a disk to a point in the file system.

It is something that does take a bit of getting used to. Once you do, though, a number of other opportunities present themselves. One interesting thing is the ability to link files and other folders to other parts of the file system. This is different to the Window's idea of a shortcut. This is something that actually happens at file system level, with the result that no matter what programs you use, they will always be able to access the material in the same way.

Well, finally Microsoft have cottoned on to the fact that this might be a good idea. Hence in Vista, and now in Windows 7, there is the command

mklink

This [m]a[k]es a [link] between a folder in one place and a placeholder in another place on your filesystem.

Why would you want to bother with this? I will give you one, frankly excellent, example. Steam on SSD.

See, if you jump on the hype bandwagon and get yourself a solid state disk, you will rapidly discover 2 things.

1. It wasn't just hype, they really are excellent bits of kit.
2. There just is not enough space on the drive for all the stuff you want to put on it.

The second issue arises because SSD's are many times more expensive per gigabyte than traditional spinny magnetic disks. Realistically, in early 2011, 64Gb disk drives are reasonably priced, and 120Gb drives are the economic limit for most users.

If you are using 64Gb, this means that once you have stuck the bloated behemoth that is Windows on the disk, you'll be left with, realistically, 40Gb. Games these days can take up north of 15Gb (Dragon Age Origins I am looking at you). Clearly, space is going to be an issue. The main problem here is that Steam will only install games to one location. It will not let you say this game this disk, this game the other disk.

Game files are stored in the folder [Steam]/steamapps/common/[name of game folder]. Once you have installed the games you can move the [name of game folder] to another, slower, disk and run the command:

mklink /j "[Steam Folder]/steamapps/common/[name of game folder]" "[Other disk and folder]/[name of game folder]"

This will establish a hard link to the game folder on another disk. As far as Steam is concerned the game is still where it started, and it will run fine.

What this needs is some enterprising type to make an application that keeps track of your steam folders and moves them on and off the SSD for you managing the links all the time. Also, a profile system per game would be excellent, so you could just move off the movie files for instance, leaving the rest of the game on the SSD. Said enterprising type could then flog the software to Valve who should just incorporate the facility in Steam, thereby making my life just a little bit easier.

This person, for instance.

Downloading Flash Videos

OK, so is your ISP are bandwidth throttling you, getiplayer sorts out patchy video from the Beeb, but what about youtube et al? There is actually a simple way to store flash video using ubuntu without the need to purchase downloadsaverproextreme. You just grab the downloaded file from Ubuntu's temp space.

First of all you want to be sure that firefox knows it has enough cache space to download whatever it is we are viewing. Enter this into the address bar of firefox:

about:config

Then search for the following settings and change them as suggested:

browser.cache.disk.capacity = 500000
media.cache_size = 500000

Incidentally, none of that may be necessary, but the next step certainly is. Change to the temporary folder:

cd /tmp

And view the contents of the folder:

ls -alhSr

That lists [a]ll files with a[l]l information, with file sizes in [h]uman readable format, [r]everse ordered by [S]ize so that the largest is last. I prefer it in the easier to remember sequence:

ls -Sharl

Now, go to firefox, open the webpage that the video is on, and start the video playing. Pause it right away so that the stutter of slowly downloading video doesn't drive you mental. Run the [ls] command above from time to time and you can watch the video file increase in size. It is probably going to end up as the largest file in the /tmp folder - and therefore at the bottom of the list.

Copy this file to ~/Videos, while the firefox tab is still open. This is important, if you navigate away from the page the temporary file is deleted immediately. If the video is from youtube it should just be a x264 video and aac audio in a flv wrapper. You can change wrappers to .mp4 to make the video file compatible with your video player of choice with ffmpeg, while keeping the actual video and audio data unchanged with this command:

ffmpeg -i [your filename here].flv -vcodec copy -acodec copy [output name].mp4

Booting VirtualBox from USB

If you have a Live OS installed onto a USB Key, you may want to test it by booting it from VirtualBox. It turns out that this is a real pain in the arse. There is apparently no way to do this natively, but there is a work around. What you have to do is use a command line (yay!) utility to make a fake hard disk image which points to the USB Key. So VirtualBox THINKS it is booting from a hard disk image but is ACTUALLY booting from the USB Key.

Anyway. The appropriate command in Ubuntu looks like this:

sudo VBoxManage internalcommands createrawvmdk -filename ~/test_usb.vmdk -rawdisk /dev/sdc

That works if you want the new fake disk image to be called [test_usb] and to be stored in [~/] home. Also it links to a USB disk mounted at [/dev/sdc]. The [.vmdk] bit is just the file type for VirtualBox disks.

If you want to do this on Windows, you run a very similar command which looks like this:

VBoxManage internalcommands createrawvmdk -filename "%USERPROFILE%\test_usb.vmdk" -rawdisk \\.\PhysicalDrive1

Apparently [%USERPROFILE%] is the Windows equivalent of [~]. The tricky bit is going to be [PhysicalDrive1]. You are going to have to work out what the number that goes on the end of that is, just as you would have to find out where a USB Key had been mounted in Linux. To do this in Windows run the command line [diskpart] program and then the command [list disk] and it should pop out with a list of the disks attached to the machine with their numbers. It looks like this on my virtual XP:

Microsoft DiskPart version 5.1.3565

Copyright (C) 1999-2003 Microsoft Corporation.
On computer: SB-VIRTUAL

DISKPART> list disk

  Disk ###  Status      Size     Free     Dyn  Gpt
  --------  ----------  -------  -------  ---  ---
  Disk 0    Online        10 GB      0 B

DISKPART>

So I would want to use PhysicalDrive0 in that case. That would be monumentally stupid though, because I would mount the disk I was using to run my OS from in another machine AT THE SAME TIME. I do not want to imagine what the crash would look like.

You will also need to navigate to the install directory for VirtualBox to run the command (the VBoxManage command, not the diskpart command). Linux sticks the program file in the path, but Windows doesn't.

If it didn't go without saying all you do next is just add the .vmdk file as the principal hard disk to your virtual machine.