Testing and Fixing SD Cards

You can test for badblocks in a [n]on-destructive way by running:

sudo badblocks -snp # /dev/sdxy

where # is the number of passes to run. This report[s] the progress to you as it runs.

You can then run:

sudo fsck.vfat /dev/sdxy -a -w

This [a]utomatically fixes problems by [w]riting changes to the disk. Any dodgy files you had will be renamed.

Finally, if you have cleared the disk of any remaining stuff, you can run a deep [w]rite and read test by running:

sudo badblocks -swp # /dev/sdxy

If you do not find any physical bad blocks, that suggests that the problem was a software one, and the card remains safe to use.

Making a Natty Live CD

This is further update of my Maverick and Lucid instructions.

I am assuming that you have got to a terminal window by following the start of the Lucid instructions either on a Windows or Ubuntu system.

First of all we are going to need some extra packages for our running system to manipulate the CD image. To install these, run this command from a terminal window.

sudo aptitude install squashfs-tools genisoimage

The CD image contains one large archive file which stores all of the disk environment for the LiveCD. The squashfs-tools handles this archive. Basically, we unpack everything, add our extra packages, and then repack everything. The genisoimage [gen]erates the final [iso] [image] which we put onto the USB Key in due course.

First, make a working directory in [~] home:

mkdir ~/live
cd ~/live

Next we are going to actually mount the CD image so we can use it as if we had burned it to a CD and inserted it. We need to create the mount point first of all:

mkdir mnt
sudo mount -o loop /media/[whatever]/ubuntu-11.04-desktop-i386.iso mnt

The [whatever] will differ depending on exactly where your system mounts the USB Key you just plugged in. The loop [o]ption allows us to mount the image file as a folder.

Now we need to copy all of the files off the mounted image APART from the large squashed file. Again we want a separate folder to store these files:

mkdir extract-cd
rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd

Next we need to extract the big archive file:

sudo unsquashfs mnt/casper/filesystem.squashfs

It uncompresses to a folder name we want to change by the typical linux method of [m]o[v]ing it.

sudo mv squashfs-root edit

We are going to be chrooting into the filesystem we just unpacked, and we want to use some of the files on our existing machine to point the way to the internet, and to give us access to our current hardware:

sudo cp /etc/resolv.conf edit/etc/
sudo cp /etc/hosts edit/etc/
sudo mount --bind /dev/ edit/dev

Now we chroot in and mount some virtual filesystems:

sudo chroot edit
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts

We also need to set some system variables and create a symbolic link for some reason or another.

export HOME=/root
export LC_ALL=C
dbus-uuidgen > /var/lib/dbus/machine-id
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

I think the dbus command generates a code for this specific machine that some installation stuff may need. No idea what the [initctl] stuff is all about.
Excellent. We are now at the point where we can start to install stuff. First of all, and this was fucking frustrating trying to work this out, we need to enable the universe and multiverse repositories if we want to install packages from them. We need to do this in command line:

sudo nano /etc/apt/sources.list

Let's add the PPA's that I covered in a previous post so we can install the useful software they include:

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo add-apt-repository ppa:jd-team/jdownloader

This next command adds an external repository (which is a bigger and more complex type of PPA) to the list of places we can download stuff from. It's a biggie but it basically is just a series of commands that run in sequence.

sudo wget http://www.medibuntu.org/sources.list.d/`lsb_release -cs`.list --output-document=/etc/apt/sources.list.d/medibuntu.list; sudo apt-get -q update; sudo apt-get --yes -q --allow-unauthenticated install medibuntu-keyring; sudo apt-get -q update

Before installing stuff, it might be a good idea to clean out some stuff we are not going to use. First of all have a look at all the installed packages in order of size:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | less

The first on the list looks like a massive package, but what you have to understand is that this is showing you the UNCOMPRESSED sizes. Yeah, thanks for that. If you check on http://packages.ubuntu.com/lucid for the ubuntu-docs information, you find it is only taking up a few hundred Kb when squashed.
So what can you remove? Evolution is a prime candidate. Not much use on a LiveCD. You will be using webmail from a LiveCD.

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep evolution

This will show you all packages which have evolution in the title. It should look like this:

9824 evolution-common
5560 libevolution
2148 evolution-exchange
1580 evolution-data-server
1084 evolution
764 evolution-plugins
380 evolution-data-server-common
124 evolution-indicator
88 evolution-webcal

You can remove all, apart from evolution-data-server-common which is needed by other applications, by running this command:

apt-get remove --purge evolution-common libevolution evolution-exchange evolution-data-server evolution-webcal evolution evolution-plugins evolution-indicator evolution-webcal

The language packs also take up a lot of space, and I do not need anything but English. Find these by running:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep language-

and then remove the ones we do not want:

apt-get remove --purge language-pack-gnome-xh-base language-pack-xh-base language-pack-gnome-zh-hans-base language-pack-zh-hans-base language-pack-gnome-es-base language-pack-gnome-pt-base language-pack-gnome-de-base language-pack-pt-base language-pack-es-base language-pack-de-base

There are other language packs, but they are to small to worry about clearing up unless you are intent on getting this image as small as possible. The final obvious low hanging fruit are foreign font sets. Do a search for [t]rue[t]ype[f]ont packages:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep ttf

12580 ttf-unfonts-core - Korean
6196 ttf-takao-pgothic - Japanese
5456 ttf-thai-tlwg - Thai
5184 ttf-wqy-microhei - Chinese
4204 ttf-freefont - Latin, keep
2628 ttf-indic-fonts-core - Indian
2592 ttf-dejavu-core - Latin, keep
2336 ttf-ubuntu-font-family - These are new to Natty, and are used widely in ubuntu, so best keep them.
1724 ttf-liberation - Latin, keep
592 ttf-khmeros-core - Cambodian
500 ttf-opensymbol - Symbols, needed for OpenOffice, keep
220 ttf-punjabi-fonts - Punjabi
144 ttf-lao - Lao, where ever Lao is
116 ttf-kacst-one - Arabic

apt-get remove --purge ttf-unfonts-core ttf-takao-pgothic ttf-thai-tlwg ttf-wqy-microhei ttf-indic-fonts-core ttf-khmeros-core ttf-punjabi-fonts ttf-lao ttf-kacst-one

I am not going to be printing from the LiveCD so I can remove the software and drivers as follows. Do not worry about the ubuntu-desktop meta file - it is just an easy way of ensuring that all the ubuntu basic stuff is installed. As soon as printing is removed, the installation no longer qualifies as an ubuntu desktop but it doesn't actually remove the rest of the stuff.

apt-get remove --purge cups hplip-data libgutenprint2

I am also going to remove banshee, the rhythmbox replacement, because it does take up a lot of space, and like evolution it is really the type of application you need to set up on an installed machine rather than running from a LiveCD. I have no intention of playing sudoku.

apt-get remove --purge banshee gnome-sudoku

Even after all of those deletions, once I recompressed the image I found I had saved a paltry 90Mb or so. Ho hum. Once you have carried out all the removals, a good strategy is to upgrade all your remaining packages to the latest versions. You do NOT want to upgrade the Kernel or Grub because that causes bad things to happen and will stop the Live USB stick booting. So, create the following files to 'pin' those packages to their current versions:

sudo cat > hold_back_kernel << "EOF"
Package: linux-generic linux-headers-generic linux-image-generic linux-restricted-modules-generic
Pin: version 2.6.38.8.22
Pin-Priority: 1001
EOF
sudo mv hold_back_kernel /etc/apt/preferences.d/
sudo cat > hold_back_grub << "EOF"
Package: grub-common
Pin: version 1.99~rc1-13ubuntu3
Pin-Priority: 1001
EOF
sudo mv hold_back_grub /etc/apt/preferences.d/
sudo apt-get update

You can check that the version numbers are correct (they should be for the Live CD you have downloaded) and check the system knows about the new rules by running these commands:

sudo apt-cache policy
dpkg -l linux-generic
dpkg -l grub-common

The last few lines of all of that should look like this (you can see the version numbers match up):

Pinned packages:
     linux-headers-generic -> 2.6.38.8.22
     linux-image-generic -> 2.6.38.8.22
     grub-common -> 1.99~rc1-13ubuntu3
     linux-generic -> 2.6.38.8.22
root@Dellbuntu:/# dpkg -l linux-generic
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  linux-generic                                 2.6.38.8.22                                   Complete Generic Linux kernel
root@Dellbuntu:/# dpkg -l grub-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  grub-common                                   1.99~rc1-13ubuntu3                            GRand Unified Bootloader, version 2 (common files)

You should now be able to run the upgrade excluding those troublesome packages.

sudo apt-get upgrade

We can now run a massive install command to add the extra packages that we want.

sudo apt-get install \
\
libgtk2.0-dev bison texinfo \
\
flashplugin-installer openjdk-6-jre icedtea6-plugin \
\
libreoffice-java-common libreoffice-l10n-en-gb libreoffice-help-en-gb openoffice.org-hyphenation-en-us openoffice.org-thesaurus-en-us \
\
ttf-mscorefonts-installer \
\
smplayer vlc avidemux audacity \
\
totem-plugins-extra gstreamer0.10-pitfdll gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse \
\
non-free-codecs libavcodec-extra-52 libdvdcss2 libdvdread4 libdvdnav4 \
\
lame mjpegtools twolame mpeg2dec liba52-0.7.4-dev ffmpeg ffmpeg2theora w32codecs \
\
keepassx handbrake-gtk jdownloader \
\
xchm comix gqview pdfmod \
\
wine \
\
transmission \
\
dcraw gimp gimp-data-extras gimp-help-en \
\
celestia celestia-common celestia-common-nonfree stellarium googleearth-package lsb-core \
\
git-core subversion libssl-dev \
\
monodevelop

To install Google Earth from the package downloaded above, you run:

make-googleearth-package --force

You then need to install the package the previous command creates. At the time of writing the version numbering creates a package which can be installed with this command (but this is likely to change):

sudo dpkg -i googleearth_6.0.2.2074+0.6.0-1_i386.deb

I also want a bit of software which is not available in the repositories. Truecrypt is a handy encryption system. Truecrypt is a pain in the arse since they want you to accept their stupid licence instead of just releasing under the GPL.

cd /tmp
wget http://www.truecrypt.org/download/truecrypt-7.0a-linux-x86.tar.gz
tar -xzvf truecrypt-7.0a-linux-x86.tar.gz
./truecrypt-7.0a-setup-x86

Extract the package file - it automatically stores it in /tmp. We need to extract it to /, and it auto installs to the correct folders. So:

cd /
tar -xzvf /tmp/truecrypt_7.0a_i386.tar.gz

And that is that for Truecrypt.

If you were stupid enough to upgrade the kernel package, it is extremely likely that doing a upgrade of all the packages will change the kernel version. To ensure that the new kernel is actually used, you need to go into the ...
~/live/edit/boot
...folder and copy the latest versions of the vmlinuz compressed kernel and the initrd.img files to the ...
~/live/extract-cd/casper
...folder. You then need to delete the existing initrd.lz file, and rename the initrd.img file you just copied over to replace it. Do the same with the vmlinuz files. This has NEVER worked for me so I do not upgrade the kernel package.

If you want the clock in the LiveCD machine to show the proper time, take a moment and set your time zone and keyboard for UK use:

sudo setxkbmap gb
sudo cp -v --remove-destination /usr/share/zoneinfo/Europe/London /etc/localtime

Of course, the keyboard command does not fucking work. I now have no fucking idea how to change the keyboard map from the command line. Brilliant. I mean there must be a file somewhere, anywhere, that actually stores these settings. Where? No fucking clue. Moving on:

We now need to clean up some user account stuff incase any of the installed packages made changes:

awk -F: '$3 > 999' /etc/passwd
usermod -u 500 $hit #where hit is any user ID greater than 999

Right, and we are good to do a general clean up:

apt-get clean
rm -rf /tmp/* ~/.bash_history
rm /etc/resolv.conf
rm /var/lib/dbus/machine-id
rm /sbin/initctl
dpkg-divert --rename --remove /sbin/initctl
umount -l /proc
umount /sys
umount /dev/pts
exit
sudo umount edit/dev

The [rm] commands obviously [r]e[m]ove stuff, and the remaining commands undo the setup commands we used to get into the [chroot] environment. Virtually every time I do this I get a fucking annoying error telling me that it can't umount these things because they are in use. Well, you know what I say to that? Hello Mr. Reboot.

Once the system has come back on, fire up a terminal and:

cd ~/live

Now, update the .manifest file which is a list of installed packages:

chmod +w extract-cd/casper/filesystem.manifest
sudo chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest
sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop

Now delete the existing squashed archive and replace it. Don't worry if it cannot find one, there shouldn't be one the first time you run through these instructions.

sudo rm extract-cd/casper/filesystem.squashfs
sudo mksquashfs edit extract-cd/casper/filesystem.squashfs

That last command will take most time of anything here, as it (re)compressess all the packages we want. If you are feeling particularly narcisstic you can edit the disk details (change the image name) that pop up on boot:

sudo nano extract-cd/README.diskdefines

Now we need to rebuild the md5sum check file:

cd extract-cd
sudo rm md5sum.txt
find -type f -print0 | sudo xargs -0 md5sum | grep -v isolinux/boot.cat | sudo tee md5sum.txt

That will probably take two [sudo] password requests - one for the straight [sudo] and one for the [| sudo] piped version. Do not know why. Irritating as fuck.

And finally we need to build a new iso image:

sudo mkisofs -D -r -V "$IMAGE_NAME" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o ../ubuntu-11.04-desktop-i386-custom.iso .

Once you have that image you use the unetbootin software (which comes in both windows and linux flavours) to load the image onto the USB Key. You need to use version 494 at least to get it to work with Natty. Job done.

Adding a PPA in Maverick

Happy day for those ubuntu users who want the video transcoding application Handbrake. I find this useful because it is the only application that will spit out video that an iPhone or iPad will actually play. Anyway, my live cd instructions contained a breathless explanation of how to build handbrake from source. You no longer need to do that. You just need a ppa instead.

OK, what's a PPA? It is like an official ubuntu source of packages, but it is not maintained by canonical. It stands for personal package archive. These are useful for all kinds of things. Here it is used for software not tested by the people who make ubuntu, and which doesn't form part of the ubuntu repositories of packages.

In addition, a useful download manager is also now available for ubuntu in a ppa: jdownloader. What we are about to do is add two ppa's to our list of sources of ubuntu applications, then update our list of what applications are contained in all of our sources, and then we will install the two applications in question.

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo add-apt-repository ppa:jd-team/jdownloader
sudo apt-get update
sudo apt-get install handbrake-gtk jdownloader

Batch flv to mp4 conversion

If you have been happily downloading and storing flash videos you may end up with a whole pile of flv files. My instructions above included a command to convert each file into an mp4 file which is much easier to work with.

What about converting a whole folder full at once?

Try this:

for file in *.flv ; do ffmpeg -i $file -acodec copy -vcodec copy ${file%.flv}.mp4 ; done

That three stage command does the following (in english):

The first bit before the semi colon means, [for] each [*] file that you find in the current folder with the extension [flv], take the full name of the file forward into the next part of the command as a variable [file]. The next bit says that you [do] the command [ffmpeg] using as an [i]nput the variable [$] called [file]. The options for the ffmpeg command are to [copy] the [a]udio [codec] and the [v]ideo codec, so no recompression. You retain the same data but in a different container. The output of the command is a file with a name made up of the variable [$] called [file] minus [%] the characters [.flv] PLUS the characters [.mp4]. The last bit after the second semi colon closes the command sequence started by the [do]. If you felt brave you could stick a [r]e[m]ove command in here to get rid of your input files in before the [; done].

Listening to BBC Radio Shows on an MP3 player

In the name of bastard fuck, this was far fucking harder than it had any right to be.

You have Windows Media Centre installed on your PC. Either XP, or Vista or 7. Whatever. Fine. You record DVB-T1 broadcast TV programs and watch them quite happily. You record the odd radio program (also broadcast over DVB-T1), and you can play it back on the TV. But who wants to sit around the TV listening to radio programs, what do you think this is, the 1930's?

You have recorded an audio file. You want to listen to the audio on another device. How fucking difficult do you think this is going to be? Oh, boy you are in for a fucking treat.

So, how do we do this? I should interject here and say that there is a fairly straightforwards, albeit time-consuming, way to deal with this. You fire up audacity, change the input source to be the sound card audio out, then hit record and play whatever it is you want to record. The problem with this is that you are recording a compressed track so you are going to lose quality. In my view the BBC are the experts when it comes to digital audio compression and trasmission, and it is going to spoil whatever they broadcast if I recompress it. Its like sitting in the back row of the cinema with your digital video camera. You are not exactly going to walk out with a blue ray quality version of the film are you?

First thing. If using Vista or Windows 7, media centre spits out files with the extension .wtv. These are unreadable by anything but windows, so we need to convert them into a more common format. Helpfully if you right click on the file you will find an option to convert it to dvr-ms file.

What I would then ordinarily do is fire the file up in Video Redo. This is an excellent product for taking in a dvr-ms file, cutting out all the bits you do not want (ads and so forth) and then spitting out the original video and audio streams in an mpg file. You can then happily load that mpg file into avidemux, or your editor of choice and convert it to whatever format you desire.

However, Video Redo shits itself when it is given a dvr-ms file without any video. It just cannot handle it. The more recent versions of Video Redo may be different, but I am not paying for an upgrade for something this simple.

So we want to extract the audio in some other fashion. When something is difficult, then make it harder by doing it on the command line. If we want robust command line editing tools then we want Linux. I have tried ffmpeg under ubuntu with a command that looks a bit like this:

ffmpeg -i filename.dvr-ms -acodec copy -o filename.mp2

Doesn't fucking work. Apparently ffmpeg also cannot handle the earth shattering reality of a dvr-ms file without a fucking video stream.

How about:

mplayer -dumpaudio -dumpfile audio.mp2 filename.dvr-ms

No fucking joy. Cannot ever find an audio stream. It turns out that vlc is practically the only fucking thing on the planet that will spit the untouched audio out of a BBC Digital TV Broadcast Radio Recording. The command is this gem:

cvlc "[filename].dvr-ms" vlc://quit --sout '#transcode{vcodec=none}:duplicate{dst=std{access=file,mux=raw,dst="[filename].mp2"}}'

That's a fucking little beauty isn't it. Far more than you could ever realistically want to know about those options can be found here and here.

How to Post Maths in Blogspot: MathJax & QuickLatex

Edit your template and add the following before the closing "head" tag:

<script type=\"text/x-mathjax-config\">
MathJax.Hub.Config({
  MMLorHTML: {
    prefer: {Firefox: \"HTML\"}
  }
});
</script>
<script type=\"text/javascript\" src=\"http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full\"></script>

Then you just stick the TeX stuff in backslash square brackets for standalone formulae or backslash rounded brackets for in text symbols.

To get graphics you need to use Quicklatex. You need to drop down the choose options box on that page and then insert this text to activate the Tikz graphics package, with the tikz library that lets you calculate co-ordinates using algebra:

\usepackage{tikz}
\usetikzlibrary{calc}

You then get the URL of an image which you can stick in your blog.

Can I Exchange this for something that works?

OK, so they you are running Ubuntu on your work PC. You can connect to the office Exchange server using Evolution so you can get all your email. You are doing all of this using open source software. You are most likely feeling pretty smug.

Eventually you will want to leave the office, so you want to fire on the out of office notifications in Evolution. You read about how to do so. You find out you need to follow this chain of menus and buttons:

Edit -> Preferences -> Edit -> Account Editor -> Exchange Settings -> I am out of the office.

However, at the second "Edit" you get a "Evolution Error" window informing you that the lovely open source program you were oh so smug about "could not read out-of-office state"

Bastard. The problem is that if you google for help on this error, the search terms "state" and "evolution" tend to pollute the results with results about the particular subset of Americans who could politely be called reality-challenged. They could also impolitely be described as the type of fucking cretins who would happily watch their friends and family combusting in agonising pain if their sacred text of choice said that humans were impervious to fire.

Even searching in the Ubuntu Forums, I cannot find an answer to this. Or, come to it, why even though I have selected Global Contact List for Auto Completion in the settings, it doesn't fucking auto complete email addresses from the Global Contacts List.