Friday 28 January 2011

get_iplayer 2011

[This Update has now been Updated]

Time, I think, for an update on the get_iplayer situation. For the last few months I have found that my previous instructions work just fine for downloading SD content. They were borked though, for HD content. Which has been a crying shame, because that's the whole sodding point of get_iplayer as far as I am concerned. Want to watch BBC HD content but have an ISP comprised of a larger collection of cocks than the mid-west's bi-annual prettiest rooster award? Tough, your bandwidth will never be enough to watch streaming HD. Unless, of course, you set your alarm for 3 a.m.

So, time for an update, especially when I noticed that my previous and well thought out instructions were being described as out of date. The shame.

I find it is best to start with a vanilla version of Ubuntu when working out how to do stuff like this, otherwise you never know which package you installed six months ago and forgot about is actually essential to the whole enterprise. You then write down a careful list of instructions for someone else to follow who has NOT installed the critical package, and it doesn't sodding work, and you end up looking like a half-wit.

So, best to use the latest Ubuntu LiveCD in a virtual machine for testing purposes. [You can then make sure that everything actually works. If it doesn't the problem is somewhere at your end. If it does work you can go on to try to install the software on your distribution of choice, and enjoy some dependency hell, but knowing all the time that the theory is sound. None of this is necessary, if you want to you can just skip over all the virtualisation stuff and get straight to the software, go ahead.]

A perfectly sensible virtual machine is VirtualBox. You will find the installer package here. Grab the one for your architecture (intel/amd basically). You can even install the Windows version and then run Ubuntu inside the VirtualBox, and follow along with these instructions if you so desire.

If you are running it from Ubuntu, you also need to install the following package to get a screen resolution more than 800x600. This is optional if tiny resolutions do not bother you, or if you remember the time when 640x480 was standard and 800x600 was High-End.

sudo apt-get install virtualbox-ose-guest-x11

Now reboot your host machine (the physical machine you are working on).

When it comes back on, you may need to run the following command to get VirtualBox to work - I need to using Lucid:

sudo modprobe -r kvm_intel

Welcome back Windows people. Next step is to grab the Ubuntu CD from here. 32bit is fine. [I have also tested this with the latest (10th February 2011) daily build of Natty, and it still works fine - in fact the VirtualBox extensions aren't needed to get a decent resolution.] Once that has downloaded, fire up VirtualBox and create a new machine. I gave mine 1024Mb RAM. For our current purposes we will not actually be installing the OS, just testing get_iplayer and friends, so do not bother with a Hard Disk. In my version, that is all that is needed to create the machine, which will of course NOT WORK, because there is no boot media.

So, next click on the machine, and hit the settings button. I then went to the Storage option on the left hand side of the window which pops up. I clicked on the shiny CD with 'empty' next to it, and then in the drop down menu under Attributes on the right hand side I changed it to Primary Master. I then clicked the small CD button next to the drop down menu and choose to use a virtual CD/DVD file. I then just navigated to where ever I downloaded the Ubuntu LiveCD.

Next, to optionally get a better screen resolution, I went back to the middle panel, and clicked on the IDE Controller line and clicked the CD image with the green plus on it to add a new CD drive. I opted to choose a disk, and navigated to

/usr/share/virtualbox/VBoxGuestAdditions.iso

I am not sure where that file will be located on a Windows install, but you can presumably search for it. [I tested this on Windows 7 and it worked fine. The .iso I refer to in stored in the Program Files folder where VirtualBox is installed.] It is not essential for what comes next though - it just make using VirtualBox a nicer experience.

I then fiddled about with some other settings. I activated USB 2.0, and I wanted to us 2D acceleration but that only works on Windows machines. Having done all that I just clicked OK at the bottom right of the window. I then just made sure my Machine was still selected at the left hand side of the window and hit the start button. I chose to try Ubuntu rather than install.

OK. We should now be at the Ubuntu desktop. It is a bit cramped at 800x600. If you are fine with that, just scroll down. If not, mount the Additions CD (just click on it in the places menu) pop open a terminal window and run:

sudo /media/VBOXADDITIONS_4.0.2_69518/VBoxLinuxAdditions.run

It took a few seconds to install and then invited me to restart the machine. That would be DAFT on a live cd, so instead just shut down X:

sudo /etc/init.d/gdm stop

And when the dust has settled, you need to switch to a text terminal screen. You would ordinarily do this with Control + Alt + F1, but if you are using an Ubuntu host, that will just change you to the terminal screen on your host. Instead use the RIGHT Control Key, NO Alt, and hit F1. Now restart X:

startx

Bingo, better screen resolution.

Now, onto get_iplayer. Welcome to people who could not be bother with VirtualBox.

First open a terminal, make sure you have the latest package information and install some software we will be needing:

sudo apt-get update
sudo apt-get install git-core build-essential subversion libssl-dev ffmpeg

If you are playing along at home, you should now be able to paste commands straight into the Virtual machine. [So what is this extra software? 'git-core' and 'subversion' do essentially the same thing. These are programs that let you download software, including source code, from internet sites. The software maintainers are able to keep track of different versions of their software and you can interact with it. All we will be doing is downloading the latest version. 'build-essential' is a collection of tools that let us compile software from source code into executable files and libraries. 'libssl-dev' contains software that needs to be incorporated in the programs we are downloading when they compile. SSL is a secure communication system for the internet, and it is already installed on your machine. However, we need the 'dev'elopment files for it, so that it can get built into the new software. All you have at the moment is the end product. Finally 'ffmpeg' is a powerful command line video converter which is going to be used to convert the downloaded video from flash format to the easier to use .mp4 format.]

Once that software has downloaded, make sure your terminal prompt shows you are in a sensible place. Your home directory is just fine:

ubuntu@ubuntu:~$

If not, change to your home directory by running:

cd ~

Now grab the latest version of get_iplayer by running this command (it will make a directory called get_iplayer in directory you run the command from, which is why we made sure you were in home):

git clone git://git.infradead.org/get_iplayer.git

Move into the directory that was just made and download the latest rtmpdump (which will be downloaded into its own folder much like get_iplayer):
cd get_iplayer
svn co svn://svn.mplayerhq.hu/rtmpdump/trunk rtmpdump

Then move in to the rtmpdump folder and build the software:
cd rtmpdump
make SYS=posix

The [SYS=posix] tells it to build the Linux version, as opposed to Windows or whatever else is possible. I then test ran the rtmpdump program that had been created and got this:

./rtmpdump
./rtmpdump: error while loading shared libraries: librtmp.so.0: cannot open shared object file: No such file or directory

This got no better when I [make install]ed the rtmpdump software. I would prefer to keep everything in [~] home if I can. The error was complaining that it could not find a library. The library was right there in a subfolder, so I just made a link to the Ubuntu library folder:

sudo ln -sv ~/get_iplayer/rtmpdump/librtmp/librtmp.so.0 /lib

I then tested again, and got:

RTMPDump v2.3(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPLERROR: You must specify a hostname (--host) or url (-r "rtmp://host[:port]/playpath") containing a hostname

Which was much better. Now to set up get_iplayer, I dropped back down to the get_iplayer directory, and ran the program once to update its plugins from the net:

cd ..
./get_iplayer

And then I setup the usual preferences I use:

./get_iplayer --prefs-add --modes=flashhd,flashvhigh,flashhigh,flashnormal,iphone
./get_iplayer --prefs-add --flvstreamer "/home/ubuntu/get_iplayer/rtmpdump/rtmpdump"

It was quite happy being told to use rtmpdump under the flvstreamer preference. Job is, as they say, a good'un.

Friday 21 January 2011

Acronis True Image Home 2011

Through several annoying, and a few heartbreaking, incidents I have learned well the IT lesson that "Data you do not have backed up is data you do not want". So I back stuff up. I back stuff up in a number of ways.

Firstly, I have an external drive from seagate (the clicky clicky special referred to in my post about encryption). It has a 5 year warranty, which is a good sign (and the clicky clicky sound has gone away). On that drive I mirror My Music, My Documents, My Videos, My Pictures and My Dropbox folders. I use Allway Sync for this, and it is pretty good. It is available as a "free" version, but is actually a nagware version, because after you have synced so many files it starts hassling you to buy the full version. It is, however, very intuitive to use. File synchronization software is not, by and large, written by people who have an innate talent for user interface design. When you run Allway Sync it pops up suggestion tip boxes telling you what to click and where to type to do a basic sync. You can completely ignore these if you like, and you will do so once you get to grips with it, but it is very welcome at the start.

With this backup method, what I end up with is a copy of all my data on an external disk. All I need to do is connect up the external drive, fire up Allway Sync and run the saved settings. The length of the backup just depends on how much stuff has changed since the last time I did a backup. It can be very very quick in deed. I am now into the habit of running one everytime I buy some new iTunes music, or upload some new digital photos from my camera. I treat all new material as not really existing until such time as I have run a sync.

It is also worth pointing out that you can set the software up to sync in one direction or both directions. Both directions is useful if you are moving the external drive between machines, where you may be adding data to it from both machines. The two way sync makes sure that every machine has the same data on it as the other machines. Ideal for sharing an iTunes library around the house. If running with a one way sync - just to back up stuff from one machine, you can also choose NOT to replicate deletions, so you keep a copy of all files on the external drive even if you delete them (accidentally!) on the main machine.

The second backup method I use is Dropbox. This allows you to select a folder on your machine which is then replicated on the Dropbox company's servers. If you install the software on another machine you get a copy of the folder on that machine as well. It works with Windows and Linux. I have it installed on every machine I use (netbook, HTPC, Desktop, and both of my work machines). That is a mixture of Ubuntu and Windows 7 machines. I get exactly what it says on the tin - one folder which is on all of those machines.

Dropbox, like AllwaySync, comes in a free version. You get 2Gb of space to use before you have to part with any cash. I just use the free version, and keep the folder contents to the really important stuff. Like, for instance, password safe databases, bitlocker recovery keys and so on. This backup method is very useful because it is off site storage. So if the house burns to the ground, I still have all the data in my dropbox folder. Also, if the dropbox company goes out of business I have all the data in the folder on 5 separate machines in three different buildings. Quite, quite, secure.

The third type of backup I do is to take an image of my system partition on various machines. If you are connected to the internet, you are going to get some sort of problem software installed at some point. Or you are going to change a stupid setting in the OS which is a pain to put right. Or you may just have a good old fashioned hardware failure. The BEST way of resolving the foregoing issues that I have found is not to bother with manuals, or anti-malware programs, but just to say "fuck it" and restore the last working image of the system partition. As a rough rule of thumb every gigabyte of data takes about 1 minute to restore. So if you install windows and a couple of bits of software you are only looking at 10-15 minutes to get back to a fresh working machine. You can EASILY spend that amount of time trying to get rid of whatever rootkit has infected your machine.

The software that I use for this task is True Image Home from Acronis. Unlike the other options I outlined above, this is not free (although there is a free trial). It does a couple of very useful things which made me spend money on it. Firstly, it lets you mount the archives it creates as virtual disks, so if you just want to grab a file or two out of it you can. Secondly, it lets you build a bootable USB stick so you can boot from that and use the software without having installed it. Go on, use Windows Backup to restore an image to a bare metal HDD. Not possible. You need to install Windows FIRST. Also the USB Key approach will happily back up and restore Linux partitions as well.

I had an older version of the software which I had to replace because it did not recognise my SATA connected hard disks. When browsing in the shop, I found two different versions of the 2011 software: a netbook "optimized" version which came on a USB key, and a Desktop version. The Desktop version cost £10 more. Now, I decided that I could put up with the inconvenience of using software designed for a netbook screen on a larger screen in exchange for a ready made USB key and the £10 price difference.

It turns out, however, that when Acronis say netbook "optimized" they actually mean netbook "crippled". Because you see (and it mentions this nowhere on the fucking box) it ONLY WORKS ON NETBOOKS. Yes, the software starts up on the desktop and says to itself, "hang the fuck on, this is not a netbook processor, I am going to go in a huff". The software was therefore completely fucking useless. Apart from on my netbook. Worked fine there.

So back to the shop. Explain predicament. Part with £10. Get full version. Go home. Install full version. Run a system disk backup.

Now, fast forward to last Saturday. I was downloading some hefty multipart files and the machine kept crashing. Odd, I thought. Possibly a hardware issue. I broke out in a cold sweat with thoughts of the Hardware Fuckup trilogy. Thankfully though, it did not seem to be hardware related. All temps were fine, and it kept going for a few more hours before crashing again. Hmm. Quick browse with Malwarebytes Anti-Malware (again the free version) revealed that yes, a nasty infection of something or another had taken hold. Bugger.

First solution is the "fuck it" solution. Fire up Acronis, tell it to restore from the previous Wednesday's backup. It tells me it needs to restart the machine to restore to the system drive. Fine, fine, just get on with it. It restarts ... back into Windows. No sign of a backup being restored. Arse. Next, grab a USB stick, set it up as bootable media and boot from that. Fine. Select archive to restore. It cannot be restore because there may not be enough space left to boot. What. The. FUCK. is that about? This is a mirror of a whole disk. It is not possible to fit more stuff in the image than was on the disk in the first fucking place.

I have never seen this message on earlier version of True Image. So I take a punt and download the trial version of the 2010 software. It complains that the image is corrupted. A Ha! says I. Maybe the error message is some sort of default and actually the archive is bollocksed. So I reinstall the 2011 version and validate the archive. The archive is fine. Fucking arseholes. This must be a backwards compatibility issue. So I then use the 2011 software to convert the archive from True Image format to Windows Backup format, boot to the USB key version 2010 and ... it complains about the image being too big for the blah blah. However, it does at least let me IGNORE THE BULLSHIT ERROR and just restore the image anyway. Which is what it is doing now. [UPDATE] It has now finished and the restore went fine. So if you backup using TIH2011, all you need to do to restore an image is to convert it to a Windows Backup Image, remove TIH2011, install TIH2010, create a USB rescue disk, boot from the same and restore the Windows Backup Image. Simples.

I also discovered a little gem of a bug in the software. I like incremental backups, where only the changes since the last backup are recorded. It lets you keep lots of different versions for relatively little hard disk space. There is only one problem with this option in the 2011 version. It does not fucking work. With one archive, which extends to 2Gb, I watched as it made the first archive file, then a day later made the second (7Kb - no changes), then a day later made the third file (also 7Kb no changes) by OVERWRITING THE FIRST FILE. Fucking brilliant. All data gone. Even when all of the data is demonstrably there it complains about being unable to find the original volumes. Fucking useless. I now have two 7Kb files and no 2Gb data anywhere.

Apparently both of these problems are known in the world of Acronis Users. This has left me fucking pissed off. I have backup software which, at a fairly fundamental level, does not fucking work. If it wasn't bad enough that it overwrites backup files, it also DOES NOT LET YOU RESTORE FROM BACKUPS THAT YOU HAVE MADE. Backup software which just backs up and does not restore IS NOT BACKUP SOFTWARE; it is a useless pile of crap.

I loved the previous version of Acronis I had. I can't even remember where I got it - it may have come bundled with the Seagate. It worked fine. I used it for months doing regular backups of my important stuff, and I did restores from time to time. It just would not detect my SATA hard disks on my new motherboard. Now I have the latest 2011 version which is useless.

Now, what about this Norton Ghost stuff then?

Friday 14 January 2011

Finding correct Ubuntu packages

I have ranted at length in earlier posts about the hassle of trying to install software from source on Ubuntu. What you end up with is a conflict between the Ubuntu package naming conventions and what the source code developer has put in its error messages. See for instance the texinfo/makeinfo nonsense here.

I have stumbled upon an excellent way to resolve this. What you do is go to http://packages.ubuntu.com, and then scroll down to the "Search the contents of packages" section.

In the "Keyword" box type in the name from the source code error - such as "makeinfo". In the dropdown list next to "Distribution" choose the version of Ubuntu you are running. Then hit "Search".

Hey presto, the third result is [/usr/bin/makeinfo] and the associated package is [texinfo]. Brilliant.

Friday 7 January 2011

Installing Conky

cd /dev/shm
wget http://sourceforge.net/projects/conky/files/conky/1.8.1/conky-1.8.1.tar.bz2/download
cd /dev/shm
tar -xjvf conky-1.8.1.tar.bz2
cd conky-1.8.1
./configure --prefix=/usr --enable-wlan --disable-lua --enable-x11 --enable-curl
make -j2
#make install
/dev/shm/conky-1.8.1/src/conky

cat > ~/.conkyrc << "EOF"
# set to yes if you want Conky to be forked in the background
background yes

# Use Xft?
use_xft yes

# Set conky on the bottom of all other applications
#on_bottom yes

# Xft font when Xft is enabled
xftfont Deja Vu Sans Mono:size=7.5

# Text alpha when using Xft
xftalpha 0.15

# Update interval in seconds
update_interval 5.0

# This is the number of times Conky will update before quitting.
# Set to zero to run forever.
total_run_times 0

# Create own window instead of using desktop (required in nautilus)
own_window no

# If own_window is yes, you may use type normal, desktop or overide
#own_window_type desktop

# Use pseudo transparency with own_window?
#own_window_transparent yes

# If own_window is yes, these window manager hints may be used
#own_window_hints undecorated,below,skip_pager

# Use double buffering (reduces flicker, may not work for everyone)
double_buffer yes

# Minimum size of text area
minimum_size 280 5

# Draw outlines?
draw_outline no

# Draw borders around text
draw_borders no

# Stippled borders?
stippled_borders 8 no

# border margins
#border_margin 4

# border width
border_width 1

# Default colors and also border colors
default_color white

# Text alignment, other possible values are commented
#alignment top_left
alignment top_right
#alignment bottom_left
#alignment bottom_right
#alignment none

# Gap between borders of screen and text
# same thing as passing -x at command line
gap_x 12
gap_y 12

# Subtract file system buffers from used memory?
no_buffers yes

# set to yes if you want all text to be in uppercase
uppercase yes

# number of cpu samples to average
# set to 1 to disable averaging
cpu_avg_samples 2

# number of net samples to average
# set to 1 to disable averaging
net_avg_samples 2


# Force UTF8? note that UTF8 support required XFT
override_utf8_locale yes

# Add spaces to keep things from moving about? This only affects certain objects.
use_spacer none

# variable is given either in format $variable or in ${variable}. Latter
# allows characters right after the variable and must be used in network
# stuff because of an argument

# stuff after 'TEXT' will be formatted on screen

TEXT
$color
${color orange}SYSTEM ${hr 2}$color
$nodename $sysname $kernel on $machine

${color orange}CPU ${hr 2}$color
${freq}MHz   Load: ${loadavg}   Temp: ${acpitemp}
$cpubar
${cpugraph 000000 ffffff}
NAME             PID       CPU%      MEM%
${top name 1} ${top pid 1}   ${top cpu 1}    ${top mem 1}
${top name 2} ${top pid 2}   ${top cpu 2}    ${top mem 2}
${top name 3} ${top pid 3}   ${top cpu 3}    ${top mem 3}
${top name 4} ${top pid 4}   ${top cpu 4}    ${top mem 4}

${color orange}MEMORY / DISK ${hr 2}$color
RAM:   $memperc%   ${membar 6}$color
Swap:  $swapperc%   ${swapbar 6}$color
Root:  ${fs_free_perc /}%   ${fs_bar 6 /}$color

${color orange}NETWORK (${addr eth1}) ${hr 2}$color
Down: $color${downspeed eth1} k/s ${alignr}Up: ${upspeed eth1} k/s
${downspeedgraph eth1 25,140 000000 ff0000} ${alignr}${upspeedgraph eth1 25,140 000000 00ff00}$color
Total: ${totaldown eth1} ${alignr}Total: ${totalup eth1}
${execi 30 netstat -ept | grep ESTAB | awk '{print $9}' | cut -d: -f1 | sort | uniq -c | sort -nr}
$color Wifi Quality: ${color #07dbf4}${wireless_link_bar eth1}

${color orange}Battery: ${color #ec0000}
${battery BAT1} $color
${battery_bar BAT1}

${color orange}Master Volume:${color}${execi 3 ~/conky_volume.sh}
#The next line does not work if the script outputs text, such as mute.  Need to fix this.
${execibar 3 ~/conky_volume.sh}
EOF

cat > ~/conky_volume.sh << "EOF"
#!/bin/bash
amixer get Master | awk -F'[]%[]' '/%/ {if ($7 == "off") { print "Mute" } else { print $2"%" }}'
EOF
chmod +x ~/conky_volume.sh