get_iplayer update

So the BBC turn out to be a bunch of arses. They got grumpy about open source software accessing the iplayer media. So the nice man who made the get_iplayer software, and who didn't want to lose his house, stopped.

Helpfully, because this happens in the world of open source, someone has picked up the torch. You can now install the wonder that is get_iplayer as follows (in ubuntu):

sudo apt-get install git-core build-essential

This installs a program called 'git' which is used for distribution and collaborating of and on open source projects. It also installs the essential software for compiling programs from source code. Should have probably mentioned that before.

It's best then to make yourself a temporary directory, just in case anything goes wrong. Once there you run the following command:

git clone git://github.com/jjl/get_iplayer.git

and that should download the latest version for you. You then change to the new get_iplayer directory and run:

cd get_iplayer
wget http://download.savannah.gnu.org/releases-noredirect/flvstreamer/source/flvstreamer-2.1c.tar.gz

This downloads the latest version of the actual download software. Uncompress it and build as follows:

tar -xzvf flvstreamer-2.1c.tar.gz
cd flvstreamer
make flvstreamer
make streams
cd ..

Then configure the software settings:

./get_iplayer --prefs-add --modes=flashhd,flashvhigh,flashhigh,flashnormal,iphone
./get_iplayer --prefs-add --flvstreamer "./flvstreamer/flvstreamer"

And the job should be done.

Virtualdub

Time for something slightly different.

If you buy a Sanyo Xacti VPC-HD2000 video camera, you get two things, decent HD video, and lots of shaking.

The camera spits out shakey .mp4 files with h264 video and aac audio streams.

There is an excellent and free deshaker filter available for the virtualdub video editing software for windows.

The deshaker filter only works with virtualdub. Virtualdub does not work with mp4 files. Hmm. Arse.

What you need to do is install ffdshow-tryouts and the Directshow Input Driver for virtualdub. You need to make sure you get version 0.7 at least of the Driver, because 0.5 would not fucking work.

Make sure that ffdshow is set to decode h264 in the VFW configuration dialogue box.

Stick both the plugin files in the VirtualDubPortable\App\VirtualDub\plugins folder, and there you go.

Audio still will not work, however.

Downloading Source Code: Making an Arse of it.

I set out in my last post how to download and compile LZMA. It was, hopefully, relatively straightforwards. Now, what about when it goes wrong. I give you: compiling nano from source code.

Nano is a command line text editor. It is not OpenOffice.org. Is it easy to install? Let's see.

mkdir build-work
cd build-work/
wget http://www.nano-editor.org/dist/v2.1/nano-2.1.10.tar.gz
tar -xzvf nano-2.1.10.tar.gz
cd nano-2.1.10

So far, so LZMA. Let's have a look at the README and INSTALL files. Hmm, INSTALL is not much use:

"These are generic installation instructions."

In other words they are not tailored in anyway for nano, it is just telling you about configure/make/make install.

How about README?

How to compile and install nano

Download the nano source code, then:
tar zxvf nano-x.y.z.tar.gz
cd nano-x.y.z
./configure
make
make install

It's that simple.

Well that sounds promising! Hang on though, there is another README.SVN file; what's that?

GNU nano is available from SVN, but building this needs a bit more care
than the official stable and unstable tarballs.

Well, lets steer well clear of that then, sounds tricky. What ever SVN is. So:

./configure

configure: WARNING:
*** No curses lib available.  Consider getting the official ncurses
*** distribution from ftp://ftp.gnu.org/pub/gnu/ncurses if you get
*** errors compiling nano.

Hmm. Not sure about this. However "if" you get errors. Let's leave this for now.

configure: WARNING: *** Couldn't successfully compile basic color test with or without _XOPEN_SOURCE_EXTENDED
configure: WARNING: *** This build may not compile. Consider configuring with --disable-color or installing ncurses

Ah, no ncurses = use the --disable-color option. Right-o.

checking for HTML support in groff... groff: can't find `DESC' file
groff:fatal error: invalid device `html' (try installing the `groff' package?)
no
*** Will not generate HTML version of man pages  ***
*** Consider installing a newer version of groff with HTML support ***

No groff (whatever that is - sounds like something you tell your kids will come and eat them unless they behave themselves) = no manpages. Don't care about documentation - on we go!

So we need to clean up to do another configure with the no colour option? Sure.

make clean
./configure --disable-color

Result? It is still whining about the lack of a curses lib. Whatever that is. I have personally assembled a fucking large curses lib by now, I can tell you. Still - 'if' ... Let's make it.

make

Hmm. Doesn't seem to have worked - if you peer really closely you will see the following in the output:

makeinfo --no-split --html <> nano.html
/bin/bash: makeinfo: command not found

Right, so what the fuck is makeinfo then? I installed make, presumable makeinfo is something to do with it. Why isn't it installed?

sudo apt-get install makeinfo

E: Couldn't find package makeinfo

Fucking great. Gooooooooogle. Ah. Makeinfo is installed with the texinfo package. Of course it fucking is. Why not.

sudo apt-get install texinfo

Let's clean up and try again.

make clean
./configure --disable-color
make

Oh, for fuck's sake. In a word 'error'. In fact, in many words 'error error error' - just like a 1950's robot with smoke coming out of it's domed head about to go on a a rampage. Right, must be that ncurses nonsense then. Let's try and install it.

sudo apt-get install ncurses

Package ncurses is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package ncurses has no installation candidate

FOR FUCK'S SAKE - don't tease me - tell me WHICH FUCKING PACKAGE REFERS TO IT! *give up* *try groff instead*

sudo apt-get install groff

SWEET JESUS - it worked! Right:

make clean
./configure --disable-color
make

Oh, fucking hell. Back to Google, where I find, eventually, that ncurses actually installed using:

sudo apt-get install libncurses5-dev

Well, clearly it is. I mean of course. The thing is that this is obviously the file that apt-get KNEW ABOUT BUT KEPT A FUCKING SECRET.

make clean
./configure --disable-color
make

It WORKED, it actually FUCKING WORKED! Hang on let's try

./src/nano

YES!

Remember:

It's that simple.

Bastards.

Downloading Source Code - 'Making' it work. Ha.

OK, you have now downloaded your source code. You have worked out that you need to COMPILE the source code to get it to run. You do this by running the 'make' command. All good. So how exactly do you do this again?
OK, I am going to use the LZMA-Utils package as an example. This is a bunch of utilities which use LZMA compression.
So, make yourself some space to work in (ALL of this is going to be command line):

mkdir ~/build-work
cd ~/build-work
wget http://tukaani.org/lzma/lzma-4.32.7.tar.gz

Now, look at the extension to the filename '.tar.gz'. We know that means we have to run:

tar -xzvf lzma-4.32.7.tar.gz

to uncompress the file. Now change to the directory you just created:

cd lzma-4.32.7

Now comes the fun part. You may want to set some configuration options in your program. You may want a video encoder to use specific codecs, or to install the program to a certain location, or to use a certain language, or only to install certain parts of itself. You usually achieve all these goals with the 'configure' command. This is actually a script which comes with the program which sets it all up for you. If there is no configure script that usually means that the program is utterly basic (which may be a good thing) with no options.
For the purposes of this little demonstration, we are not going to actually install the software, because that would be stupid as it would probably break your installation of Linux. If we were going to install the software, we would want to make sure that we specified an install path when we ran the configure script. This option will vary from package to package; you should check the INSTALL or README file in the working directory. If there is no such file, it means the author of the source code is a moron. In this case the option is:

--prefix=/usr

to make it install to the /usr folder. PLEASE NOTE this does not actually install the software here when you run 'make' - it just configures the scripts with an install location. You still need to run a separate command to actually perform the install in due course.
You normally set other options in the same way - you can get a list of options by using the option:

--help

So lets run the script.

configure

configure: command not found

Oh, for fucks sa... oops

./configure

checking if debugging code should be compiled... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... no
checking for g++... no
checking for c++... no
checking for gpp... no
checking for aCC... no
checking for CC... no
checking for cxx... no
checking for cc++... no
checking for cl.exe... no
checking for FCC... no
checking for KCC... no
checking for RCC... no
checking for xlC_r... no
checking for xlC... no
checking for C++ compiler default output file name... 
configure: error: C++ compiler cannot create executables
See `config.log' for more details.

Again, some brilliant communication by way of error messages here. What the script is trying to tell us is that it has run a series of checks to make sure that the computer it is running on is capable of actually compiling the software. It is checking for special software which the 'make' command uses to load the human readable source code and convert it (or compile it) into computer readable code that can be run. It has come to the conclusion that that software does not work, or is missing.
What would be helpful is if it could tell you how to get the software. That is actually really easy. In ubuntu just run:

sudo apt-get install build-essential

to install a group of packages that handle software compilation. These include the following:
dpkg-dev: This is a list of packages which are essential. Yes, this is getting recursive. It's not really compiler stuff, but compression software, the patch command and the perl computer language. You probably have most of this installed anyway, but by telling the computer to install this package, it makes absolutely sure you have everything on the list.
g++: This is really the meat and potatoes of the compilation software. This package includes the Gnu C Complier - the 'gcc' in the error message above, as well as come ancillary software.
libc6-dev: this is a bunch of support files for the C language.
make: Go on, just guess what this package contains.
After your installation, try the configure command again.
You should see a stream of messages coming up which I am not going to copy here, but for the sake of comparison with the above here is the first few:

checking if debugging code should be compiled... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for g++... g++
checking for C++ compiler default output file name... a.out
checking whether the C++ compiler works... yes

Note the 'yes' rather than 'no' answers. Everything should be fine, so run:

make

to compile the software. This will take a few seconds or so.Once the make command has stopped, hopefully without a trail of error messages, you can run:

make check

to run some built in tests. It should tell you that:

All 2 tests passed

Let's draw a veil over the appalling use of grammar. To actually run your newly compiled software you can run:

./src/lzma/lzma -V

which should return the message:

LZMA command line tool 4.32.7
LZMA SDK 4.32

If so, congratulations, you have built software from source code. To install the software - DO NOT DO THIS NOW - you would run:

make install

You will get an error about running this unless you are root, so you would run:

sudo make install

instead. This just copies the files that were created in the 'make' step to the appropriate places in the file system. Once that has finished you can delete the working folder, and you are done!

Downloading Source Code - How'd you run the File?

So, you have managed to download some source code. What on earth do you do with it next? Well, if you do what I first did, you end up with a file and you want to run it. Having experience only of Windows to that point, I double clicked the file that I had downloaded. That only opened a folder with no files in it that appeared to be executable. Hmm.

OK, so doing some sensible checks through Google I discover that I have to turn on a file's executable flag before being able to run it. Right, sounds a bit like adding .exe to the end of the filename, not too onerous. Lets have a go at thi... Wait a fucking minute, what the fuck is chmod 0755? What on earth does that do? After another age of research (bear in mind I am just trying to execute a fucking file) I discover that linux has a whole series of file permissions. They can each be on or off. So what you end up with is a string of '1's or '0's to indicate the state of various permissions. In other words it is a binary number. When you convert the binary number to decimal you end up with something as intuitively obvious as 0755. Fucking spiffing. If I wanted to work in binary I would grow another 38 fucking legs and call myself an 8086 central processing unit.

Thankfully, there is a far more sensible alternative to that command. You can simply ask the system to:

 chmod +x filename

This [ch]anges the [mod]e of the [filename] to allow it to be e[x]ecuted. You can now try and run the file by double clicking the file. AAAAAAND nothing happens. OK, so lets drop to a command line, and change into the directory the file is in. Now type in the name of the file, AAAAAAND

command not found

Yes it is! I have found it! Look there it is in the fucking directory I typed the command in.

At this point I have to say I gave up. Several years later in my experience with Linux I found out that as with microsoft operating systems there is a PATH variable which contains a list of locations in the filesystem in which there are executable files. When the system reports "command not found", it actually means that the file has not been found in any of the directories listed in the PATH variable. The genius part of all this is that Linux DOES NOT HAVE YOUR CURRENT FOLDER IN THE PATH. That means that even if you can see the file you want to run you can't. What you need to do is to type in:

./

before the name of the file which tells the system that you want to run a file which is in the same directory. OK, so lets do that.

Cannot execute binary file

Brilliant. What next? After extensive research I discovered that you cannot simply run the file you have downloaded. Instead you have to 'make' it first. Which came as something of a fucking shock I can tell you - I thought the whole point was some other bugger made it and I used it? Oh, I see, to 'make' a file actually means to compile a working executable file from the source code? Well that makes sense, as long as you change the command from 'make' to 'compile'.

Find out the joys of trying to compile files next time.

Downloading Source Code - What and Why?

You know what one of the biggest pains in the arse is for a new open source user? Open source code.

Yeah, who'da thunk it?

Y'see with windows flavour operating system you download a program. You usually get an install file which you double click, progress through some questions about where you want to install the software, delete the name of the software writer in the install path so you can actually find the fucker in "Program Files" in the future, and click on install.

You then just run the software from a shortcut on the desktop or the Start Menu. Usually either the desktop or the start menu gets bloated with entries to the point where you have to buy a new computer.

Alternatively, you can download a zip file, which you just uncompress to a folder and then make your own shortcuts, and just run the program from that folder.

When you 'run' a program, what I mean is that you execute a file. This is usually a file with the extension .exe, or sometimes .com. Occasionally you run a batch file (extension .bat) which then handles the execution of the .exe file in due course.

With an advanced distribution of Linux, such as Ubuntu, you install new programs using a package management facility. What this means is that Canonical, the company behind Ubuntu, maintain a collection of software that is 'approved' to work with Ubuntu. The installation of the software is completely automated, once you have chosen it in the software store.

Ubuntu, in effect, handles the downloading, and installing of the software. You are left with a menu entry to run the software.

This is, actually, really easy. Like many things that are really easy it is also inflexible. Go on, try and reorganise your menu entries. I dare you.

So what if you want to install software that is not on Ubuntu's approved list? Well then we go from 'really easy' to mind disturbingly fucking hard.

Occasionally, just occasionally, software is provided for download in the same sense as windows install files. You get an archive containing the program files, and any support files. You install the software by double clicking the file (or by typing some ludicrously long command at the command line). The archive itself has a set of instructions built in that tells it where to install the files to, and where (or if) to make the menu entry. So far so good.

However, different distributions of Linux organise their file structures in different ways. So software may be installed in the Linux equivalent of: C:\Program Files\[location] or C:\Application Files\[location] or C:\Some Other Weird Place\[location], depending on your particular distribution. This means that one archive will not work on all distributions of Linux. Ubuntu and other Debian based distributions install archives are called .deb files. Fedora, and other Redhat based distributions use .rpm files.

You cannot use .rpm files on a system expecting .deb files and vice versa (well, OK, if you use a package converter you might be able to. Best of luck with that, let me know how the reinstall goes).

OK, so we have the Distribution approved files in their collections (known as repositories), we have a small fraction of software developers who go to the trouble of making a .deb AND a .rpm AND a .whatever file for their software. What about the other software that you may find in random places in the internet? What about the software that you download in a .tar.gz or a .tar.bz2 file? What the fuck do you do with these? And why would you want to have anything to do with them anyway?

You will eventually encounter and download these files because, while the repository system is useful, it does not have all the software that you may want, or if it does it may not be the version you want. See for instance my vocal complaint about the version of ALSA that came with Ubuntu Karmic. It needed to be upgraded to a newer version, but that version was not supported by Karmic (it is by Intrepid).

Also, some software may be so new it is simply not included in the formal repository system. Lastly, the software may be deliberately excluded from the repositories for intellectual property reasons (if all the source is not free for instance).

See next time for more exciting revelations about source code! Gasp in awe as I try in vain to run a program by making HandBrake-0.9.4.tar.bz2 an executable file and typing its name! Groan in agony as you realise that after downloading the 32k file you have also to download 100Mb of other software just to get it to work - its .net all over a-fucking-gain!