Formatting Video for iPad

This, I have to say, has been one of the most gratifying things to work out in my entire experience of working stuff out. The iPad is an excellent device for consuming media, especially video. The problem is getting video in a format suitable for the iPad without forking over dosh to Apple.

The best I have been able to come up with to date is Handbrake, which has built in modes for the iPad. What it doesn't do is batch convert a whole directory to iPad format.

So joy of joys, I present the Ubuntu commands necessary for this (assuming you have installed the requisite packages all of which are listed in my how to build a custom Live CD posts).

target_resolution="1024x576" &&
target_bitrate="2.5M" &&
minimum_bitrate="0k" &&
maximum_bitrate="3.5M" &&
audio_bitrate="256k" &&
audio_sample_rate="48000" &&
audio_channels="2" &&
input_extension="mkv" &&
number_of_threads="8" &&
for file in *.$input_extension; do \
ffmpeg -y -i "$file" \
-pass 1 -f mp4 \
-s $target_resolution -vcodec libx264 -threads $number_of_threads -b $target_bitrate \
-bt 100k -maxrate $maximum_bitrate -minrate $minimum_bitrate -bufsize 2M \
-flags2 +mixed_refs -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 2 -refs 2 -coder 0 -me_method umh -me_range 90 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 \
-rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -g 90 -an /dev/null && \
ffmpeg -y  -i "$file" \
-pass 2 -f mp4 \
-acodec libfaac -ar $audio_sample_rate -ab $audio_bitrate -ac $audio_channels \
-s $target_resolution -vcodec libx264 -threads 8  -async 2205 -b $target_bitrate \
-bt 100k -maxrate $maximum_bitrate -minrate $minimum_bitrate -bufsize 2M \
-flags2 +mixed_refs -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -subq 5 -trellis 2 -refs 2 -coder 0 -me_method umh -me_range 90 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 \
-rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 -g 90 \
"${file%.$input_extension}.mp4" \
 ; done

You fill in the details between the quotes at the beginning, copy and paste the code into a command line in a directory containing video files of type .input_extension and it will convert all the files to .mp4 suitable for playing on an iPad. All files will have exactly the same resolution and bitrate, which may not be desirable, so check in advance by examining your files.

You can change the target bitrate to whatever you like, within reason, and the iPad will still play the files. You will want to carefully check the aspect ratio of the files going in to make sure that these settings work. If they start in 16:9 this will spit them out in 16:9 at 1024x576. 4:3 files will need the target_resolution changed to "1024x768" for instance.

Don't make the mistake I made of running several instances of this command at the same time - because they will overwrite each others variables.

As a bonus, here are the commands necessary to extract audio from a clip in an iPod/iPhone friendly format:

audio_bitrate="128k" &&
audio_sample_rate="48000" &&
audio_channels="2" &&
input_extension="flv" &&
for file in *.$input_extension; do \
ffmpeg -y -i "$file" -vn \
-acodec libfaac -ar $audio_sample_rate -ab $audio_bitrate -ac $audio_channels \
"${file%.$input_extension}.m4a" \
; done

Same rules apply.

Fixing a fucked Grub2

I have recently encountered a fucked Grub2. This is the latest version of Grub that comes with, amongst other things, Ubuntu Natty. You can't fix it the way I described in the past. That's for Grub1 only.

Why did I end up with a fucked grub? I was stupid. It was entirely my fault. You see, I BELIEVED that this time, when I upgraded my Ubuntu installation from Natty to Oneiric, I wouldn't end up with an unresponsive pile of crap as the result. More fool me.

Because I am not a total buffoon, I made a drive image before attempting the upgrade. I thought I would give True Image Home 2011 another go at being an actual backup program rather than a pointless waste of time. Surprisingly it actually worked. Just about. It had a funny check box thing with some nonsensical question about hard drive ID or some such other thing. I didn't check it. In my experience checking boxes when you do not know what they do is a bad thing.

I now know what this box was asking. It was asking "do you want me not to fuck up your grub when I restore this image?". Oh dear.

So, I was left with a non-working grub. Helpfully I managed to sort it all out, by running these commands from a terminal window on a LiveUSB system.

First of all I mounted the recalcitrant drive into the LiveUSB environment. That's sounds scary. In practice, it means I clicked on the "Places" menu and selected the anonymous drive that was the same size as my Ubuntu, rather than Home, partition on the disk in question.

I then got my terminal window open, and found out where this partition had been mounted by running

mount | tail -1

This command prints the last line [tail -1] of the output of the command to tell you about all [mount]ed partitions. The partition we are interested in should be the last one, because we only just mounted it.

The output I got was along these lines:

/dev/[sdxy] on /media/[string of letters and numbers] type ext4 (rw,nosuid,nodev,uhelper=devkit)

That told me the name of the partition [/dev/[sdxy]] and where it was mounted [/media/[string of letters and numbers]]. That's all I need to know. The next job was to double check that this was the right partition. To do that I just [l]i[s]ted the contents of the [boot] folder on that partition as follows:

ls /media/[string of letters and numbers]/boot

This properly displayed the contents of the boot folder, so I knew I was on the right track. The last step was to run the command to fix everything. This was:

sudo grub-install --boot-directory=/media/[string of letters and numbers]/boot /dev/[sdx]

Please note that I was installing the grub to /dev/[sdx] and NOT /dev/[sdxy], for whatever x and y I got.

And then, surprisingly, it worked fine.

More Ubuntu Wifi Joy

My neighbours are flooding my house with super strength wifi signals. or so it seems. I can't get a reliable signal in the upstairs room where the desktop machine is located. My routers is already set on the "I hate my neighbours, make my signal as loud as possible", setting and that is no longer doing the business.

The obvious solution is to change the channel on which the router is broadcasting. The router I use has an auto setting for this, which I presume means is scans around to find a clear channel and then uses that.

This would be an excellent solution if solving IT problems was completely unlike playing fucking Jenga. Yes, this setting change has fixed the reception problem upstairs, but now the netbook refuses to detect the wifi at all. Fucking spiffing.

This is a driver and software problem. I know this because I can boot to a USB Key, and it works find. I can dual boot to my LFS installation (see posts passim) and that works fine as well. It's just the installed ubuntu that doesn't fucking work. All I did was change the wifi channel for fucks sake. No amount of cursing and swearing appeared to fix this.

I eventually decided to manually install the same network drivers as I have running on the LFS installation. This worked fine. Until the next time I rebooted, when they were overwritten by later drivers, which don't fucking work. I have tried to uninstall the newer drivers, but it's as hard to get rid of the fuckers as it is Michael Meyers. Every time I reboot, the bastards pop into existence once more. Pain in the arse.

(Incidentally the fucking useless drivers, on whatever channel my router is now using because they worked find on the old one, are the broadcom hybrid sta drivers 5.100.something.something. The drivers that actually fucking work are version 5.60.48.36)

All of this explains why I now have a script on my desktop called "get_my_fucking_network_working.sh" which contains the following commands:

cd /tmp
cp '/home/[user]/Dropbox/Essential Drivers/Wireless Drivers/hybrid-portsrc-x86_32-v5.60.48.36.tar.gz' .
mkdir hybrid_wl 
cd hybrid_wl 
tar -xzvf ../hybrid-portsrc-x86_32-v5.60.48.36.tar.gz
unzip '/home/[user]/Dropbox/Essential Drivers/Wireless Drivers/sta_5.60.48.36_2.6.33_kernel_patch.zip'
patch -p0 < patch
unzip '/home/[user]/Dropbox/Essential Drivers/Wireless Drivers/sta_5.60.48.36_2.6.34_multicast_kernel_patch.zip'
patch -p0 < patch_hybrid_multicast
make clean
make
sudo rmmod b43 
sudo rmmod b44 
sudo rmmod b43legacy
sudo rmmod wl
sudo rmmod ssb 
sudo rmmod ndiswrapper 
sudo rmmod lib80211_crypt_tkip
sudo rmmod lib80211
sudo modprobe lib80211
sudo insmod wl

I have to run this script every time I boot. Joy of all joys.

Evolution

Unable to retrieve message Lost connection to Evolution Exchange backend process Absolute fucking bastard. I hate evolution and the way it connects, or rather doesn't connect, to exchange server, with the passion of a thousand, whatever it is's that makes gamma-ray bursts.

Recently

Recently, and surprisingly, not much has been pissing me off IT wise.  The main current niggle is that neither OSX or Windows 7 seem keen to boot from an eSata caddie, whereas Ubuntu just laps it up.

I do have a bit of a grumble, and warning to those less wary, though.  Having purchased an SSD I became something of an evangelist for them.  I convinced the boss at work to buy 14 64Gb SSD drives, and I proceeded to replace the whole office's HDD's with them.

Sadly not a week after delivery we got an email from Kingston, the manufacturer, to advice that there was a teensie problem with the firmware, which has a tiny chance of bricking the drive.  So we update all the firmware as suggested.

Yesterday I sent the third and fourth drives (out of 14) back to Kingston for warranty replacement.  They are SV100S2/64G drives, is anyone is interested.  When they brick, they really do brick.  No detection by the bios, no nothing.

On the plus side, Kingston's RMA system is very good, and we got the replacements for the first drives through quickly.  It's just a shame we have to use it at all.

Installing OSX

Having failed to install OSX from the completely genuine (not really) disk image that I had obtained, I opted to shell out a few quid for a completely genuine (yes, actually) install disk. I ebayed a box set with iWork '09 and iLife '11 which came with all sorts of goodies like word processors, video editors and so on. All for £69. Microsoft, are you listening?

I then had a whole new problem when I tried to boot the system. I got a kernel error. This just froze up the whole machine when I tried to boot. I took out the PCI-E Soundblaster card, and hey presto no more kernel errors. This did not bode well though, as I did want sound eventually. Anyway, ploughing on.

I then had exactly the same bloody problem as with the previous attempt. It would freeze when starting up the installer. What to do?

The solution was actually very simple, but hideously complicated at the same time. What I had to do was to change the boot parameters for the OSX Install disk. This is like adding [nomodeset] when booting a Live Ubuntu Image with dodgy video drivers installed.

The command that I needed to add on were:

PCIRootUID=1

Once I did that the system was prepared to boot. However the wifi was not working. This is a pain, because part of the installation procedure allows you to set up you itunes account, and it needs a network connection for this. It WOULD get an internet connection if I booted the installer in safe mode (by adding [-x] as a boot parameter). Which is just a joy.

The next problem is getting the system to boot without using the iBoot disk. To do so I needed to use the multibeast program to install whatever magical stuff it needed to install to get the system to boot from the hard disk. The two essential parts of multibeast are EasyBeast and the System Tools. Apart from that I just selected packages to get my wireless card working. Happily once I rebooted, I got wifi working just fine. My video card was not detected though, and I put this down to getting an install disk that pre-dated (10.6.3) my video card (6850 Ati). So I then upgraded to OSX version 10.6.8. And hurrah! The video started working properly! And the wireless card was fucked.

Brilliant. I rebooted in safe mode [-x], and the kernel messages flying past tended to suggest the wireless card was detected and working, but that was all academic because the video was now fucked and showing a grey screen only.

Fantastic. It was at this point, after much searching on the internet, that I discovered that I needed to use the further boot parameters:

max_valid_dma_addr=1024

to get the wireless card to work with the latest version of OSX. Apparently this has something to do with how much memory your system has. All I needed to do was to pop those parameters into the /Extra/Library/Preferences/SystemConfiguration/com.apple.Boot.plist file:

<key>Kernel Flags</key>

<string>arch=i386 max_valid_dma_addr=1024</string>

Having stuck those settings in I now have a machine happily booting OSX 10.6.8, with fully working video up to 1080p resolutions, and with fully working wifi. Sound, no so much. The Soundblaster card I have is working after a fashion, but it is unusably crackly. I need to try a variety of the VoodooHDA drivers to find the best one. Alternatively, I need to try it without the card to see if it will output sound over the HDMI cable from the Graphics card. At one point it had detected that as a possible audio out option, but I couldn't connect it up to a receiver at the time to find out if it was actually working.

Installing Latex and PGF

I smugly advised in my last post that you should simply look at my post to identify how to install Latex with PGF Tikz support. Turns out my previous post was as deficient as a very deficient thing on a day of particular deficiency.

Ahem.

What you need to do is to grab two files off t'internet. These are the TexLive Latex distribution (this is a biggie) and the PGF add on.

The TexLive distribution is best downloaded as a single file if you are going to be pissing about trying to repeatedly install it to get the fucker working. It is 2Gb though, so set aside some time for the download. You can find various download options, including torrents, at this page. My nearest CTAN mirror is in the UK, so my command to download the DVD Image is:

cd ~
wget http://mirror.ox.ac.uk/sites/ctan.org/systems/texlive/Images/texlive2011.iso

You can then grab the PGF file by running these commands:

wget http://sourceforge.net/projects/pgf/files/pgf/version%202.10/pgf_2.10.tds.zip/download
mv download pgf_2.10.tds.zip

However, if it is regularly updated, you would probably just want to go to this page and download the latest version manually.

To install the software first you need to mount the DVD image. You can do that as follows:

mkdir ~/texlivedvd
sudo mount -o loop ~/texlive2011.iso ~/texlivedvd

You can then run the graphical installer from TexLive. It requires the perl-tk package, which you can install as follows if you do not already have it:

sudo apt-get install perl-tk

To then run the installer, you use this command:

sudo ~/texlivedvd/install-tl -gui perltk

You want to install as comprehensive a system as possible, but you can omit language packs you are not going to use to save some time and space.

The best idea I have found to avoid permissions bullshit is to install to your home directory, so you would choose an install path like [~/texlive].

Once installed, you want to tell your system where to find the nice new files, which you can do as follows:

cat >> ~/.bashrc << "EOF"
PATH=~/texlive/2011/bin/i386-linux:$PATH
export PATH
MANPATH=~/texlive/2011/texmf/doc/man:$MANPATH
INFOPATH=~/texlive/2011/texmf/doc/info:$INFOPATH
export MANPATH
export INFOPATH
EOF
source ~/.bashrc

What that command does is add those lines to your [.bashrc] file. They add the folders containing the TexLive install to your path so that other programs (like ktikz) can find them. The last command just processes the file to prevent you from having to reboot. I am not entirely sure, but I think this may only affect programs run from the command line, because I have been having problems with ktikz if I run it from the desktop icon. Installing PGF is much easier. Just change into the correct TexLive directory, and unpack the archive:

cd ~/texlive/2010/texmf/
sudo unzip ~/pgf_2.10.tds.zip
sudo texhash

The [texhash] command just (I think) lets the rest of TexLive know that this extra stuff has been installed. That's it for PGF. To install ktikz you are going to need a few packages. Install them as follows:

sudo apt-get install build-essential cmake libqt4-dev qt4-dev-tools libpoppler-qt4-dev kdelibs5 kdelibs5-dev khelpcenter4

Basically what you are doing here is installing software that supports KDE applications. If you happen to be running KDE anyway you should have most of this stuff. Once installed, you can grab the ktikz source code, compile and install it using these commands:

cd ~
wget http://www.hackenberger.at/ktikz/ktikz_0.10.tar.gz
tar -xzvf ktikz_0.10.tar.gz
cd ktikz
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix` ..
make -j2
sudo make install

That should be that. Remember that ktikz needs to load up the PGF packages you want to use. Common ones for me are loaded as follows:

\usetikzlibrary{calc,through, intersections,decorations.text, decorations.pathreplacing}