Friday, 6 May 2011

Making a Natty Live CD

This is further update of my Maverick and Lucid instructions.

I am assuming that you have got to a terminal window by following the start of the Lucid instructions either on a Windows or Ubuntu system.

First of all we are going to need some extra packages for our running system to manipulate the CD image. To install these, run this command from a terminal window.

sudo aptitude install squashfs-tools genisoimage

The CD image contains one large archive file which stores all of the disk environment for the LiveCD. The squashfs-tools handles this archive. Basically, we unpack everything, add our extra packages, and then repack everything. The genisoimage [gen]erates the final [iso] [image] which we put onto the USB Key in due course.

First, make a working directory in [~] home:

mkdir ~/live
cd ~/live

Next we are going to actually mount the CD image so we can use it as if we had burned it to a CD and inserted it. We need to create the mount point first of all:

mkdir mnt
sudo mount -o loop /media/[whatever]/ubuntu-11.04-desktop-i386.iso mnt

The [whatever] will differ depending on exactly where your system mounts the USB Key you just plugged in. The loop [o]ption allows us to mount the image file as a folder.

Now we need to copy all of the files off the mounted image APART from the large squashed file. Again we want a separate folder to store these files:

mkdir extract-cd
rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd

Next we need to extract the big archive file:

sudo unsquashfs mnt/casper/filesystem.squashfs

It uncompresses to a folder name we want to change by the typical linux method of [m]o[v]ing it.

sudo mv squashfs-root edit

We are going to be chrooting into the filesystem we just unpacked, and we want to use some of the files on our existing machine to point the way to the internet, and to give us access to our current hardware:

sudo cp /etc/resolv.conf edit/etc/
sudo cp /etc/hosts edit/etc/
sudo mount --bind /dev/ edit/dev

Now we chroot in and mount some virtual filesystems:

sudo chroot edit
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts

We also need to set some system variables and create a symbolic link for some reason or another.

export HOME=/root
export LC_ALL=C
dbus-uuidgen > /var/lib/dbus/machine-id
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

I think the dbus command generates a code for this specific machine that some installation stuff may need. No idea what the [initctl] stuff is all about.
Excellent. We are now at the point where we can start to install stuff. First of all, and this was fucking frustrating trying to work this out, we need to enable the universe and multiverse repositories if we want to install packages from them. We need to do this in command line:

sudo nano /etc/apt/sources.list

Let's add the PPA's that I covered in a previous post so we can install the useful software they include:

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo add-apt-repository ppa:jd-team/jdownloader

This next command adds an external repository (which is a bigger and more complex type of PPA) to the list of places we can download stuff from. It's a biggie but it basically is just a series of commands that run in sequence.

sudo wget http://www.medibuntu.org/sources.list.d/`lsb_release -cs`.list --output-document=/etc/apt/sources.list.d/medibuntu.list; sudo apt-get -q update; sudo apt-get --yes -q --allow-unauthenticated install medibuntu-keyring; sudo apt-get -q update

Before installing stuff, it might be a good idea to clean out some stuff we are not going to use. First of all have a look at all the installed packages in order of size:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | less

The first on the list looks like a massive package, but what you have to understand is that this is showing you the UNCOMPRESSED sizes. Yeah, thanks for that. If you check on http://packages.ubuntu.com/lucid for the ubuntu-docs information, you find it is only taking up a few hundred Kb when squashed.
So what can you remove? Evolution is a prime candidate. Not much use on a LiveCD. You will be using webmail from a LiveCD.

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep evolution

This will show you all packages which have evolution in the title. It should look like this:

9824 evolution-common
5560 libevolution
2148 evolution-exchange
1580 evolution-data-server
1084 evolution
764 evolution-plugins
380 evolution-data-server-common
124 evolution-indicator
88 evolution-webcal

You can remove all, apart from evolution-data-server-common which is needed by other applications, by running this command:

apt-get remove --purge evolution-common libevolution evolution-exchange evolution-data-server evolution-webcal evolution evolution-plugins evolution-indicator evolution-webcal

The language packs also take up a lot of space, and I do not need anything but English. Find these by running:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep language-

and then remove the ones we do not want:

apt-get remove --purge language-pack-gnome-xh-base language-pack-xh-base language-pack-gnome-zh-hans-base language-pack-zh-hans-base language-pack-gnome-es-base language-pack-gnome-pt-base language-pack-gnome-de-base language-pack-pt-base language-pack-es-base language-pack-de-base

There are other language packs, but they are to small to worry about clearing up unless you are intent on getting this image as small as possible. The final obvious low hanging fruit are foreign font sets. Do a search for [t]rue[t]ype[f]ont packages:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep ttf

12580 ttf-unfonts-core - Korean
6196 ttf-takao-pgothic - Japanese
5456 ttf-thai-tlwg - Thai
5184 ttf-wqy-microhei - Chinese
4204 ttf-freefont - Latin, keep
2628 ttf-indic-fonts-core - Indian
2592 ttf-dejavu-core - Latin, keep
2336 ttf-ubuntu-font-family - These are new to Natty, and are used widely in ubuntu, so best keep them.
1724 ttf-liberation - Latin, keep
592 ttf-khmeros-core - Cambodian
500 ttf-opensymbol - Symbols, needed for OpenOffice, keep
220 ttf-punjabi-fonts - Punjabi
144 ttf-lao - Lao, where ever Lao is
116 ttf-kacst-one - Arabic

apt-get remove --purge ttf-unfonts-core ttf-takao-pgothic ttf-thai-tlwg ttf-wqy-microhei ttf-indic-fonts-core ttf-khmeros-core ttf-punjabi-fonts ttf-lao ttf-kacst-one

I am not going to be printing from the LiveCD so I can remove the software and drivers as follows. Do not worry about the ubuntu-desktop meta file - it is just an easy way of ensuring that all the ubuntu basic stuff is installed. As soon as printing is removed, the installation no longer qualifies as an ubuntu desktop but it doesn't actually remove the rest of the stuff.

apt-get remove --purge cups hplip-data libgutenprint2

I am also going to remove banshee, the rhythmbox replacement, because it does take up a lot of space, and like evolution it is really the type of application you need to set up on an installed machine rather than running from a LiveCD. I have no intention of playing sudoku.

apt-get remove --purge banshee gnome-sudoku

Even after all of those deletions, once I recompressed the image I found I had saved a paltry 90Mb or so. Ho hum. Once you have carried out all the removals, a good strategy is to upgrade all your remaining packages to the latest versions. You do NOT want to upgrade the Kernel or Grub because that causes bad things to happen and will stop the Live USB stick booting. So, create the following files to 'pin' those packages to their current versions:

sudo cat > hold_back_kernel << "EOF"
Package: linux-generic linux-headers-generic linux-image-generic linux-restricted-modules-generic
Pin: version 2.6.38.8.22
Pin-Priority: 1001
EOF
sudo mv hold_back_kernel /etc/apt/preferences.d/
sudo cat > hold_back_grub << "EOF"
Package: grub-common
Pin: version 1.99~rc1-13ubuntu3
Pin-Priority: 1001
EOF
sudo mv hold_back_grub /etc/apt/preferences.d/
sudo apt-get update

You can check that the version numbers are correct (they should be for the Live CD you have downloaded) and check the system knows about the new rules by running these commands:

sudo apt-cache policy
dpkg -l linux-generic
dpkg -l grub-common

The last few lines of all of that should look like this (you can see the version numbers match up):

Pinned packages:
     linux-headers-generic -> 2.6.38.8.22
     linux-image-generic -> 2.6.38.8.22
     grub-common -> 1.99~rc1-13ubuntu3
     linux-generic -> 2.6.38.8.22
root@Dellbuntu:/# dpkg -l linux-generic
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  linux-generic                                 2.6.38.8.22                                   Complete Generic Linux kernel
root@Dellbuntu:/# dpkg -l grub-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  grub-common                                   1.99~rc1-13ubuntu3                            GRand Unified Bootloader, version 2 (common files)

You should now be able to run the upgrade excluding those troublesome packages.

sudo apt-get upgrade

We can now run a massive install command to add the extra packages that we want.

sudo apt-get install \
\
libgtk2.0-dev bison texinfo \
\
flashplugin-installer openjdk-6-jre icedtea6-plugin \
\
libreoffice-java-common libreoffice-l10n-en-gb libreoffice-help-en-gb openoffice.org-hyphenation-en-us openoffice.org-thesaurus-en-us \
\
ttf-mscorefonts-installer \
\
smplayer vlc avidemux audacity \
\
totem-plugins-extra gstreamer0.10-pitfdll gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse \
\
non-free-codecs libavcodec-extra-52 libdvdcss2 libdvdread4 libdvdnav4 \
\
lame mjpegtools twolame mpeg2dec liba52-0.7.4-dev ffmpeg ffmpeg2theora w32codecs \
\
keepassx handbrake-gtk jdownloader \
\
xchm comix gqview pdfmod \
\
wine \
\
transmission \
\
dcraw gimp gimp-data-extras gimp-help-en \
\
celestia celestia-common celestia-common-nonfree stellarium googleearth-package lsb-core \
\
git-core subversion libssl-dev \
\
monodevelop

To install Google Earth from the package downloaded above, you run:

make-googleearth-package --force

You then need to install the package the previous command creates. At the time of writing the version numbering creates a package which can be installed with this command (but this is likely to change):

sudo dpkg -i googleearth_6.0.2.2074+0.6.0-1_i386.deb

I also want a bit of software which is not available in the repositories. Truecrypt is a handy encryption system. Truecrypt is a pain in the arse since they want you to accept their stupid licence instead of just releasing under the GPL.

cd /tmp
wget http://www.truecrypt.org/download/truecrypt-7.0a-linux-x86.tar.gz
tar -xzvf truecrypt-7.0a-linux-x86.tar.gz
./truecrypt-7.0a-setup-x86

Extract the package file - it automatically stores it in /tmp. We need to extract it to /, and it auto installs to the correct folders. So:

cd /
tar -xzvf /tmp/truecrypt_7.0a_i386.tar.gz

And that is that for Truecrypt.

If you were stupid enough to upgrade the kernel package, it is extremely likely that doing a upgrade of all the packages will change the kernel version. To ensure that the new kernel is actually used, you need to go into the ...
~/live/edit/boot
...folder and copy the latest versions of the vmlinuz compressed kernel and the initrd.img files to the ...
~/live/extract-cd/casper
...folder. You then need to delete the existing initrd.lz file, and rename the initrd.img file you just copied over to replace it. Do the same with the vmlinuz files. This has NEVER worked for me so I do not upgrade the kernel package.

If you want the clock in the LiveCD machine to show the proper time, take a moment and set your time zone and keyboard for UK use:

sudo setxkbmap gb
sudo cp -v --remove-destination /usr/share/zoneinfo/Europe/London /etc/localtime

Of course, the keyboard command does not fucking work. I now have no fucking idea how to change the keyboard map from the command line. Brilliant. I mean there must be a file somewhere, anywhere, that actually stores these settings. Where? No fucking clue. Moving on:

We now need to clean up some user account stuff incase any of the installed packages made changes:

awk -F: '$3 > 999' /etc/passwd
usermod -u 500 $hit #where hit is any user ID greater than 999

Right, and we are good to do a general clean up:

apt-get clean
rm -rf /tmp/* ~/.bash_history
rm /etc/resolv.conf
rm /var/lib/dbus/machine-id
rm /sbin/initctl
dpkg-divert --rename --remove /sbin/initctl
umount -l /proc
umount /sys
umount /dev/pts
exit
sudo umount edit/dev

The [rm] commands obviously [r]e[m]ove stuff, and the remaining commands undo the setup commands we used to get into the [chroot] environment. Virtually every time I do this I get a fucking annoying error telling me that it can't umount these things because they are in use. Well, you know what I say to that? Hello Mr. Reboot.

Once the system has come back on, fire up a terminal and:

cd ~/live

Now, update the .manifest file which is a list of installed packages:
chmod +w extract-cd/casper/filesystem.manifest
sudo chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest
sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop

Now delete the existing squashed archive and replace it. Don't worry if it cannot find one, there shouldn't be one the first time you run through these instructions.
sudo rm extract-cd/casper/filesystem.squashfs
sudo mksquashfs edit extract-cd/casper/filesystem.squashfs

That last command will take most time of anything here, as it (re)compressess all the packages we want. If you are feeling particularly narcisstic you can edit the disk details (change the image name) that pop up on boot:

sudo nano extract-cd/README.diskdefines

Now we need to rebuild the md5sum check file:
cd extract-cd
sudo rm md5sum.txt
find -type f -print0 | sudo xargs -0 md5sum | grep -v isolinux/boot.cat | sudo tee md5sum.txt

That will probably take two [sudo] password requests - one for the straight [sudo] and one for the [| sudo] piped version. Do not know why. Irritating as fuck.

And finally we need to build a new iso image:

sudo mkisofs -D -r -V "$IMAGE_NAME" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o ../ubuntu-11.04-desktop-i386-custom.iso .

Once you have that image you use the unetbootin software (which comes in both windows and linux flavours) to load the image onto the USB Key. You need to use version 494 at least to get it to work with Natty. Job done.

No comments:

Post a Comment