Testing Natty on Nvidia Hardware

The BIG THING about the Natty version of Ubuntu is that it comes with a prettified GUI. It has a pop out panel on the left hand side of the screen containing your launcher icons. This is absolutely NOT the same as Windows 7's task bar icons OR OSX's panel'o'icons because both of them are on the BOTTOM of the screen. Ahem.

Want to see what this looks like with the live cd? Have nvidia hardware? Tough shit. The nvidia driver installed on the live cd is a) open source (yay!) and b) does not support all the bells and whistles that this flashy new panel needs (boo!). So you will have to install the proper nvidia drivers. Which means rebooting. Which is pointless on a live cd.

Oh dear. This is going to be a pain in the fucking arse isn't it? Yes, yes it is.

For a start a couple of useful commands for this kind of fiddling are:

sudo service gdm stop

This stops dead the GUI leaving you to happily fiddle with graphics driver settings.

To restart the GUI you would just restart the gdm service yes? Yes.

sudo service gdm start

OK. So we have shut down the GUI. Now we want to download the nvidia drivers. I am presuming you have internet access from the LiveCD. If not, see my earlier posts on that hilarious situation. Download the drivers with this command:

wget http://uk.download.nvidia.com/XFree86/Linux-x86/270.41.19/NVIDIA-Linux-x86-270.41.19.run

Make the driver file executable:

sudo chmod +x NVIDIA-Linux-x86-270.41.19.run

And run it:

./NVIDIA-Linux-x86-270.41.19.run

You will now get some whining messages about nouveau bollicksing up the whole plan. This is the open source nvidia software we are trying to replace. So lets just [r]e[m]ove the nouveau [mod]ule:

rmmod nouveau

Doesn't fucking work.

sudo rmmod nouveau

Still doesn't fucking work. Google.

sudo rmmod --force nouveau

Oh for fucks sake. It still doesn't fucking work. It turns out the nouveau driver is deeply embedded in the OS. It ain't possible to just remove it. It has taken over something called the framebuffer. Without getting horrendously technical (shorthand for I have no fucking clue) this is the thing that gives you the text command line that you are using. So ubuntu is trying to stop you doing the I.T. equivalent of sawing off the branch you are standing on. If you turn off the framebuffer you will get a blank screen. Which is not great as far as user interfaces go, but still probably a better experience than Windows ME.

At this point you have two choices. First you can disable the in-depth use of nouveau, which will then allow you to remove it without fucking up the framebuffer. This needs to be done at boot time, each and every time you boot the LiveCD. Secondly, you can write a script that forces ubuntu to remove nouveau (fucking up the framebuffer in the process) and then immediately installs the nvidia drivers with the least input necessary.

The benefit with the first option is that you never get left with a blank screen, but it is a pain. The benefit with the second option is that it can all be scripted, but leaves you with a blank screen for several minutes while it installs the drivers. If it fails for any reason during the blank screen period, you're fucked.

So, option one first. Copy your downloaded driver file somewhere safe (USB Key). Or just download it again next time, what do I care. If you are using a LiveCD, reboot your LiveCD. When it starts back up it flashes up a symbol of a keyboard and a hand. Hit any key at that stage and you will get a boot menu. You want to chose to Test Ubuntu. You then want to hit F6 for other boot options.

If you are using the LiveCD image from a USB Key via UNetbootin, then just highlight the "Test..." option and hit the tab key to edit the boot command.

You should now be able to edit the boot command (it is a long string of options. See the joys of grub elsewhere in this blog for info on what this does). At the end of the boot command type in:

nomodeset

Now complete the boot as normal. Copy back, or re-download the nvidia driver to the home directory. Now, hit CTRL+ALT+F1 to get to a text interface, and make sure the driver file is executable:

sudo chmod +x NVIDIA-Linux-x86-270.41.19.run

:and shut down the GUI:

sudo service gdm stop

:and run:

sudo rmmod nouveau

It should now work. You can now install the driver:

sudo ./NVIDIA-Linux-x86-270.41.19.run

And it should eventually install happily. Answer the questions sensibly, ignoring any errors. Then just run this command to restart the GUI:

sudo service gdm start

Bask in the awe of the flashy bar.

Option 2 is handled like this. Boot as normal - no need for any fancy boot commands. Copy back, or re-download the nvidia driver to the home directory. Open a terminal window and paste this code into it to set up your script:

sudo cat > inst_nv.sh << "EOF"
echo 0 > /sys/class/vtconsole/vtcon1/bind
rmmod nouveau
/etc/init.d/consolefont restart
rmmod ttm
rmmod drm_kms_helper
rmmod drm
chmod +x ./NVIDIA-Linux-x86-270.41.19.run
./NVIDIA-Linux-x86-270.41.19.run --accept-license --no-questions --no-nouveau-check --run-nvidia-xconfig --ui=none
service gdm start
EOF
chmod +x ./inst_nv.sh

(What does this script do? Well, the first line is a command which disconnects the frame buffer from the graphics driver. I think. The second command removes the graphics driver. The third command attempts, but I think fails, to restore some sort of text interface. The next three commands remove more unneeded modules connected with the graphics driver. The [chmod] command makes sure that the nvidia file is executable, and the command after that executes it. All the options added onto the execution of the driver package are designed so that it should just install without asking any stupid questions which you will be unable to read. The last command restarts the GUI.)

You now shut down gdm as before from the CTRL+ALT+F1 text interface:

sudo service gdm stop

:and run the script from the text command prompt:

sudo ./inst_nv.sh

Your screen will go blank. Your cd/dvd drive will hopefully spin up and down. After 5 minutes or so, hopefully, you will get a GUI. If it doesn't work, reboot and try option 1.

And hey fucking presto, there is the snazzy new bar.

Admittedly, after all that, it may be a touch underwhelming.

Manually Add DNS Servers

My work machine is an Ubuntu machine on a Microsoft Small Business Server network. It gets all of its network settings by way of DHCP from the SBS Server. Sometimes the SBS Server has to be restarted. Most of the software I use can run offline, so that is nor normally a problem.

What is a problem is that I immediately lose internet access because the SBS Server operates as a [DNS]. This is frustrating. Helpfully, Ubuntu has an option to manually add some extra DNS's which SHOULD take over the burden from the SBS Server if it goes down.

To achieve this (in Lucid), go System -> Administration -> Network.

From the program that pops up, select the Network Device drop down menu. Choose your network device. This will be ethX if you are connected by a wired connection. Then click on the Configure button next to the device name. From the new window that pops up, choose the DNS tab. You then need to click on the button between the Help and Close buttons at the bottom of the window, and putin your admin password to allow you to change the settings.

Now, just click on Add, and type in your new DNS IP address.

Hopefully this change means that the next time the server goes offline, I can still use the internet.

Testing and Fixing SD Cards

You can test for badblocks in a [n]on-destructive way by running:

sudo badblocks -snp # /dev/sdxy

where # is the number of passes to run. This report[s] the progress to you as it runs.

You can then run:

sudo fsck.vfat /dev/sdxy -a -w

This [a]utomatically fixes problems by [w]riting changes to the disk. Any dodgy files you had will be renamed.

Finally, if you have cleared the disk of any remaining stuff, you can run a deep [w]rite and read test by running:

sudo badblocks -swp # /dev/sdxy

If you do not find any physical bad blocks, that suggests that the problem was a software one, and the card remains safe to use.

Making a Natty Live CD

This is further update of my Maverick and Lucid instructions.

I am assuming that you have got to a terminal window by following the start of the Lucid instructions either on a Windows or Ubuntu system.

First of all we are going to need some extra packages for our running system to manipulate the CD image. To install these, run this command from a terminal window.

sudo aptitude install squashfs-tools genisoimage

The CD image contains one large archive file which stores all of the disk environment for the LiveCD. The squashfs-tools handles this archive. Basically, we unpack everything, add our extra packages, and then repack everything. The genisoimage [gen]erates the final [iso] [image] which we put onto the USB Key in due course.

First, make a working directory in [~] home:

mkdir ~/live
cd ~/live

Next we are going to actually mount the CD image so we can use it as if we had burned it to a CD and inserted it. We need to create the mount point first of all:

mkdir mnt
sudo mount -o loop /media/[whatever]/ubuntu-11.04-desktop-i386.iso mnt

The [whatever] will differ depending on exactly where your system mounts the USB Key you just plugged in. The loop [o]ption allows us to mount the image file as a folder.

Now we need to copy all of the files off the mounted image APART from the large squashed file. Again we want a separate folder to store these files:

mkdir extract-cd
rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd

Next we need to extract the big archive file:

sudo unsquashfs mnt/casper/filesystem.squashfs

It uncompresses to a folder name we want to change by the typical linux method of [m]o[v]ing it.

sudo mv squashfs-root edit

We are going to be chrooting into the filesystem we just unpacked, and we want to use some of the files on our existing machine to point the way to the internet, and to give us access to our current hardware:

sudo cp /etc/resolv.conf edit/etc/
sudo cp /etc/hosts edit/etc/
sudo mount --bind /dev/ edit/dev

Now we chroot in and mount some virtual filesystems:

sudo chroot edit
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts

We also need to set some system variables and create a symbolic link for some reason or another.

export HOME=/root
export LC_ALL=C
dbus-uuidgen > /var/lib/dbus/machine-id
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

I think the dbus command generates a code for this specific machine that some installation stuff may need. No idea what the [initctl] stuff is all about.
Excellent. We are now at the point where we can start to install stuff. First of all, and this was fucking frustrating trying to work this out, we need to enable the universe and multiverse repositories if we want to install packages from them. We need to do this in command line:

sudo nano /etc/apt/sources.list

Let's add the PPA's that I covered in a previous post so we can install the useful software they include:

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo add-apt-repository ppa:jd-team/jdownloader

This next command adds an external repository (which is a bigger and more complex type of PPA) to the list of places we can download stuff from. It's a biggie but it basically is just a series of commands that run in sequence.

sudo wget http://www.medibuntu.org/sources.list.d/`lsb_release -cs`.list --output-document=/etc/apt/sources.list.d/medibuntu.list; sudo apt-get -q update; sudo apt-get --yes -q --allow-unauthenticated install medibuntu-keyring; sudo apt-get -q update

Before installing stuff, it might be a good idea to clean out some stuff we are not going to use. First of all have a look at all the installed packages in order of size:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | less

The first on the list looks like a massive package, but what you have to understand is that this is showing you the UNCOMPRESSED sizes. Yeah, thanks for that. If you check on http://packages.ubuntu.com/lucid for the ubuntu-docs information, you find it is only taking up a few hundred Kb when squashed.
So what can you remove? Evolution is a prime candidate. Not much use on a LiveCD. You will be using webmail from a LiveCD.

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep evolution

This will show you all packages which have evolution in the title. It should look like this:

9824 evolution-common
5560 libevolution
2148 evolution-exchange
1580 evolution-data-server
1084 evolution
764 evolution-plugins
380 evolution-data-server-common
124 evolution-indicator
88 evolution-webcal

You can remove all, apart from evolution-data-server-common which is needed by other applications, by running this command:

apt-get remove --purge evolution-common libevolution evolution-exchange evolution-data-server evolution-webcal evolution evolution-plugins evolution-indicator evolution-webcal

The language packs also take up a lot of space, and I do not need anything but English. Find these by running:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep language-

and then remove the ones we do not want:

apt-get remove --purge language-pack-gnome-xh-base language-pack-xh-base language-pack-gnome-zh-hans-base language-pack-zh-hans-base language-pack-gnome-es-base language-pack-gnome-pt-base language-pack-gnome-de-base language-pack-pt-base language-pack-es-base language-pack-de-base

There are other language packs, but they are to small to worry about clearing up unless you are intent on getting this image as small as possible. The final obvious low hanging fruit are foreign font sets. Do a search for [t]rue[t]ype[f]ont packages:

dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -nr | grep ttf

12580 ttf-unfonts-core - Korean
6196 ttf-takao-pgothic - Japanese
5456 ttf-thai-tlwg - Thai
5184 ttf-wqy-microhei - Chinese
4204 ttf-freefont - Latin, keep
2628 ttf-indic-fonts-core - Indian
2592 ttf-dejavu-core - Latin, keep
2336 ttf-ubuntu-font-family - These are new to Natty, and are used widely in ubuntu, so best keep them.
1724 ttf-liberation - Latin, keep
592 ttf-khmeros-core - Cambodian
500 ttf-opensymbol - Symbols, needed for OpenOffice, keep
220 ttf-punjabi-fonts - Punjabi
144 ttf-lao - Lao, where ever Lao is
116 ttf-kacst-one - Arabic

apt-get remove --purge ttf-unfonts-core ttf-takao-pgothic ttf-thai-tlwg ttf-wqy-microhei ttf-indic-fonts-core ttf-khmeros-core ttf-punjabi-fonts ttf-lao ttf-kacst-one

I am not going to be printing from the LiveCD so I can remove the software and drivers as follows. Do not worry about the ubuntu-desktop meta file - it is just an easy way of ensuring that all the ubuntu basic stuff is installed. As soon as printing is removed, the installation no longer qualifies as an ubuntu desktop but it doesn't actually remove the rest of the stuff.

apt-get remove --purge cups hplip-data libgutenprint2

I am also going to remove banshee, the rhythmbox replacement, because it does take up a lot of space, and like evolution it is really the type of application you need to set up on an installed machine rather than running from a LiveCD. I have no intention of playing sudoku.

apt-get remove --purge banshee gnome-sudoku

Even after all of those deletions, once I recompressed the image I found I had saved a paltry 90Mb or so. Ho hum. Once you have carried out all the removals, a good strategy is to upgrade all your remaining packages to the latest versions. You do NOT want to upgrade the Kernel or Grub because that causes bad things to happen and will stop the Live USB stick booting. So, create the following files to 'pin' those packages to their current versions:

sudo cat > hold_back_kernel << "EOF"
Package: linux-generic linux-headers-generic linux-image-generic linux-restricted-modules-generic
Pin: version 2.6.38.8.22
Pin-Priority: 1001
EOF
sudo mv hold_back_kernel /etc/apt/preferences.d/
sudo cat > hold_back_grub << "EOF"
Package: grub-common
Pin: version 1.99~rc1-13ubuntu3
Pin-Priority: 1001
EOF
sudo mv hold_back_grub /etc/apt/preferences.d/
sudo apt-get update

You can check that the version numbers are correct (they should be for the Live CD you have downloaded) and check the system knows about the new rules by running these commands:

sudo apt-cache policy
dpkg -l linux-generic
dpkg -l grub-common

The last few lines of all of that should look like this (you can see the version numbers match up):

Pinned packages:
     linux-headers-generic -> 2.6.38.8.22
     linux-image-generic -> 2.6.38.8.22
     grub-common -> 1.99~rc1-13ubuntu3
     linux-generic -> 2.6.38.8.22
root@Dellbuntu:/# dpkg -l linux-generic
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  linux-generic                                 2.6.38.8.22                                   Complete Generic Linux kernel
root@Dellbuntu:/# dpkg -l grub-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                                       Description
+++-=============================================-=============================================-==========================================================================================================
ii  grub-common                                   1.99~rc1-13ubuntu3                            GRand Unified Bootloader, version 2 (common files)

You should now be able to run the upgrade excluding those troublesome packages.

sudo apt-get upgrade

We can now run a massive install command to add the extra packages that we want.

sudo apt-get install \
\
libgtk2.0-dev bison texinfo \
\
flashplugin-installer openjdk-6-jre icedtea6-plugin \
\
libreoffice-java-common libreoffice-l10n-en-gb libreoffice-help-en-gb openoffice.org-hyphenation-en-us openoffice.org-thesaurus-en-us \
\
ttf-mscorefonts-installer \
\
smplayer vlc avidemux audacity \
\
totem-plugins-extra gstreamer0.10-pitfdll gstreamer0.10-ffmpeg gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-multiverse gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-multiverse \
\
non-free-codecs libavcodec-extra-52 libdvdcss2 libdvdread4 libdvdnav4 \
\
lame mjpegtools twolame mpeg2dec liba52-0.7.4-dev ffmpeg ffmpeg2theora w32codecs \
\
keepassx handbrake-gtk jdownloader \
\
xchm comix gqview pdfmod \
\
wine \
\
transmission \
\
dcraw gimp gimp-data-extras gimp-help-en \
\
celestia celestia-common celestia-common-nonfree stellarium googleearth-package lsb-core \
\
git-core subversion libssl-dev \
\
monodevelop

To install Google Earth from the package downloaded above, you run:

make-googleearth-package --force

You then need to install the package the previous command creates. At the time of writing the version numbering creates a package which can be installed with this command (but this is likely to change):

sudo dpkg -i googleearth_6.0.2.2074+0.6.0-1_i386.deb

I also want a bit of software which is not available in the repositories. Truecrypt is a handy encryption system. Truecrypt is a pain in the arse since they want you to accept their stupid licence instead of just releasing under the GPL.

cd /tmp
wget http://www.truecrypt.org/download/truecrypt-7.0a-linux-x86.tar.gz
tar -xzvf truecrypt-7.0a-linux-x86.tar.gz
./truecrypt-7.0a-setup-x86

Extract the package file - it automatically stores it in /tmp. We need to extract it to /, and it auto installs to the correct folders. So:

cd /
tar -xzvf /tmp/truecrypt_7.0a_i386.tar.gz

And that is that for Truecrypt.

If you were stupid enough to upgrade the kernel package, it is extremely likely that doing a upgrade of all the packages will change the kernel version. To ensure that the new kernel is actually used, you need to go into the ...
~/live/edit/boot
...folder and copy the latest versions of the vmlinuz compressed kernel and the initrd.img files to the ...
~/live/extract-cd/casper
...folder. You then need to delete the existing initrd.lz file, and rename the initrd.img file you just copied over to replace it. Do the same with the vmlinuz files. This has NEVER worked for me so I do not upgrade the kernel package.

If you want the clock in the LiveCD machine to show the proper time, take a moment and set your time zone and keyboard for UK use:

sudo setxkbmap gb
sudo cp -v --remove-destination /usr/share/zoneinfo/Europe/London /etc/localtime

Of course, the keyboard command does not fucking work. I now have no fucking idea how to change the keyboard map from the command line. Brilliant. I mean there must be a file somewhere, anywhere, that actually stores these settings. Where? No fucking clue. Moving on:

We now need to clean up some user account stuff incase any of the installed packages made changes:

awk -F: '$3 > 999' /etc/passwd
usermod -u 500 $hit #where hit is any user ID greater than 999

Right, and we are good to do a general clean up:

apt-get clean
rm -rf /tmp/* ~/.bash_history
rm /etc/resolv.conf
rm /var/lib/dbus/machine-id
rm /sbin/initctl
dpkg-divert --rename --remove /sbin/initctl
umount -l /proc
umount /sys
umount /dev/pts
exit
sudo umount edit/dev

The [rm] commands obviously [r]e[m]ove stuff, and the remaining commands undo the setup commands we used to get into the [chroot] environment. Virtually every time I do this I get a fucking annoying error telling me that it can't umount these things because they are in use. Well, you know what I say to that? Hello Mr. Reboot.

Once the system has come back on, fire up a terminal and:

cd ~/live

Now, update the .manifest file which is a list of installed packages:

chmod +w extract-cd/casper/filesystem.manifest
sudo chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest
sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop

Now delete the existing squashed archive and replace it. Don't worry if it cannot find one, there shouldn't be one the first time you run through these instructions.

sudo rm extract-cd/casper/filesystem.squashfs
sudo mksquashfs edit extract-cd/casper/filesystem.squashfs

That last command will take most time of anything here, as it (re)compressess all the packages we want. If you are feeling particularly narcisstic you can edit the disk details (change the image name) that pop up on boot:

sudo nano extract-cd/README.diskdefines

Now we need to rebuild the md5sum check file:

cd extract-cd
sudo rm md5sum.txt
find -type f -print0 | sudo xargs -0 md5sum | grep -v isolinux/boot.cat | sudo tee md5sum.txt

That will probably take two [sudo] password requests - one for the straight [sudo] and one for the [| sudo] piped version. Do not know why. Irritating as fuck.

And finally we need to build a new iso image:

sudo mkisofs -D -r -V "$IMAGE_NAME" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o ../ubuntu-11.04-desktop-i386-custom.iso .

Once you have that image you use the unetbootin software (which comes in both windows and linux flavours) to load the image onto the USB Key. You need to use version 494 at least to get it to work with Natty. Job done.