Friday 25 February 2011

Cleaning Up a Virtualbox XP Install

(Or frankly just cleaning up an XP install. But then XP natively will have a big chunky disc to spread all of its insanity into, whereas a Virtual XP install will probably be quite restricted.)

I have started using VirtualBox to run XP on my Ubuntu workstation just in case I have any compatibility problems with native Ubuntu programs. Like, for instance, the Exchange Server connection in Evolution. This part seems to have been programmed with all the precision and attention to detail of a Jackson Pollock painting. Want to move a large email to a public folder? "The connection to the Exchange Server has been lost". OK maybe that was a tad ambitious. How about just opening a sodding email? "No chance mate, you're getting the raw HTML of the Outlook Web Access login screen instead".

So an XP guest OS with Outlook installed seems like a good idea to cover the rare occasions when Evolution acts like a total dick. Like Tuesdays. Or Thursdays.

I am no expert when it comes to the arcana of Microsoft licensing, but my case has an official sticker on it with a product key, so I think I am safe running XP on it using that key, albeit as a Virtual Machine. Same goes for the Office suite that came with the machine as well. It isn't being used anywhere else since I wiped the disk and stuck Ubuntu on a couple of years ago.

Or Wednesdays, come to think of it.

Well, happily the installation and activation of both XP and Office went swimmingly. My installation CD was only SP1. I installed SP3 over the top of it, but now I have a large collection of crap in the Windows folder to be used in the unlikely event that I ever want to NOT have SP3 installed. Space in general is a bit of an issue. I have a 40Gb SSD drive in the machine, which is more than fine for Ubuntu, but is slightly pushing it for Ubuntu + XP, even as a VM. I gave the virtual machine a 10Gb HDD, which I think is about the bare minimum for an XP install. After installation of Office and a bit of Proprietary Management Software, it is down to less than 3Gb free. Which is not ideal.

So lets fire up Tree Size Free and have a look at where all the space is going.

Biggest Culprit is the Windows folder with 3.5Gb
Next up is the root folder of C: with 1.5Gb
Next is the User folder with about 1Gb in it.
Lastly there is about half a Gb in Program Files.

Yes that adds up to about 6.5Gb. Where has the remaining half gigabyte gone? Two words. System Restore. Great idea in general, no use on a space limited task dedicated machine if you do regular back ups. To remove this, Start -> Right Click on -> My Computer -> Left Click on -> Properties -> System Restore Tab -> Turn Off System Restore Check Box. It is not that hard is it?

OK, first proper clean up - I had left the installation files for the proprietary software lying on the desktop. Delete = 300Mb back. The rest of the User folder was an archive of emails amounting to a couple of hundred Mb which I uploaded to the server, and the local copy of my exchange emails. Can't do anything about them, or indeed the Program Files. Need those to actual run the applications. Still, we've cleared half a Gb. Lets look at the root of C: next.

It has one big file in it: pagefile.sys. It is a whole 1.5Gb itself. This is the famous windows pagefile which the system uses if windows runs out of RAM. Lets run a basic test by opening all the applications installed. Word, Excel, Outlook and the Proprietary software. Done that, and the pagefile usage barely his 250Mb. So, let's trim back the 1.5Gb pagefile to a more sensible 512Mb to give us some headroom.

To do this we go to Start -> Right Click on -> My Computer -> Left Click on -> Properties -> Advanced Tab -> Performance Settings Button -> Advanced (Again) -> Virtual Memory Change Button. I chose a custom size, starting at 512Mb and limited to 1024Mb. Much more sensible for a machine with 1Gb of ram to start with.

Lastly we have the Windows folder. Most of this we DO NOT WANT TO FUCK WITH. However, there are a number of temporary files that we can get rid of. The reason is that XP keeps the files that were replaced by each update or Service Pack install just in case you want to undo the update. As long as it works, I don't. And, as with the System Restore rational, I will be keeping a backup of the install so I don't need the install to back itself up. This is dead easy with virtual disks - you just make a copy of the disk image somewhere. For instance with this command:

sudo apt-get install p7zip-full
7za a XP_10Gb.vdi.lzma XP_10Gb.vdi

So, what can we get rid of in Windows? There are four main folders that I looked at:

C:\Windows\$NtUninstall[blahblah] <- lots of these
C:\Windows\Driver Cache\I386
C:\Windows\ServicePackFiles\I386
C:\Windows\SoftwareDistribution\Download

The first type are just lots of hidden (Explorer Window -> Tools Menu -> Folder Options -> View Tab -> Show Hidden Files and Folders Radio Button & Uncheck Hide Protected Operating System Files) folders storing uninstall data for most windows updates. It is Completely messy of XP to just dump them in the Windows folder.

The next three folders all store downloaded files that have been installed. You MAY need these again if you are installing any new software. What I did was copy all of the foregoing directories into a backup folder, and then stuck that on my NAS [Network Attached Storage]. Bear in mind that you need to install any outstanding updates before moving these files otherwise the update will break and need to be downloaded again.

Also in Windows is a folder called Installer. Do not mess with this. But if you do want to mess with this, google for msicuu2.exe and use that rather than doing it manually.

Anyway, with that clear out, the Windows folder was reduced from 3.5Gb to 1.7Gb, which is ideal. I now have just under 7Gb free, representing a space recovery of 4Gb on a 10Gb drive. Good Stuff.

There is also a temp folder has appeared in the root of C: called MSOCache. This is apparently a copy of the Office install files - presumably in case I want to install any optional items, or parts I chose not to install to begin with. It is only 100Mb or so, and it also appears in the Disk Cleanup program (Start -> My Computer -> Right Click -> C: -> Properties -> Disk Cleanup Button) so I can clean this up easily at a later date if I want.

Why oh fucking why are the stupid Uninstall, Driver Cache, Service Pack, and Downloaded files not also in the Disk Cleanup program? As Advanced options perhaps? As it stands I will have to go through this lengthy process every so often to clean out these unnecessary files. Pain.

Friday 18 February 2011

LAP - Accessing NTFS Partitions

Eventually you may want to use your Linux USB system to access a Windows HDD. It won't be a problem if the windows system uses a FAT32 disk, but those things are rare outside USB Keys now. Most windows HDDs will be formatted with NTFS, and that required some tweaking to get my system to talk to it. First of all I needed to recompile the kernel. Yes, really. If you followed my instructions on how to compress your kernel source directory using LZMA then you can just run the following command to uncompress it to the ramdisk. Makes things a bit quicker, but you need a sensible (2G) amount of ram to make this work.

cd /dev/shm
cat /sources/linux.tar.lzma | lzma -d | tar x
cd linux-2.6.32.8
make mrproper
cp /boot/config-2.6.32.8 ./.config
make LANG=en_GB.utf8 LC_ALL= menuconfig

You then go into the Filesystems section and activate NTFS support (including write support) and also turn on Fuse (as a module). You can now exit the menu system. You compile the kernel, make a backup of your old kernel and settings and copy the new kernel into place by running these commands:

make $CORES_TO_USE
make modules_install
cp -v /boot/vmlinux-2.6.32.8-lfs-6.6 /boot/vmlinux-2.6.32.8-lfs-6.6.backup
cp -v /boot/System.map-2.6.32.8 /boot/System.map-2.6.32.8.backup
cp -v /boot/config-2.6.32.8 /boot/config-2.6.32.8.backup
cp -v arch/x86/boot/bzImage /boot/vmlinux-2.6.32.8-lfs-6.6
cp -v System.map /boot/System.map-2.6.32.8
cp -v .config /boot/config-2.6.32.8

You would be best to reboot now, to make sure the new kernel is working. After you get back up and running, you then download the source code for the necessary software in the usual way:

cd /sources/extras
wget http://prdownloads.sourceforge.net/fuse/fuse-2.7.4.tar.gz
wget http://www.ntfs-3g.org/ntfs-3g-2009.4.4.tgz

Now, move to the ramdisk and uncompress and compile FUSE. I am not entirely clear what FUSE is, but it seems to be a bit of software that lets you load weird and wonderful filesystems.

cd /dev/shm
tar -xzvf /sources/extras/fuse-2.7.4.tar.gz
cd fuse-2.7.4
./configure --prefix=/usr &&
make $CORES_TO_USE
make install &&
rm -rf /etc/init.d
cd ..
rm -rvf fuse-2.7.4

The installation is pretty basic. Not sure why it wants to delete the whole of [/etc/init.d] though. Still my LFS install doesn't use that, so hey ho. Next up is the ntfs driver itself:

tar -xzvf /sources/extras/ntfs-3g-2009.4.4.tgz
cd ntfs-3g-2009.4.4
./configure --prefix=/usr --libdir=/usr/lib &&
make $CORES_TO_USE
make install
cd ..
rm -rvf ntfs-3g-2009.4.4

Again, all pretty straightforwards. We just need to make sure it knows where to put its libraries.

You then mount NTFS partitions with:

mount -t ntfs-3g [device] [mountpoint]

Friday 11 February 2011

Fixing a fucked Grub

If you followed along with my instructions you will have a handy little grub.conf file in your /etc folder. This means that if your grub gets corrupted for whatever reason, it should be a simple matter to get it up and running again. Well, simple for Linux anyway.

Boot from another version of Linux - either from another partition in dual boot, or Live media. Run the command:

sudo fdisk -l

Check the partition ID for the one you want. In my case it was sda7. Now what we want to do is mount the whole of our LFS partition into the filesystem we are currently using. We are then going to chroot (remember that) into that system and effectively reinstall grub, thus:

sudo mkdir /media/problem
sudo mount /dev/sda7 /media/problem
sudo mount -v --bind /dev /media/problem/dev
sudo mount -vt devpts devpts /media/problem/dev/pts
sudo mount -vt tmpfs shm /media/problem/dev/shm
sudo mount -vt proc proc /media/problem/proc
sudo mount -vt sysfs sysfs /media/problem/sys
sudo mount -v --bind /tmp /media/problem/tmp
sudo chroot "/media/problem" /usr/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/bash --login

MAKE SURE the contents of /etc/grub.conf match your the partition that you want to fix. If you want to install Grub to sda7 for instance, you want to see [hd0,6] in the .conf file. If you have another bootloader in the MBR of the drive for dual booting, you want BOTH entries for in the .conf file to show [hd0,6] not one at [hd0]. If you installed to a USB Key and are reinstalling Grub on a different machine, the chances that all the partitions are the same number are very very poor indeed. If that is the case you will need to [nano] into the [conf]iguration file to set it right.

grub --batch < /etc/grub.conf

Job is, as they say, a good'un.

Friday 4 February 2011

LAP - WPA Wireless Security

OK, this one was one of the tasks I set myself at the end of my project last year. I just could not get to grips with the wpa-supplicant documentation. It seemed from reading it that I would have to rebuild my network driver or get a new one or, or, some other bloody stupid thing. Anyway, pleasantly it turned out to be considerably easier than I had lead myself to believe.

So, grab the source code, and move to the ramdisk in the usual way:

cd /sources/extras
wget http://hostap.epitest.fi/releases/wpa_supplicant-0.6.10.tar.gz
cd /dev/shm

Now, we unpack the source code, and build it with these commands. The [defconfig] is a default config file which basically contains all the usual settings that you would use a [./configure] script to set. It is fucking painful to wade through, and was really what put me off this whole endeavour to date. Helpfully, though, with my Compaq Mini, it worked fine with the default settings. i haven't tested it on any other hardware yet, and it may well b0rk if I do.

tar -xzvf /sources/extras/wpa_supplicant-0.6.10.tar.gz
cd wpa_supplicant-0.6.10
cd wpa_supplicant &&
cp defconfig .config
make $CORES_TO_USE
cp -v wpa_cli wpa_supplicant wpa_passphrase /sbin
cd ..
rm -rvf wpa_supplicant-0.6.10

You will notice we did not [make install]. There as no need, we are just installing the three programs that we [c]o[p]ied to [/sbin].

The software also needs a configuration file to work. To connect to a WPA2 enable network with a personal key, you would make the following configuration file:

cat > /etc/wpa_supplicant.conf << "EOF"
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1

network={
        ssid="NAME OF NETWORK"
        proto=RSN
        key_mgmt=WPA-PSK
        pairwise=CCMP TKIP
        group=CCMP TKIP
        psk="VERY SECRET PASSPHRASE"
}
EOF

The NAME OF NETWORK and VERY SECRET PASSPHRASE should be obvious. If you just want to connect to a WEP enabled network, wpa_supplicant can handle that as well. Just make this configuration file instead:

cat > /etc/wpa_supplicant.conf << "EOF"
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
network={
        ssid="NAME OF NETWORK"
        key_mgmt=NONE
        #wep_key0="passphrase"
        wep_key0=HEXCODE
        wep_tx_keyidx=0
}
EOF

Obviously you set the [wep_key0] to whatever you use, either hexcode or passphrase. The [#] operates in the usual way, so that line is blanked out in the example above.

You have the option of putting both the WPA2 and WEP in the same .conf file. What you do is just copy the [network={...}] bit and paste it at the bottom of your .conf file. If you move your laptop/netbook around different locations, you will end of with a list of different network settings. What you then have to do is, at the end of each [network={...}] bit, put a line just before the [}] saying [priority=] and then a number. The software will try the highest numbered networks first, so make sure you give your usual networks the highest priorities.

Incidentally if you want to use WPA rather than WPA2 and you have a Compaq Mini netbook, then as far as my rigorous testing has been able to determine, you are going to have to learn to live with disappointment. I have tried three separate routers, and it just doesn't fucking work.

Lastly we need to create a script to run on boot to run the software:

cat > ~/wifi_wpa_wl.sh << "EOF"
rmmod b43 
rmmod ssb 
rmmod wl
rmmod lib80211
modprobe lib80211
modprobe wl
wpa_supplicant -D wext -i eth1 -c/etc/wpa_supplicant.conf -B
dhcpcd eth1
EOF

You will spot the similarity to the iwconfig wifi script I used in the past. The [-D wext] setting tells the software it can use the standard linux [w]ireless [ext]tensions, and not some fancy driver. These must be installed with the kernel, because I do not remember compiling them. The [i]interface and [c]onfiguration file commands are self explanatory. The [B] option tells it to run as a daemon.

Useful tips:
Run the wpa_supplicant command from the script with -d instead of -B to test the settings and report back error messages. The command:

wpa_passphrase ESSID PASSPHRASE

(where ESSID is the name of your wireless network, and PASSPHRASE is, well, work it out) will output an encrypted version of your passphrase for you to pop into the wpa_supplicant.conf file instead of the clear text. Remember to:

chmod 600 /etc/wpa_supplicant.conf

to protect the file from all but root. You can also run:

wpa_cli status
wpa_cli scan
wpa_cli scan_results

to obtain some self evident information.