LAP - Toolchain, Part 7 - Finish

I mentioned at a much earlier stage that I was write permitting a program to allow it to be edited later.  Now is that later.  What we are doing is clearing out unnecessary documentation and debugging symbols to reduce the size of the Toolchain.  Not strictly necessary, but you may well want to back up the toolchain once you are finished building it so you do not have to go through all of this again.  If so, you would want the backup to be as small as possible.  Also if you are installing to a small disk then you will need lots of space to install the software to come.

We achieve this size reduction quite simply:

strip --strip-debug /tools/lib/*
strip --strip-unneeded /tools/{,s}bin/*
rm -rf /tools/{,share}/{info,man}

The strip command is clever enough to remove the debug stuff automatically, but the command to [r]e[m]ove the documentation is a bit more brutal.

Finally, the toolchain is currently owned by the lfs user, because that is how we made it.  This will screw it up once we chroot into it, because there is no lfs user in the new system; only the host.  SO we need to change the ownership of the whole tools directory.  First exit the su - lfs system:

exit

Then simply:

chown -R root:root /media/lfs/tools

This command [ch]anges the [own]ership of the [/media/lfs/tools] folder to the user [root] of group [root].  Simple enough.  Oh, hang on, not it isn't.  Error fucking permission bullshit city again.

sudo chown -R root:root /media/lfs/tools

That's got the bastard.

The toolchain is DONE!  Have a beer.

LAP - Toolchain, Part 6 - General

We are now in the realm of general packages.  The first is ncurses.  Sadly this is not a library of choice swear words, instead it is a library which helps make text user interfaces.

tar -xzvf $LFS/sources/ncurses-5.7.tar.gz 
cd ncurses-5.7
./configure --prefix=/tools --with-shared --without-debug --without-ada --enable-overwrite

The [with-shared] option makes the software installed shared libraries.  Which sounds a bit communist.  [without-ada] excludes a type of compiler that we are never going to use.  Finally [enable-overwrite] lets us install to the common /tools/include directory instead of its own special directory.

make $CORES_TO_USE
make install
cd ..
rm -rvf ncurses-5.7

Now for BASH, the Bourne Again SHell.  This is the program that provides the flashing command prompt interface.

tar -xzvf $LFS/sources/bash-4.1.tar.gz
cd bash-4.1
./configure --prefix=/tools --without-bash-malloc

The [without-bash-malloc] option turns off Bash's ability to allocate memory itself.  Instead it will be forced to rely on the Glibc utilities, which is fine.

make $CORES_TO_USE
make tests

Well that was another fucking helpful test routine.  There is a lot of garbage and a distinct lack of PASS or FAIL.  Lets move on.

make install

Again, some programs expect you to be using a different shell program.  To fool them, make a symbolic link to Bash:

ln -vs bash /tools/bin/sh
cd ..
rm -rvf bash-4.1

Next is the bzip2 compression program, which is really simple.

tar -xzvf $LFS/sources/bzip2-1.0.5.tar.gz
cd bzip2-1.0.5
make $CORES_TO_USE
make PREFIX=/tools install
cd ..
rm -rvf bzip2-1.0.5

We are obviously going to need some basic utilities.  Stuff like copy, remove, chown etc etc.  Stuff that we use all time time in this installation, anyway.

tar -xzvf $LFS/sources/coreutils-8.4.tar.gz
cd coreutils-8.4
./configure --prefix=/tools --enable-install-program=hostname

Ordinarily the hostname program is not installed, but we want it for a test suite later, so the option [enable-install-program=hostname] makes sure it is installed.

make $CORES_TO_USE

Despite the fact that I am getting a bit disillusioned with these checks, I am going to give this a go with this package as well.  We just run the tests that work if you are root.  I think these work here, because the current lfs user has full write permission to the current directory.

make RUN_EXPENSIVE_TESTS=yes check

======================= 
All 185 tests passed 
(17 tests were not run) 
======================= 

That looks good.  Incidentally, do not flick through the book and get confused between the toolchain installation of coreutils and the actual installation, because then you will waste a good 5 minutes of your life that you will never see again chasing permissions bullshit messages.

make install

The command to become a superuser is

su

For security reasons, the installation of coreutils will not install this program, leaving us to do it manually because we do need it.  To distinguish it from the actual su program that will install happily in the chroot environment in due course, copy it as follows (giving it a different name in the process):

cp -v src/su /tools/bin/su-tools
cd ..
rm -rvf coreutils-8.4

Diffutils is a small program which reports the differences between files or directories.  It is used, amongst other things, to produce the patch files.

tar -xzvf $LFS/sources/diffutils-2.8.1.tar.gz
cd diffutils-2.8.1
./configure --prefix=/tools
make $CORES_TO_USE
make install
cd ..
rm -rvf diffutils-2.8.1

The next one is another small package – Findutils, which is used for … finding stuff, obviously.

tar -xzvf $LFS/sources/findutils-4.4.2.tar.gz
cd findutils-4.4.2
./configure --prefix=/tools
make $CORES_TO_USE
make check

As with DejaGNU I am told how many passes were expected, but not how many were actually passed.  Fantastic.  No obvious errors though.

make install
cd ..
rm -rvf findutils-4.4.2

Gawk is another programming language, that the book says is used to manipulate text files.

tar -xjvf $LFS/sources/gawk-3.1.7.tar.bz2
cd gawk-3.1.7
./configure --prefix=/tools
make $CORES_TO_USE
make check

Alarmingly, I got this message:

3 TESTS FAILED

However, looking through the report it seems that the tests failed in the section headed:

======== Starting tests that can vary based on character set or locale support ======== 

So I am going to assume that the failure are related to the fact that we are not bothering with locale support in the major packages in this toolchain.

make install
cd ..
rm -rvf gawk-3.1.7

Since we are talking about the locale support, time to install something to deal with it.  Gettext is a bit of software that seems to handle translations into local languages.  Sounds a bit like a star trek universal translator, and I bet it hisnae ivver been tae the weedge.  Anyway:

tar -xzvf $LFS/sources/gettext-0.17.tar.gz
cd gettext-0.17

This one also uses a special build folder, but it is already ready to go.

cd gettext-tools
./configure --prefix=/tools --disable-shared

Again, we do not want to share … something … at this stage, hence the option.  At this point we only want one small part of the package, so we run the make command as follows:

make $CORES_TO_USE -C gnulib-lib
make $CORES_TO_USE -C src msgfmt

The [C] option tells the make command to go into that directory ONLY when looking for compilation instructions.  We then copy the resulting file manually to install it.

cp -v src/msgfmt /tools/bin
cd ../..
rm -rvf gettext-0.17

The grep program carries out search functions.

tar -xjvf $LFS/sources/grep-2.5.4.tar.bz2
cd grep-2.5.4
./configure --prefix=/tools --disable-perl-regexp --without-included-regex

The [disable-perl-regexp] command stops an unnecessary function being added which may cause contamination from the host machine.  The [without-included-regex] forces the compile to use code from Glibc for a regular expression library instead of code that is included with grep.

make $CORES_TO_USE
make check

====================== 
All 13 tests passed 
(1 tests were not run) 
====================== 

Excellent.

make install
cd ..
rm -rvf grep-2.5.4

We already have the bzip2 software installed, now it is time to add the standard gzip.

tar -xzvf $LFS/sources/gzip-1.4.tar.gz
cd gzip-1.4
./configure --prefix=/tools
make $CORES_TO_USE
make check

Happily this told me that:

Test succeeded.

make install
cd ..
rm -rvf gzip-1.4

You may have noticed that we are building packages in alphabetical order now.  We have been doing so since Bash.  Next up is M4.  This is a bit of software that processes macros.  It is apparently used in making 'configure' scripts.

tar -xjvf $LFS/sources/m4-1.4.13.tar.bz2
cd m4-1.4.13
./configure --prefix=/tools
make $CORES_TO_USE
make check

====================== 
All 69 tests passed 
(8 tests were not run) 
====================== 

Yippee!

make install
cd ..
rm -rvf m4-1.4.13

Make is obviously an essential part of the toolchain.

tar -xjvf $LFS/sources/make-3.81.tar.bz2
cd make-3.81
./configure --prefix=/tools
make $CORES_TO_USE
make check

351 Tests in 96 Categories Complete ... No Failures :-)

Awwww, that's nice, I got a smiley!

make install
cd ..
rm -rvf make-3.81

We will also obviously need patch.

tar -xjvf $LFS/sources/patch-2.6.1.tar.bz2
cd patch-2.6.1
./configure --prefix=/tools
make $CORES_TO_USE

This produced the following error message, which was a little disconcerting:

/media/lfs/sources/patch-2.6.1/src/patch.c:1542: warning: the use of `mktemp' is dangerous, better use `mkstemp'

However, when I ran:

make check

I got:

All tests succeeded!

make install
cd ..
rm -rvf patch-2.6.1

PERL or Practical Extraction and Report Language, is another computing language to be installed.

tar -xjvf $LFS/sources/perl-5.10.1.tar.bz2
cd perl-5.10.1

For the first time in a while we need to patch the source code.  This is something to do with adapting it to the Glibc library.

patch -Np1 -i $LFS/sources/perl-5.10.1-libc-1.patch

PERL has to be different and has a special approach to the Configuration.

sh Configure -des -Dprefix=/tools -Dstatic_ext='Data/Dumper Fcntl IO POSIX'

I have no idea what the stuff in single quotes actually is, but according to the book it tells the build to only make absolutely essential parts of PERL.  Again when we come to 'make' we tell it to only build part of the whole package:

make $CORES_TO_USE perl utilities ext/Errno/pm_to_blib

Got another warning:

POSIX.c:(.text+0x5643): warning: the use of `tmpnam' is dangerous, better use `mkstemp'

This is another suggestion about using mkstemp instead of what we are using.  I also do not know what a blib is, but is sounds a bit like the noise of a dripping tap.  Probably not to do with plumbing though.  Again, because we are only building bits, we need to do the install manually:

cp -v perl pod/pod2man /tools/bin
mkdir -pv /tools/lib/perl5/5.10.1
cp -Rv lib/* /tools/lib/perl5/5.10.1
cd ..
rm -rvf perl-5.10.1

AAAAAAAhhh, Fuck, look out it is SED!  I will type this from behind the sofa.

tar -xjvf $LFS/sources/sed-4.2.1.tar.bz2
cd sed-4.2.1
./configure --prefix=/tools
make $CORES_TO_USE
make check

====================================================== 
All 65 tests behaved as expected (4 expected failures) 
======================================================

Thank fuck for that, the last thing I would want is for a problem with sed.  I am not entirely sure about the expected failures, though.  What is the point of running the test when you know it is goinf to fail?

make install
cd ..
rm -rvf sed-4.2.1

We have bzip2, we have gzip and now we are getting to the arse end of the alphabet we have tar.

tar -xjvf $LFS/sources/tar-1.22.tar.bz2
cd tar-1.22
./configure --prefix=/tools
make $CORES_TO_USE
make check

70 tests were successful. 
7 tests were skipped. 

make install
cd ..
rm -rvf tar-1.22

The final package that we install is Texinfo.  This software is all to do with the manipulation of info pages, which are supposed to be similar to man pages, but with more detail.

tar -xzvf $LFS/sources/texinfo-4.13a.tar.gz
cd texinfo-4.13a

No such directory eh?  Let me see about … Bugger.

cd texinfo-4.13
./configure --prefix=/tools
make $CORES_TO_USE
make check

I thought I was going to get:

================== 
All 0 tests passed 
================== 

Which was, obviously, just fucking brilliant, but then I scrolled the screen up and I got this instead:

=================== 
All 23 tests passed 
===================

Which is more sensible.

make install
cd ..
rm -rvf  texinfo-4.13

LAP - Toolchain, Part 5 - TCL, Expect, DejaGNU

OK, I am getting a bit fed up of bintuils and gcc by now, so I am glad to see that we are moving on to something new.  The next package to install is another language, like c or c++.  This is called Tool Command Language.  Other packages are going to use it in the future.

tar -xzvf $LFS/sources/tcl8.5.8-src.tar.gz
cd tcl8.5.8

Note, we have to use the [z] option instead of [j] because this is a gzipped file, not a bzip2'd file.  Also, note that, helpfully, when you unpack the file it unpacks to /tcl8.5.8, NOT /tcl8.5.8-src as you may have been led to believe.

If you have a look in the folder you will not find a configure script, you need to change to the unix directory first, before running what is now a standard configure/make command.

cd unix
./configure --prefix=/tools
make $CORES_TO_USE

Something we have touched on in passing so far are test suites.  These are command that you run on compiled software AFTER compilation but BEFORE installation.  They are supposed to tell you if something has gone wrong, but in my experience they give you lots of frightening looking warnings  which actually have no significance at all.  Ordinarily with the toolchain software we do not need to bother with running tests – other than the careful tests that GCC is working properly.  Still, we want to be thorough so lets do one now:

TZ=UTC make test

The [TZ=UTC] bit means Timezone=Universal Central Time.  The output of all that makes no sense to me whatsoever, but there do not appear to be any ERROR ERROR MUST KILL HUMAN RACE type messages.  Now just install the compiled programs:

make install

The software that we are installing comes with stuff called 'debugging symbols' included.  I am not entirely sure what these are, but in my mind they are little signs that programs hold up when they have turned pear shaped saying things like 'ooops' and 'my bad', and 'I shouldn't have done that should I?'.  These are not necessary, because I have no fucking intention of debugging any thing.  Later on we will be removing these debugging symbols to free up space.  We need to write enable one of the files we just installed so that we can change it later on.

chmod -v u+w /tools/lib/libtcl8.5.so

That [ch]anges the [mod]e of the file by allowing the current [u]ser to [w]rite to it.  Next we need to install Tcl's private headers.  I am not entirely sure what a private header is, but it probably involves a sealed room and a prostitute.

make install-private-headers

The program we have installed has its version number attached to it, which means some other software may not be able to find it, so lets just symbolic link away the problem, and then clean up.

ln -sv tclsh8.5 /tools/bin/tclsh
cd ../..
rm -rvf tcl8.5.8

Next up is Expect.  This is an extension to the TCL package we just installed.

tar -xzvf $LFS/sources/expect-5.43.0.tar.gz
cd expect-5.43

Patch away a bug that causes problems with GCC, and some sort of other bug that the book is curiously reticent about.

patch -Np1 -i $LFS/sources/expect-5.43.0-spawn-1.patch
patch -Np1 -i $LFS/sources/expect-5.43.0-tcl_8.5.5_fix-1.patch

Warning, more sed crap coming up!  Again, we edit a file (the configure script) to force it not to use the host system stuff, and then run the configure script.

cp -v configure{,.orig}
sed 's:/usr/local/bin:/bin:' configure.orig > configure
./configure --prefix=/tools --with-tcl=/tools/lib --with-tclinclude=/tools/include --with-x=no

The first two options make sure that this extension to TCL knows about the TCL we have just installed.  The last option tells it that we have not yet installed the 'X' window system.

make
make test

I've not bothered with the $CORES setting for this, as it takes a whopping 7 seconds to run on my 1.6Ghz celeron single threaded. I ended up with:

Files with failing tests: logfile.test send.test spawn.test

I have done everything right so far, so I am going to ignore that.  And according to the book failures are not surprising and not critical.  So why did we run the tests again?  Anyway, when we install the software we tell it not to install any scripts:

make SCRIPTS="" install
cd ..
rm -rvf expect-5.43 

Next is DejaGNU.  This is a package based on TCL and Expect which assist with running tests.

tar -xzvf $LFS/sources/dejagnu-1.4.4.tar.gz
cd dejagnu-1.4.4
./configure --prefix=/tools
make install
make check

The output of the check is unclear, it tells me that it expected 62 tests to pass, but brilliantly fails to tell me how many actually passed.  That's it though, just the clear up to go.

cd ..
rm -rvf dejagnu-1.4.4

LAP - Toolchain, Part 4 - Rebuild

It is now time to go back to binutils.  Last time we built this we used the compilers and other software from Ubuntu.  In particular the software that we installed in the build-essential package.  It is now time to build binutils again, but this time with the new software that we just made.

tar -jxvf $LFS/sources/binutils-2.20.tar.bz2
mkdir -v binutils-build
cd binutils-build
CC="$LFS_TGT-gcc -B/tools/lib/" AR=$LFS_TGT-ar RANLIB=$LFS_TGT-ranlib ../binutils-2.20/configure --prefix=/tools --disable-nls --with-lib-path=/tools/lib

The [CC="$LFS_TGT-gcc -B/tools/lib/" AR=$LFS_TGT-ar RANLIB=$LFS_TGT-ranlib] bit is specifically telling the program to compile using our new tools and not the ones on the host system.  We have already seen the [prefix] and [disable-nls] options.  The last option again forces the compiler to use the new stuff and now the host programs.

make $CORES_TO_USE
make install

We also now need to do a bit of work to prepare for the next phase.  Essentially we are preparing to reverse the step where we pointed to the /tools directory for all compiling operations.  To do so we will need a new version of some of the binutils software.  We could uncompress the archive and make the software at that stage, but it is easier to just do it just now.

make -C ld clean

This cleans out the directory ld so we can:

make -C ld LIB_PATH=/usr/lib:/lib

This rebuilds the ld program, pointing at the normal /usr and /lib locations that will finally be used rather than the temporary /tools.

cp -v ld/ld-new /tools/bin

This just [c]o[p]ys the file we just made to its destination.  Do not worry that it is in /tools, we will sort that all out in due course.

Lets clean up again.

cd ..
rm -rfv binutils-2.20 binutils-build

Now, lets do the same to GCC.

tar -jxvf $LFS/sources/gcc-4.4.3.tar.bz2
cd gcc-4.4.3

For the first time we need to patch the software before we can use it.  The reason for this is that this software does not expect to be used in the way the the Linux From Scratch install uses it.  So we need to literally reprogram it.  Helpfully someone has put together a list of the changes that need to be made, and we apply them by running this command:

patch -Np1 -i $LFS/sources/gcc-4.4.3-startfiles_fix-1.patch

This runs the [patch] command using the [i]nput file [cdrom/sources/gcc-4.4.3-startfiles_fix-1.patch].  The option [N] means that it will skip over patches that appear to have already been applied.  The [p1] bit affects how filenames in the files to be patched are treated.  Effectively it lops off part of the path of each filename.  Why this is necessary is beyond me.  The output of that command looked like this to me:

patching file gcc/gcc.c 
Hunk #1 succeeded at 6467 (offset 97 lines).

Part of the compilation of GCC calls a script to fix broken parts of GCC or Glibc.  The authors of Linux From Scratch are confident that this is not needed, and indeed worried that if it is run it may bring in some material from the host system.  The script is disabled as follows:

cp -v gcc/Makefile.in{,.orig}
sed 's@\./fixinc\.sh@-c true@' gcc/Makefile.in.orig > gcc/Makefile.in

I am beginning to see a theme developing here.  If the 'sed' command is used it is usually followed by some bizarre string of characters that seem to magically achieve the desired effect.  The first command here just makes a backup of the file [Makefile.in] before the 'sed' weirdness is let loose on it.  Only sensible if you ask me.

In a similar vein to the last one, we need to force some behaviour that would ordinarily not happen.  This is something to do with bootstrapping, which sounds like a punishment in a nineteenth century boys school.

cp -v gcc/Makefile.in{,.tmp}
sed 's/^T_CFLAGS =$/& -fomit-frame-pointer/' gcc/Makefile.in.tmp > gcc/Makefile.in

There's that fucking sed again.  Next we need to, again, ensure that we are building using the new tools and not anything on the host system.  This involves editing lots of text files, which would be boring, so lets do it automatically.

for file in $(find gcc/config -name linux64.h -o -name linux.h -o -name sysv4.h)
do
  cp -uv $file{,.orig}
  sed -e 's@/lib\(64\)\?\(32\)\?/ld@/tools&@g' -e 's@/usr@/tools@g' $file.orig > $file
  echo '
#undef STANDARD_INCLUDE_DIR
#define STANDARD_INCLUDE_DIR 0
#define STANDARD_STARTFILE_PREFIX_1 ""
#define STANDARD_STARTFILE_PREFIX_2 ""' >> $file
  touch $file.orig
done

Roughly speaking this sets up a loop (the bit between the do and done) for every file found by the bit in brackets in the first line.  Because this involves more sed insanity I refuse to try and work out what else is going on.  When I run it, it seems to edit about 20 or so files.  This is again a command that you paste into the terminal in one go.

We need the maths libraries again, so unpack them once more:

tar -jxf $LFS/sources/mpfr-2.4.2.tar.bz2
mv -v mpfr-2.4.2 mpfr
tar -jxf $LFS/sources/gmp-5.0.0.tar.bz2
mv -v gmp-5.0.0 gmp

After we have moved to the temp build directory, it is time to configure the build:

mkdir -v ../gcc-build
cd ../gcc-build
CC="$LFS_TGT-gcc -B/tools/lib/" AR=$LFS_TGT-ar RANLIB=$LFS_TGT-ranlib ../gcc-4.4.3/configure --prefix=/tools  --with-local-prefix=/tools --enable-clocale=gnu  --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-languages=c,c++ --disable-libstdcxx-pch --disable-multilib --disable-bootstrap

We have now seen most of these before.  The [with-local-prefix] option is not documented in either the book or the configure --help information.  It obviously seems to be connected to the new software.  [enable-clocale=gnu] sets some locale to make sure that it does not speak german.  This time we are enabling the [shared] option, which makes about as much sense to me as disabling it – but hey ho.  The [enable-threads=posix] configures gcc for multi-threading so you can take advantage of multicore cpus.  The [ enable-__cxa_atexit] option is explained as follows in the book:

“This option allows use of __cxa_atexit, rather than atexit, to register C++ destructors for local statics and global objects.”

Make sense to you?  Makes no sense to me, but Destructors sound cool.  Do they fight Autobots?  The [enable-languages=c,c++] makes sense – we now need more language support because we are building more than Glibc with this compiler.  [disable-libstdcxx-pch] stops the compilation of something called a pre-compiled header for libstdc++, because according to the book, we have no use for it.  Lastly, the [disable-bootstrap] option stops it assaulting schoolboys.  Wait, what?  Oh, right apparently it prevents the build doing time consuming checks.

make $CORES_TO_USE
make install

The program we have just made is called gcc, but some software looks for a program called cc, so lets make a quick symbolic link:

ln -vs gcc /tools/bin/cc

To test the compiler run the test commands again:

echo 'main(){}' > dummy.c
cc dummy.c
readelf -l a.out | grep ': /tools'

The output should be the same:

[Requesting program interpreter: /tools/lib/ld-linux.so.2]

Assuming that works fine, just delete the temporary files:

rm -v dummy.c a.out

Clean up the source code files:

cd ..
rm -rvf gcc-4.4.3 gcc-build

Now, onto some new packages.

LAP - Toolchain, Part 3 - The Switch

The next stage in the procedure is to use the programs and libraries that we just installed to rebuild themselves to 100% guarantee that nothing from the host system has contaminated the installation.  We need to tell the software we have just installed to use itself now, and not anything on the host machine.  To achieve this we run the following commands:

SPECS=`dirname $($LFS_TGT-gcc -print-libgcc-file-name)`/specs
$LFS_TGT-gcc -dumpspecs | sed -e 's@/lib\(64\)\?/ld@/tools&@g' -e "/^\*cpp:$/{n;s,$, -isystem /tools/include,}" > $SPECS 
echo "New specs file is: $SPECS"
unset SPECS

I have literally no idea what a SPECS is, but I suspect it has nothing to do with lenses.  You can see in there some mention of 'tools' which is I guess what we are trying to achieve.  The 'echo' command produces this result:

New specs file is: /media/lfs/tools/bin/../lib/gcc/i686-lfs-linux-gnu/4.4.3/specs

Which sounds promising.  As suggested by the book I visually inspected that file.  It made my head hurt, and I could briefly only see in the colour magenta.  So I stopped looking at it.

It is strongly recommended that we try some tests before proceeding.  Well, OK if you insist.

echo 'main(){}' > dummy.c

This command just [echo]s the text in single quotes to the file dummy.c.  The .c extension indicates that the file contains 'C' source code.  Next we compile that code:

$LFS_TGT-gcc -B/tools/lib dummy.c

We know that the $LFS_TGT part of that resolves into i686-lfs-linux-gnu, and if you look in the /tools/bin directory you will find a file called i686-lfs-linux-gnu-gcc, so this command just executes that program.  The [-B] bit adds the following directory to the search path, which means that the gcc gets to use the Glibc libraries we installed.  I think.  So this command means compile the dummy.c program using the Glibc libraries.  The command produces a file called a.out.  We then search that command for references to the /tools directory to make sure it was used:

readelf -l a.out | grep ': /tools'

We should be told that:

[Requesting program interpreter: /tools/lib/ld-linux.so.2]

Which is exactly what we want to see.  Lets get rid of the test files:

rm -v dummy.c a.out

And we now move on to rebuild the tools we just made, this time by using themselves.  It's a bit like the snake eating its tail, but in reverse.

LAP - Toolchain, Part 2 - All about C

The next bit of software to install is GCC. This is the Gnu Compiler Collection. Essentially it is the software that we are going to use to compile all of the rest of the Linux From Scratch software. At the moment we are using the Compiler that came with Ubuntu.

This file is a biggie - perhaps the biggest we are going to compile. It is going to unpack to over 500Mb. Now this works with a 2Gb machine, but you are going to need to keep the linux memory cache in hand. The memory cache is a facility where data that linux thinks you are going to use often is loaded into RAM. This is fine most of the time and it makes your system run nicely. It is not fine when we are trying to install GCC. Irritatingly it will not automatically dump the cached data to make room for the rapidly expanding GCC folder. So, we need to clear it for it, and we do that as follows:

sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches

Keep an eye on the system - you may need to run that more than once. In fact it would probably be a good idea to get the system to repeat this command every so often. There does not appear to be a way to get it to do this by itself, so lets make a script.

cat > ~/repeat.sh << "EOF"
#!/bin/sh
# usage: repeat [x] 
while true ; do $2 ; sleep $1 ; done
EOF
chmod +x ~/repeat.sh

Now we should be able to run:

~/repeat.sh 60 "sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches"

To get it to clear the cache every minute. But of course we can't do that because the script we just made shits itself when it gets to the && bit and ignores what comes next. Brilliant. So we have to script the clear cache command AS WELL:

cat > ~/clearc.sh << "EOF"
sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches
EOF
chmod +x ~/clearc.sh

We can actually now run:

~/repeat.sh 60 ~/clearc.sh

That should give us as much free memory as possible during the compilation. If you are using a 4Gb machine, this should not worry you at all. On my 2Gb machine, I ended up with a little over 700Mb free after the build had compiled. So, the Ramdisk option is not feasible for a 1Gb machine. We unpack it in the same way as before:

tar -jxvf $LFS/sources/gcc-4.4.3.tar.bz2
cd gcc-4.4.3

There are two other bits of software that we need to unpack at the same time. These are GMP and MPFR, both of which relate to some sort of complicated mathematical function. When we are building this stage of the Toolchain, we can just uncompress them into the GCC folder, and then rename them. The compilation of GCC will then pick them up.

tar -jxf $LFS/sources/mpfr-2.4.2.tar.bz2
mv -v mpfr-2.4.2 mpfr
tar -jxf $LFS/sources/gmp-5.0.0.tar.bz2
mv -v gmp-5.0.0 gmp

Again, GCC uses a separate build directory, so we need to create it. Note that as we are currently in the GCC directory, we need to insert a '../' into the command, unlike the same command for binutils.

mkdir -v ../gcc-build
cd ../gcc-build

Now to configure the package:

../gcc-4.4.3/configure --target=$LFS_TGT --prefix=/tools --disable-nls --disable-shared --disable-multilib --disable-decimal-float --disable-threads --disable-libmudflap --disable-libssp --disable-libgomp --enable-languages=c

The first three options we have seen before. The [disable-shared] option is described by the LFS as:

“This switch forces GCC to link its internal libraries statically. We do this to avoid possible issues with the host system.”

I have no idea what that means. The [disable-multilib] command is actually described as harmless. The [--disable-decimal-float --disable-threads --disable-libmudflap --disable-libssp --disable-libgomp] are not necessary at this stage because we are only going to use this version of the GCC program to compile Glibc and then we will make another one. Lastly we only need to compile in the 'C' language now, so the last option chooses that for us. Again, in a staggering turn of events I had no problems running the script.

make $CORES_TO_USE

This will take a while – perhaps four times as long as binutils.

make install

According to the book, because we disabled the shared thingummy, something didn't get made that should have been made. As a result, and to get the Glibc software to work, we need to run this absolute HORROR of a command. I understand it up to the '-'.

ln -vs libgcc.a `$LFS_TGT-gcc -print-libgcc-file-name | sed 's/libgcc/&_eh/'`

The output of the command is:

`/media/lfs/tools/bin/../lib/gcc/i686-lfs-linux-gnu/4.4.3/libgcc_eh.a' -> `libgcc.a'

This seems to suggest that there is a link so that anything looking for the file libgcc.a file gets the libgcc_eh.a file instead. Fair enough, but how the fuck we get here from the command with its 'sed's and '&'s I have no earthly idea. Magic, I suppose. Now we need to delete these directories again:

cd ..
rm -rvf gcc-4.4.3 gcc-build

Now the next bit is a bit tricky. It looks like we are going to install the linux kernel, because we decompress the kernel package. But at the last minute we do something small and basic instead. Which is a puzzle. The puzzle is not solved by the books explanation:

“The Linux kernel needs to expose an Application Programming Interface (API) for the system's C library (Glibc in LFS) to use. This is done by way of sanitizing various C header files that are shipped in the Linux kernel source tarball.”

Holy fuck. I have sanitized my header before, but usually with some sort of medicated shampoo. I have no fucking clue what this is all about. Lets just get this over and done with as quickly as possible.

tar -jxvf $LFS/sources/linux-2.6.32.8.tar.bz2
cd linux-2.6.32.8
make mrproper

'Make Mr Proper do what?', I wonder casually.

make headers_check

This runs some sort of test. When I ran it, it complained about scsi with this error:

userspace cannot call function or variable defined in the kernel

I am not overly concerned about this, because I don't have any scsi disks. The next command installs these headers we have been hearing about to the 'dest' directory.

make INSTALL_HDR_PATH=dest headers_install

From there we copy them to the Toolchain manually. We cannot install them to the Toolchain, because apparently they would wipe it out.

cp -rv dest/include/* /tools/include

We should delete the unpacked files now, because while we will need to use them again to actually build the kernel, a) that won't be for ages, and b) they are fucking huge.

cd ..
rm -rvf linux-2.6.32.8

The next one is a biggie. It is Glibc. This package includes programs and libraries for running 'C' compiled programs. So GCC lets you compile programs in the 'C' language, and Glibc lets you actually run them.

tar -jxvf $LFS/sources/glibc-2.11.1.tar.bz2
mkdir -v glibc-build
cd glibc-build

Now we have a tricky bit here. We need to tell Glibc to compile for our i686 architecture. It is a bit huffy about this apparently, and we need to tell it to compile for i486. We could edit lots of text files to change this setting, but to make it easy on ourselves the following command does it automatically:

case `uname -m` in  i?86) echo "CFLAGS += -march=i486 -mtune=native" > configparms ;;esac

That all makes sense to someone, somewhere, whose day mostly consists of trying to get to the buckles at the back of their nice new jacket.

../glibc-2.11.1/configure --prefix=/tools --host=$LFS_TGT --build=$(../glibc-2.11.1/scripts/config.guess) --disable-profile --enable-add-ons --enable-kernel=2.6.18 --with-headers=/tools/include libc_cv_forced_unwind=yes libc_cv_c_cleanup=yes

Riiiiiiiight. Lets roll up our sleeves shall we. First option, fine. This time we have host and not target = $LFS_TGT. Fair enough. The [--build=] nonsense is apparently to ensure that the software uses the GCC compiler that we just installed in /tools. [disable-profile] disables profiles which we do not need unless, according to the book, we do need, in which case we should not disable it. Brilliant. That's clear then. [enable-add-ons] tells it to use NTPL as its threading library. I am not clear on what NTPL is, or what a threading library is, and I am still a bit confused about the profiles one, so we'll skip over that. [enable-kernel=2.6.18] is more straightforwards. We are using a kernel with a version later than that, and presumably there was a significant kernel change around version 2.6.18 so we are just telling Glibc that we are up to date. [with-headers=/tools/include] refers to the headers that we installed from the kernel package and points in their direction. [libc_cv_forced_unwind=yes] and [libc_cv_c_cleanup=yes] stop it running tests that we know will fail at this stage. Clear? Good.

make $CORES_TO_USE
make install

Lets clean up.

cd ..
rm -rvf glibc-build glibc-2.11.1

We now have the first part of our Toolchain completed – the basic utils, the compiler and the libraries needed to run C programs. Good stuff. What next?

LAP - Toolchain, Part 1 - Binutils

As a reminder - we have just logged in as the lfs user.  Let's change into the directory we are going to use to unpack and build the source code. Remember we are using /tmp as a ramdisk:

cd /tmp

If you have less than 2G of RAM you may want to use the Amiga Key itself:

cd $LFS/sources

Note, we can now use the $LFS variable instead of typing out /media/lfs every time.

We start compilation with Binutils.  These are a selection of very basic utilities that I will never use directly, but support other programs (and therefore need to be installed first.  First of all uncompress the file:

tar -xjvf $LFS/sources/binutils-2.20.tar.bz2

Binutils needs its own working directory so we make it as follows:

mkdir -v binutils-build
cd binutils-build

And finally run the configure script with this command:

../binutils-2.20/configure --target=$LFS_TGT --prefix=/tools --disable-nls --disable-werror

First we go down one directory [..] then into the [binutils-2.20] folder where we run the script.  The [target] option uses the $LFS_TGT variable we set in the .bashrc script.  It is no less scarey now, so lets just pass over it.  The [prefix] option set the install path to the /tools directory.  As we know, because of the link we have made that actually points to /media/lfs/tools.  The [disable-nls] stops some localisation stuff from being built, which is not needed at this point.  Lastly, [disable-werror] prevents the compilation stopping if it encounters an error.

In a fucking fantastic turn of events, I got precisely NO ERRORS when I ran the script.  Amazing.

To compile the script, run:

time make $CORES_TO_USE

The time command will tell me how long this took, which is a useful test because the compilation times of all the other packages are measured in terms of this one.  It's time to get a beer, because this is going to take a while.  On my 1.6Ghz Celeron it took 9 minutes 1.6 seconds (which was reduced to 5 minutes and 13 seconds when I ran it with -j2 from the /tmp ramdisk).  On my 2.66Ghz i5 it took 2 minutes 58 seconds, or 1m 2 seconds with j4.  You then install the new programs you just compiled by running:

make install

We are now off and running!  If you look in the /tools directory now, you will see that it is starting to be populated.  Because we are installing onto a small partition we do not have space to keep the source code around, so bye bye the directories:

cd ..
rm -rvf binutils-2.20 binutils-build

This [r]e[m]oves the directories.  This [r]emoves everything in the directories, [v]erifies that it has done it, and [f]orces it to just fucking do it instead of checking with me.  Do not make mistakes with this command.  You will regret it.

Next!