The Installation Backend: FAI

We have designed our Linux system so that it is easy for volunteers to install. As such, you don't need to know very much about the installation process in order to get Linux on a computer.

Having said that, understanding the machinery behind the install can be helpful. If nothing else, it will help you troubleshoot the installation process when installs fail. Learning about the installation process helps the Linux project, too: it helps volunteers understand Linux better, and it gives the project security in case the current administrators stop volunteering.

The next section introduces FAI, and documents some resources that can help you understand the package. Next, we explain the installation process - from bootup to configuration - in some detail.

FAI overview

The software we use for our backend is called FAI (Fully Automated Install). It is a set of scripts that allows Linux to be installed on many workstations at once, without human intervention. FAI can be customized to suit local needs, and we have customized FAI to install our subset of Debian automatically.

In our project, FAI scripts are responsible for:

• Partitioning and formatting hard drives.

• Installing Debian packages.

• Carrying out some customizations specific to our project. For example, we copy configuration files to make the local window manager look like Windows 95.

We do not yet exploit the full functionality of FAI. One problem is that we face many different hardware configurations. We have not figured out how to automatically detect and select packages specific to hardware, monitors and mice, so we rely on volunteers to configure X. Similarly, volunteers must create user accounts on the system. As our Linux installation matures, we may be able to automate some of these things as well.

FAI resources

The homepage for FAI is http://www.informaik.uni-koeln.de/fai . This page offers the latest version of the FAI manual, and archives of the FAI mailing list.

The FAI manual is also installed locally on the server. You can find it at

/usr/share/doc/fai/

Working through the manual and setting up FAI is the best way to understand the system. Failing that, working through the manual and comparing that to the files on our server is good too.

FAI tips

• All configuration files need to be world-readable, and all directories need to be world-executable. Otherwise FAI will not read the configuration files, and installs will fail for strange reasons.

  This problem can probably be fixed by specifying proper ownership privileges, but we don't know how to do that yet.

FAI logfiles

FAI logs configuration information and runtime messages into a directory full of logfiles. During the installation process, you can access these logfiles in the directory

/tmp/fai/

After installation, you can find the logfiles in the directory

/var/log/fai/hostname/last-install/

on the client machine, where hostname is the name of the machine.

Log files of interest include:

• rcS.log , which logs all the messages that appear on screen after the kernel has booted. If you are wondering whether a step of the FAI installation failed, this is the first place to look.

• dmesg.log , which logs all the information printed out during the booting process. This is the place to look for information about drivers detected by the system, such as network cards.

• FAI_CLASSES , which shows all the classes defined for this installation. This is the first place to check if something was mysteriously misconfigured or not installed - often the required class was not defined.

• hda_sfdisk , which shows how the first IDE drive was partitioned.

Startup

The boot process of our FAI installation goes through the following steps:

1. The BIOS finds LILO on the boot floppy.

2. LILO loads a Linux kernel from the floppy.

3. LILO connects to the network and NFS-mounts a Linux filesystem that sits on the server. This filesystem contains the FAI installer, which begins execution.

4. The installer NFS-mounts the FAI configuration files off the server. (Check this -P.)

After these steps, the FAI installer is ready to partition the local hard disks, mount local hard drives and install program files.

The following subsections explain the startup process in more detail.

BIOS and the boot loader

The BIOS of a computer contains basic information the computer needs to interact with its components. Part of the BIOS's job is to look for boot information for the computer's operating system.

The BIOS is not very smart. It knows some locations where the boot information may exist - the boot sequence - and it knows how to load enough information from those locations to get the rest of the boot loader going.

Most BIOSes allow you to boot from the floppy or the hard drive. Some allow booting from CD-ROMs, and a few allow direct booting off a network, using a network card.

Our installation uses a bootable floppy for installation. The first few sectors of the floppy contain LILO, a boot loader for Linux. The BIOS finds LILO, loads it into memory and executes it.

LILO

LILO stands for LInux LOader. Its job is to tell the computer where to find Linux components. LILO can also be used to boot other operating systems (such as Windows 95), but we do not use that functionality in FAI.

In order to boot Linux, LILO needs two pieces of information:

Kernel location: The Linux kernel is the core of the operating system. It is a program that serves as an interface between the computer's hardware and software, managing system resources, allowing access to hardware, and interacting with application programs.

There may be several different versions of the Linux kernel on a computer. LILO needs to know which one to use, and where to find it. In our installations, the kernel is located on the boot floppy disk.

Root location: The root directory is the top of the directory structure. The directory structure contains all of the user programs and data present on the system. The directory structure may be spread over many hard drives, but it has a single root directory. (The hard drive partition that contains the root directory is called the root partition.)

LILO needs to know the location of this root directory, so that it can mount the directory tree. Other filesystems (on hard drives, CD-ROMS or the network) can then be mounted relative to the root.

On standalone Linux systems, the root directory is often located on the same partition as the kernel. However, this is not required, and in our installation the root directory is located on the server. It is accessible via a protocol called NFS. LILO is smart enough to know about NFS, and it is able to tell the kernel where to find the root directory that should be used for booting.

On our server, the NFS root directory is:

/usr/local/share/fai/nfsroot/

Permission for the server to export this directory is contained in the /etc/exports file.

(Where should I be talking about NFS? Here, or elsewhere? -P.)

rcS_fai

Traditionally, when Linux boots it executes startup scripts. These scripts start services, such as X-Windows and the printing daemon. On Debian, these scripts can be found in /etc/init.d/ .

To start the install, FAI includes a script in the /etc/init.d directory called rcS_fai . The Linux kernel finds this script (or rather, a symbolic link to this file) and starts executing it.

rcS_fai contains the control code for FAI. It is the installer program; its job is to co-ordinate the installation process, calling subroutines that carry out each stage of the install.

Installation

Once the installer has started, installation proceeds in the following steps:

1. Classes are defined.

2. Local hard drives are partitioned and formatted.

3. Packages get installed and configured.

4. Local configuration changes are made.

5. GRUB is installed onto the local hard drive.

The FAI manual goes through each of these steps in detail. Over the next few sections, we will describe the basic ideas, and document local changes we made for our project.

FAI classes

FAI has the capability to install many different Linux configurations at the same time. Different configurations are specified by classes - labels that specify which configuration files FAI should use in the installation.

Each installation starts out with three classes defined: DEFAULT, LAST and a class labelled with the client's hostname.

Once an installation starts, one of the installer's first jobs is to figure out what additional classes (if any) should be defined. To do this, FAI looks in

/usr/local/share/fai/installer/classes/ .

Three different kinds of files are allowed in this directory:

• Perl or shell scripts, with names that match the regular expression 'S[0-9]*.sh,pl,source' . Each of these scripts is executed in alphabetical order. FAI considers the output of these scripts to be a new class name.

  This would be the natural way to define classes for based upon detected hardware. We currently use it to define classes based on hostname (S01alias.sh), to decide disk partitions (S07disk.pl) and to run configuration menus (S30menu.sh, S40xserver.sh).

• Files with names matching an existing class. Each word (token) in these files is considered to be a new class name. However, the contents of this file will not be executed.

• Files of the form classname.var . These files are scripts that are executed if and only if classname is a defined class.

  We use the pre-existing file DEFAULT.var for our installations, but do not define other variable files.

The installer executes any scripts matching the name of an existing class. This may define new classes, so the process iterates until no more classes have been defined. Finally, scripts of the form classname.var are executed.

We explain relevant classes in the parts of the installation where they are used. Hostnames in the classes/ directory which directly pertain to class definition are:

wc_custom:

This hostname does not call any of the standard packages or classes. It is meant as a testing hostname, so that administrators can fiddle around with new features while other people are doing standard installations.

wc_191 ... wc_198:

Each one of our standard installation FAI floppies is given one of these names. Hosts with these names use the standard disk partitioning and formatting, and choose the amount of software to install based on the local hard drive size.

The names themselves are defined in S01alias.sh .

Partitioning Hard Drives

In this phase, the installer looks in /usr/local/share/fai/installer/disk_config for partitioning information. It should partition the local hard drive(s) according to the information specified in exactly one of these files, which means exactly one class should match a filename in this directory.

FAI partitions the local hard drive(s) according to the information it finds in this file. It also formats the partitions appropriately.

One big advantage of these files is that they are robust with respect to hard drive sizes. They can size partitions proportionally according to the size of the hard drive.

(What happens if there are multiple matches? -P.)

The format of these files is described in the FAI manual. We have defined the following classes:

TWCN_DISK:

This configures a single IDE drive. It wipes out the entire drive, then creates three partitions:

• A swap partition, which ranges between 32-64MB depending on the hard drive size. (FAI determines the size of the swap partition by looking at the size of the drive.)

• A root partition, which is at least 300 MB, and can expand to the entire disk.

• A boot partition called /fai-boot, which is 2MB big.

As well, this file specifies that all partitions should be checked for errors as they are formatted. (This is the “-c” option.)

TWCN_DISK_SMALLSWAP:

Like TWCN_DISK, except that the swap size is fixed at 32MB.

TWCN_DISK_BIGSWAP:

Like TWCN_DISK, except that the swap size is fixed at 64MB.

TWCN_DISK_TEMP:

Like TWCN_DISK, except that hard drives are not checked for errors. For new installs, you should not use this class. Checking for hard drive errors adds time to the install, but it saves a lot of headaches later on.

Other classes we should add include:

• A class to deal with SCSI drives.

• A class which preserves Windows partitions. This would allow us to dual boot our installations.

• A class that deals with multiple hard drives.

After the hard drives have been partitioned and formatted, they are mounted. In Linux, hard drives must be mounted before the system can access their data.

The root partition is mounted to /tmp/target/ . Any other partitions are mounted relative to this point. For example, if we create a partition that should end up at /usr/local/ , the installer will mount this partition to /tmp/target/usr/local/ .

Base package installation

A certain subset of the Debian distribution is labelled “required”, and another subset is labelled “important”. These packages make up a barebones Debian installation.

A program called debootstrap is responsible for downloading these packages and unpacking them into /tmp/target/ .

The packages that debootstrap installs are listed in /usr/lib/debootstrap/scripts/ on the server.

Package installation

Once the local hard drives have been prepared, package installation begins. The installer NFS-mounts the local mirror on the server. It then downloads and installs packages according to the classes defined in

/usr/local/share/fai/installer/package_config/

The contents of every file matching a defined class name are used.

Again, the format of these configuration files is described in the FAI manual.

(Where can readers find out about Debian packaging policies? -P.)

Below, we document some of the more notable package configuration classes we use. This list is undoubtedly incomplete:

DEFAULT:

We have changed this file significantly from the version shipped with FAI. We consider the default installation to include very few packages: basically just a few useful utilities not provided by UNIXLIKE:

• The printing utility lprng

• The text-based web browser w3m

• bzip2 and zip and unzip, which are used for file compression.

• sudo , which allows regular users to execute superuser commands safely. (We don't use this at all.. -P.)

(Is this it? -P.)

X11:

This package contains X-Window functionality; all the packages needed for graphical mode except the X-server packages.

UNIXLIKE:

This includes almost all packages in the “standard” Debian distribution. We want this because FAI will (correctly) not install these packages by default.

These packages provide most of the functionality that one would find on a standard Linux or UNIX system, with the following exceptions:

• The text editor Emacs, provided by the EMACS package.

• Development packages, which are used to compile programs. These are provided by the DEVEL package.

• A few utilities that are annoying:
  • gpm, which controls mice in text terminals. This package is a huge pain because it interferes with regular mouse operation.

  • lpr, which is an ancient UNIX printing system. We use lprng instead.

  • mtools, which are a means of accessing files on floppy drives. These are ordinarily very useful tools, but because we automatically mount the floppy drive using the Supermount protocol, it is not safe to use mtools on our systems. (see Section (where? -P.) for more information).

• TEX packages. TEX and L^ATEX are powerful typesetting systems that take up a lot of disk space.

WINLIKE:

This package contains applications that make Linux look like Windows. Here is an incomplete list of some of the things it includes:

• Productivity applications: abiword, opera, and gmc .

• The window manager icewm, and gdm (which provides a graphical login).

• Games: the ace-of-penguin suite of card games, the arcade games koules, xkobo and xbubble, and a few other small puzzle games.

• Smaller applications: xpaint, eeyes (an image viewer), sylpheed (a mail reader), gaim (ICQ/AIM/MSN client)...

• Printing utilities printop and lprngtool, and the modem dialer gkdial

These packages are graphical. They have a lot of dependencies.

DEFAULT_ORIG:

This is the DEFAULT class that ships with FAI. We do not use it.

EMACS:

emacs is a super powerful text editor. However, it is large and thus optional.

DEVEL:

The development tools (gcc and its friends) that we removed from the default package installation.

SGML:

SGML development. No installation uses this yet.

TEX:

TEX and L^ATEX development, which is removed from the default installation.

GNUMERIC:

The gnumeric spreadsheet. Originally we included this spreadsheet by default, and it is installed in STANDARD installations, but we took it out of WINLIKE to save space on systems with smaller hard drives, and because it appears that many people do not use spreadsheets.

SCRIBUS:

A rudimentary desktop publishing program. It is more cute than functional now, but is useful for making posters.

ABIWORD_EXTRAS:

The abiword word processor has some extra components not necessary for operation, such as clip art. We take these packages out for smaller installations.

SIAG_OFFICE:

Currently only used by the TINY installation, this is an alternative office suite. It consists of a word processor, a spreadsheet and an animation program.

PKG_MANAGE:

Some bigger but useful utilities for managing Debian packages. aptitude is a useful front-end command-line apt-get utilities. deborphan is used to discover packages that can be safely removed from the system.

This package is obsolete, because we now include aptitude in the DEFAULT package.

Packages we should add include:

• Possibly a package for music, or for web development, or for the GIMP and its friends - whatever is popular with users.

FAI's package configuration system is powerful and useful, but it has its quirks:

• FAI automatically satisfies dependencies of all the packages specified in the package configuration files. However, it does not satisfy “recommended” dependencies. You must satisfy them explicitly.

• The “remove” command will fail if the package in question has not been installed.

• It used to be the case that if FAI could not find a package it needed, the installation process would halt. Thankfully, this behaviour has been changed, and FAI will simply print a warning:

  These unknown packages are removed from the installation list:

  Unfortunately, when package dependencies are wrong FAI can still get confused, and continue the installation process without installing important packages. See Appendix B.2.1 to learn more about this problem.

  It is a good idea to check the /tmp/fai/software.log file on the client before rebooting the system, to ensure that all packages installed correctly.

• FAI will by default install all packages with priority “Required” and “Important”, but will install no packages with priority “Extra”, which includes such useful utilities as less .

  To get around this, we had to manually install Debian on a machine and create a package configuration file with the standard packages. To see how to do this, see Section 8.4.

  (Wrong formatting -P.)

Configuration

In order to make our setup look like Windows, we have to tweak some configuration files. You can find the code that executes these tweaks in the installer/hooks and installer/scripts directories.

To learn about specific tweaks we make to our applications, see Section 8.1.

Finishing up

The LAST class triggers operations that clean up the Linux installation.

The GRUB and BOOT classes together trigger the BOOT script, which makes the computer bootable. The kernel to install is specified in installer/class/DEFAULT.var . The corresponding kernel package is located in installer/files/packages/