Reordering the Debian boot sequence for correctness and speed

There are subtle bugs in the Debian boot and shutdown sequence. They are hard to find, as they normally only affect rare combination of packages, and harder to fix, as they normally require the combined work of several maintainers and changes in several packages. This talk is about the release goal for Lenny to solve them, and gain a few advantages on the way.

Outline

SysV init in Debian - Booting

Note, this is the stuff going on after the initrd part is done. The very early boot is done before hard drive partitions is mounted.

1. /sbin/init start, which look at /etc/inittab and decides what to do.
2. The scripts in /etc/rcS.d/ is executed in sequence by /etc/init.d/rc, with start as the argument.
3. Depending on the runlevel, the scripts for the given runlevel is executed (normally the ones in /etc/rc2.d/) are executed in sequence, first the stop scripts with stop as their argument, next the start scripts with start as their argument. The rc*.d/ directories contain symlinks the files in to /etc/init.d/.
4. The ordering is important.
5. The single-user runlevel will present a login prompt after rcS.d/ was executed. Runlevel 1 is not the single user runlevel, but it behaves as a better single user.

SysV init in Debian - Switching runlevels

• Call telinit X where X is the new runlevel, one of S, 0, 1, 2, 3, 4, 5, 6.
• init runs in sequence all stop scripts in /etc/rcX.d/ that also has a start symlink in the previous runlevel /etc/rcY.d/, with stop as their argument.
• init then run in sequence all start scripts in /etc/rcX.d/, with start as their argument.

Note that switching to runlevel S will not run the scripts in /etc/rcS.d/. To get a similar effect after boot, switch to runlevel 1. It will (should) kill all services and prepare the machine for maintenance.

SysV init in Debian - Shutting down

This is equivalent to switching to runlevel 0 (halt) or 6 (reboot), with a minor exception. All scripts (both start and stop) are executed with the stop argument.

The ordering problem

Script ordering is vital for this to work. And how are the scripts ordered? By numbers.

And the numbers are picked using skills, knowledge and negotiation. Getting it right is often hard.

The current Debian default is wrong. Stop sequence should by default be the reverse of the start sequence. It isn't.

Reordering is hard and require cooperation between maintainers of all packages involved.

The ordering problem - an example

Given two packages with two scripts inserted with the default settings in Debian:

Package A: script_a sequence 20 (start and stop)
Package B: script_b sequence 20 (start and stop)

Along come script C, which should run before script_a and after script_b. Current solution is to change packages A and C or packages B and C to get something like this:

Package A: script_a start seq. 22, stop seq. 18
Package B: script_b sequence 20 (start and stop)
Package C: script_c start seq 21, stop seq 19

If other scripts depend on the old order of script_a, they will have to change their sequence number too. Only way to discover this is by a lot of testing, or documenting script dependencies.

An ordering solution

Let each script document its dependency, and generate sequence numbers using this dependency information. Example:

Package A: script_a depend on nothing
Package B: script_b depend on nothing
Package C: script_c depend on script_b, a dependency of script_a

Generated sequence:

script_b start seq 1, stop seq 3
script_c start seq 2, stop seq 2
script_a start seq 3, stop seq 1

An implementation of this system is the dependency based boot sequencing, provided in the insserv package. Uses format specified in Linux Software Base to document dependencies.

LSB headers for insserv

• "Provides" header, list the Facility/service provided by the script. Do not list virtual facilities (like $time)
• Runlevel entries (Default-Start and Default-Stop headers), list what runlevels to activate for this script
• Dependency entries (Required-Start, Required-Stop, Should-Start, Should-Stop, X-Start-Before, X-Stop-After), list the facilities/services needed by this script. It will start after its start dependencies and stop before its stop dependencies. The X-* entreis are reverse dependencies. Required-* are hard dependencies (will install even if they are missing), while Should-* and X-* are soft dependencies (only taken into account if scripts providing them are present).

What to list as dependencies I

If your package used the default update-rc.d settings before, this is the header for you:

### BEGIN INIT INFO
# Provides:          scriptname
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

$remote_fs is needed by all scripts using files in /usr/. $syslog is needed only by scripts starting services logging to syslog.

What to list as dependencies II

In the common case, the start and stop dependencies are identical.

Prefer virtual dependencies over specific dependencies

Virtual facilities

Linux Software Base version 3.2 define these virtual facilities:

$local_fs

all local file systems are mounted. (In Debian, / and /var/ is available)

$network

basic networking support is available. Example: a server program could listen on a socket. (In Debian, network interfaces are up)

$portmap

daemons providing SunRPC/ONCRPC portmapping service as defined in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are running.

$remote_fs

all remote file systems are available. In some configurations, file systems such as /usr may be remote. Many applications that require $local_fs will probably also require $remote_fs. (In Debian, /usr/ and NFS directories are guaranteed to be mounted)

$time

the system time has been set, for example by using a network-based time program such as ntp or rdate, or via the hardware Real Time Clock.

$syslog

system logger is operational.

$named

IP name-to-address translation, using the interfaces described in this specification, are available to the level the system normally provides them. Example: if a DNS query daemon normally provides this facility, then that daemon has been started.

All of these represent points in time during boot and shutdown.

Status of dependency based boot

Release goal for Debian Lenny.

Packages with LSB header: 654 of 866 (76%)
Unsolved BTS reports: XXX
Packages without BTS reports: ~150
Last package projected fixed 2008-07-19 with the current rate

Need better documentation

Need debian policy updates

Fix insserv bugs

References

Thank you very much

Questions?

http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html