Reordering the Debian boot sequence for correctness and speed

There are subtle bugs in the Debian boot and shutdown sequence. They are hard to find, as they normally only affect rare combination of packages, and harder to fix, as they normally require the combined work of several maintainers and changes in several packages. This talk is about the release goal for Lenny to solve them, and gain a few advantages on the way.

Outline

SysV init in Debian - Booting

Note, this is the stuff going on after the initrd part is done. The very early boot is done before hard drive partitions is mounted.

1. /sbin/init start, which look at /etc/inittab and decides what to do.
2. The scripts in /etc/rcS.d/ is executed in sequence by /etc/init.d/rc, with start as the argument.
3. Depending on the runlevel, the scripts for the given runlevel is executed (normally the ones in /etc/rc2.d/) are executed in sequence, first the stop scripts with stop as their argument, next the start scripts with start as their argument. The rc*.d/ directories contain symlinks the files in to /etc/init.d/.
4. The ordering is important.
5. The single-user runlevel will present a login prompt after rcS.d/ was executed. Runlevel 1 is not the single user runlevel, but it behaves as a better single user.

Note that because the Linux kernel is becoming more and more event based, the boot sequence is no longer sequencial.

SysV init in Debian - Switching runlevels

• Call telinit X where X is the new runlevel, one of S, 0, 1, 2, 3, 4, 5, 6.
• init runs in sequence all stop scripts in /etc/rcX.d/ that also has a start symlink in the previous runlevel /etc/rcY.d/, with stop as their argument.
• init then run in sequence all start scripts in /etc/rcX.d/, with start as their argument.

Note that switching to runlevel S will not run the scripts in /etc/rcS.d/. To get a similar effect after boot, switch to runlevel 1. It will (should) kill all services and prepare the machine for maintenance.

SysV init in Debian - Shutting down

This is roughtly equivalent to switching to runlevel 0 (halt) or 6 (reboot).

Minor exception: all scripts (both start and stop) are executed with the stop argument, ignoring their start and stop setting and confusing script writers.

Only stop scripts for services started in the previous runlevel is executed.

The ordering problem

Script ordering is vital for this to work. And how are the scripts ordered? By numbers.

And the numbers are picked using skills, knowledge and negotiation. Getting it right is often hard.

The current Debian default is wrong. Stop sequence should by default be the reverse of the start sequence. It isn't.

Reordering is hard and require cooperation between maintainers of all packages involved.

The ordering problem - an example

Given two packages with two scripts inserted with the default settings in Debian:

Package A: script_a sequence 20 (start and stop)
Package B: script_b sequence 20 (start and stop)

Along come script C, which should run before script_a and after script_b. Current solution is to change packages A and C or packages B and C to get something like this:

Package A: script_a start seq. 22, stop seq. 18
Package B: script_b sequence 20 (start and stop)
Package C: script_c start seq 21, stop seq 19

If other scripts depend on the old order of script_a, they will have to change their sequence number too. Only way to discover this is by a lot of testing, or documenting script dependencies.

An ordering solution

Let each script document its dependency, and generate sequence numbers using this dependency information. Example:

Package A: script_a depend on nothing
Package B: script_b depend on nothing
Package C: script_c depend on script_b, a dependency of script_a

Generated sequence:

script_b start seq 1, stop seq 3
script_c start seq 2, stop seq 2
script_a start seq 3, stop seq 1

An implementation of this system is the dependency based boot sequencing, provided in the insserv package. Uses format specified in Linux Software Base to document dependencies.

Checking the current boot sequence

Static checking of current headers: /usr/share/insserv/check-initd-order [-o] [-k] - report mismatch in current ordering.

Graph of the dependencies: /usr/share/insserv/check-initd-order -g - for reviewing dependencies

Enabling dependency based boot sequencing I

• Use the insserv package in unstable and testing
• Low priority debconf question.
• Only enabled after verfiying that it is safe to enable
• Converts all start symlinks in rc0.d/ and rc6.d with stop symlinks.
• Replaces update-rc.d with wrapper calling insserv
• All of rc*.d/ is reordered

Enabling dependency based boot sequencing II

# aptitude install insserv
# dpkg-reconfigure insserv
info: Checking if it is safe to convert to dependency based boot.
info: Backing up existing boot scripts in \
  /var/lib/insserv/bootscripts-20080223T0742.tar.gz
info: Reordering boot system, log to \
  /var/lib/insserv/run-20080223T0742.log
info: Recording new boot sequence in \
  /var/lib/insserv/bootscripts-20080223T0742-after.list
info: Use '/usr/sbin/update-bootsystem-insserv \
  restore' to restore the old boot sequence.
Adding `diversion of /usr/sbin/update-rc.d to \
  /usr/sbin/update-rc.d.distrib by insserv'
success: Boot system successfully converted
# /var/lib/insserv/insserv-seq-changes \
  /var/lib/insserv/bootscripts-20080223T0742.tar.gz
#

With dependency based boot sequencing

update-rc.d refuse to Insert scripts which create a loop.

update-rc.d require scripts to be inserted in dependency order.

Incorrect dependencies give the wrong boot and shutdown order.

It is possible to enable concurrent booting, running boot scripts in parallel (CONCURRENCY=startpar in /etc/default/rcS)

Disabling dependency based boot sequencing

Run dpkg-reconfigure insserv and disable it.

It is always possible to disable just after it was enabled, before any new packages are installed.

To disable it, a backup of the old boot sequence is restored if no changes has been done to the boot sequence since it was enabled.

If restoring is not possible, All postinst scripts for packages with init.d scripts will be executed again to get them to call update-rc.d and add the boot scripts again.

LSB headers for insserv

• "Provides" header, list the Facility/service provided by the script. Do not list virtual facilities (like $time)
• Runlevel entries (Default-Start and Default-Stop headers), list what runlevels to activate for this script
• Dependency entries (Required-Start, Required-Stop, Should-Start, Should-Stop, X-Start-Before, X-Stop-After), list the facilities/services needed by this script. It will start after its start dependencies and stop before its stop dependencies. The X-* entreis are reverse dependencies. Required-* are hard dependencies (will install even if they are missing), while Should-* and X-* are soft dependencies (only taken into account if scripts providing them are present).

What to list as dependencies I

If your package used the default update-rc.d settings before, this is the header for you:

### BEGIN INIT INFO
# Provides:          scriptname
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

$remote_fs is needed by all scripts using files in /usr/. $syslog is needed only by scripts starting services logging to syslog.

Virtual facilities

Linux Software Base version 3.2 define these virtual facilities:

$local_fs

all local file systems are mounted. (In Debian, / and /var/ is available)

$network

basic networking support is available. Example: a server program could listen on a socket. (In Debian, network interfaces are up)

$portmap

daemons providing SunRPC/ONCRPC portmapping service as defined in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are running.

$remote_fs

all remote file systems are available. In some configurations, file systems such as /usr may be remote. Many applications that require $local_fs will probably also require $remote_fs. (In Debian, /usr/ and NFS directories are guaranteed to be mounted)

$time

the system time has been set, for example by using a network-based time program such as ntp or rdate, or via the hardware Real Time Clock.

$syslog

system logger is operational.

$named

IP name-to-address translation, using the interfaces described in this specification, are available to the level the system normally provides them. Example: if a DNS query daemon normally provides this facility, then that daemon has been started.

All of these represent points in time during boot and shutdown.

What to list as dependencies II

Normally, the start and stop dependencies are the same.

Prefer virtual dependencies over specific dependencies.

When using specific dependencies, use the string listed in the provides header of the scripts you depend on.

Scripts started in rcS.d/ need extra care.

Depend on $remote_fs unless started in rcS.d/. Make sure /usr/ is available during start and that it is stopped before sendsigs kill all processes during shutdown.

Status of dependency based boot

Release goal for Debian Lenny.

Packages with LSB header (in Sid): 654 of 866 (76%)
Unsolved BTS reports: 85
Packages without BTS reports: ~150
Last package projected fixed 2008-07-19 with the current rate

Need better documentation

Should update Debian policy to reflect this new feature.

Two insserv bugs to fix.

• Avoid adding unwanted stop symlinks.
• Do not fail on "fake" loops. (minor)

Tracking status

References

Thank you very much

Questions?

http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html