-<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
+ <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
<link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Language" content="en">
<meta name="Author" content="Petter Reinholdtsen">
+<style type="text/css">
+PRE {
+ font-size: smaller
+}
+</style>
</head>
<body>
-<a href="200706-bootseq/200706-bootseq.html">forrige foredrag</a>
+<h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering
+the Debian boot sequence for correctness and speed</a></h1>
-<h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering the Debian boot sequence for correctness and speed</a></h1>
-
-<p>There are subtle bugs in the Debian boot and shutdown sequence.
+<p>There are subtle bugs in the Debian boot and shutdown sequences.
They are hard to find, as they normally only affect rare combination
-of packages, and harder to fix, as they normally require the combined
-work of several maintainers and changes in several packages. This
-talk is about how we can solve them, and gain a few advantages on the
-way.</p>
+of packages. They are harder to fix, as they normally require the
+combined work of several maintainers and changes in several packages.
+This talk is about the release goal for Lenny to solve them, and gain
+a few advantages on the way.</p>
<div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
<br>pere@hungry.com
- <br>FOSDEM 2008, 2008-02-26</div>
+ <br>FOSDEM 2008, 2008-02-24</div>
<!--
This talk is about how all of this can be done with Debian.
-->
-<h2>Summary</h2>
+<h2>Outline</h2>
+
+<ul>
+<li>Quick overview of the SysV init boot system
+<li>The problem with ordering
+<li>Using LSB init.d script dependencies to solve it.
+<li>Status of dependency based boot sequencing.
+<li>How to write LSB headers.
+<li>What is left to do.
+</ul>
<!--
- how does it work in debian
- effect on the boot
- insserv
+ - how to verify the boot sequence
+ - check script
+ - dependency graph using dotty
+ - how to test it
+ - overriding included headers
- speed change?
- concurrency
<h2>SysV init in Debian - Booting</h2>
<p>Note, this is the stuff going on after the initrd part is done.
-The very early boot is done before hard drive partitions is mounted.</p>
+The very early boot is done before hard drive partitions are
+mounted.</p>
<ol>
-<li>/sbin/init start, which look at /etc/inittab and decides what to
+<li>/sbin/init stats and looks at /etc/inittab to decide what to
do.</li>
-<li>The scripts in /etc/rcS.d/ is executed in sequence by
+<li>The scripts in /etc/rcS.d/ are executed in sequence by
/etc/init.d/rc, with <tt>start</tt> as the argument.</li>
-<li>Depending on the runlevel, the scripts for the given runlevel is
+<li>Depending on the runlevel, the scripts for the given runlevel are
executed (normally the ones in /etc/rc2.d/) are executed in
sequence, first the stop scripts with <tt>stop</tt> as their
argument, next the start scripts with <tt>start</tt> as their
<li>The ordering is important.</li>
-<li>The single-user runlevel will present a login prompt after rcS.d/
- was executed. Runlevel 1 is not the single user runlevel, but it
- behaves as a better single user.</li>
+<li>Runlevel 1 is not the single user runlevel. The single-user
+ runlevel will present a login prompt after rcS.d/ was executed.
+ Runlevel 1 is not the single user runlevel, but it behaves as a
+ better single user.</li>
</ol>
+<p>Note that because the Linux kernel is becoming more and more event
+based, the boot sequence is no longer sequencial.<p>
+
<h2>SysV init in Debian - Switching runlevels</h2>
<ul>
<li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
1, 2, 3, 4, 5, 6.</li>
-<li>init runs in sequence all stop scripts in /etc/rcX.d/ that also
- has a start symlink in the previous runlevel /etc/rcY.d/, with
+<li>init runs all stop scripts in /etc/rcX.d/ that also have a start
+ symlink in the previous runlevel /etc/rcY.d/ in sequence, with
<tt>stop</tt> as their argument.</li>
<li>init then run in sequence all start scripts in /etc/rcX.d/, with
<tt>start</tt> as their argument.</li>
+<li>Runlevels 0 and 6 are not behaving as you would expect.
+
</ul>
<p>Note that switching to runlevel S will not run the scripts in
<h2>SysV init in Debian - Shutting down</h2>
-<p>This is equivalent to switching to runlevel 0 (halt) or 6 (reboot),
- with a minor exception. All scripts (both start and stop) are
- executed with the <tt>stop</tt> argument.
+<p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
+(reboot).</p>
+
+<p>Minor exception: all scripts (both start and stop) are executed
+ with the <tt>stop</tt> argument, ignoring their start and stop
+ settings and confusing script writers.</p>
+
+<p>Only stop scripts for services started in the previous runlevel are
+ executed.</p>
<h2>The ordering problem</h2>
<p>Script ordering is vital for this to work. And how are the scripts
-ordered? By numbers.
+ordered? By numbers 01-99!
<p>And the numbers are picked using skills, knowledge and negotiation.
Getting it right is often hard.
-<p>The current Debian default is wrong. Stop sequence should by
-default be the reverse of the start sequence. It isn't.
+<p>The current Debian default is wrong. The stop sequence should by
+default be the reverse of the start sequence. It isn't. The default
+uses '20' for both.
-<p>Reordering is hard and require cooperation between maintainers of
-all packages involved. Given two packages with two scripts inserted
-with the default settings in Debian:
+<p>Reordering is hard and sometimes requires cooperation between
+maintainers of all packages involved.
+
+<h2>The ordering problem - an example</h2>
+
+<p>Given two packages with two scripts inserted with the default
+settings in Debian:
<p>Package A: script_a sequence 20 (start and stop)
<br>Package B: script_b sequence 20 (start and stop)
-<p>Along come script C, which should run before script_a and after
+<p>Along comes script C, which should run before script_a and after
script_b. Current solution is to change packages A and C or packages
B and C to get something like this:
<br>Package C: script_c start seq 21, stop seq 19
<p>If other scripts depend on the old order of script_a, they will
-have to change their sequence number too. Only way to discover this
-is by a lot of testing, or documenting script dependencies.
+have to change their sequence number too. The only way to discover
+this is by a lot of testing, or documenting script dependencies.
-<h2>A ordering solution</h2>
+<h2>An ordering solution</h2>
-<p>Let each script document its dependency, and generate sequence
+<p>Let each script document its dependencies, and generate sequence
numbers using this dependency information. Example:
<p>Package A: script_a depend on nothing
<br>script_a start seq 3, stop seq 1
<p>An implementation of this system is the dependency based boot
-sequencing, provided in the insserv package.</p>
+sequencing, provided by the insserv package. It uses the format
+specified in Linux Software Base to document init.d script
+dependencies.</p>
+
+<h2>Checking the current boot sequence</h2>
+
+Two options are available with the insserv package:
+
+<dl>
+<dt>Static checking of current headers:
+<br><tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt>
+<dd>report mismatch in current ordering.
+
+<dt>Graph of the dependencies:
+<br><a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a>
+<dd>for reviewing dependencies with dotty from graphviz.
+
+</dl>
+
+<h2>Enabling dependency based boot sequencing I</h2>
+
+<p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
+<ul>
+ <li>Use the insserv package in unstable or testing
+ <li>Low priority debconf question.
+ <li>Only enabled after verfiying that it is safe to enable
+ <li>Converts all start symlinks in rc0.d/ and rc6.d to stop symlinks.
+ <li>Replaces update-rc.d with a wrapper calling insserv
+ <li>Finally reorder symlinks in rc*.d/ using dependencies
+</ul>
+
+<p>Will refuse to enable when obsolete init.d scripts, loops,
+duplicate provides etc is detected</p>
+
+<h2>Enabling dependency based boot sequencing II</h2>
+
+<p><pre>
+# aptitude install insserv
+# dpkg-reconfigure insserv
+info: Checking if it is safe to convert to dependency based boot.
+info: Backing up existing boot scripts in \
+ /var/lib/insserv/bootscripts-20080223T0742.tar.gz
+info: Reordering boot system, log to \
+ /var/lib/insserv/run-20080223T0742.log
+info: Recording new boot sequence in \
+ /var/lib/insserv/bootscripts-20080223T0742-after.list
+info: Use '/usr/sbin/update-bootsystem-insserv \
+ restore' to restore the old boot sequence.
+Adding `diversion of /usr/sbin/update-rc.d to \
+ /usr/sbin/update-rc.d.distrib by insserv'
+success: Boot system successfully converted
+# <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
+ /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
+[...]
+#
+</pre>
+
+<h2>Using dependency based boot sequencing</h2>
+
+<p>update-rc.d refuses to Insert scripts which create a loop.</p>
+
+<p>update-rc.d requires scripts to be inserted in dependency order.</p>
+
+<p>Incorrect dependencies give the wrong but predictable and stable
+(as in the same all the time) boot and shutdown order.</p>
+
+<p>Every insserv upload are checked using test suite to make sure the
+generated boot sequence is correct, and that previously detected bugs
+do not show up again.</p>
+
+<p>It is possible to enable concurrent booting, running boot scripts
+ in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
+
+<p>Might even speed up boot and shutdown (initial benchmark shows
+ speedup by just reordering based on dependencies).</p>
+
+<h2>Disabling dependency based boot sequencing</h2>
+
+<p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
+
+<p>It is always possible to disable just after it was enabled, before
+any new packages are installed.</p>
+
+<p>When disabling it, a backup of the old boot sequence is restored if no
+changes have been made to the boot sequence since it was enabled.
+<p>If restore is not possible, all postinst scripts for packages with
+init.d scripts will be executed again to make them call update-rc.d
+and add the boot scripts again.
+
+<p>This is guaranteed to work if no packages have been added since it
+was enabled, and most often works if packages have been added. So if
+you change your mind, do it quickly.
<h2>LSB headers for insserv</h2>
<li>Dependency entries (Required-Start, Required-Stop, Should-Start,
Should-Stop, X-Start-Before, X-Stop-After), list the
- facilities/services needed by this script. It will start after
- its start dependencies and stop before its stop dependencies. The
- X-* entreis are reverse dependencies. Required-* are hard
- dependencies (will install even if they are missing), while
- Should-* and X-* are soft dependencies (only taken into account if
- scripts providing them are present).</li>
+ facilities/services needed by this script. The script will start
+ after its start dependencies and stop before its stop
+ dependencies. The X-* entries are reverse dependencies.
+ Required-* are hard dependencies (update-rc.d will refuse to
+ install the script if any of them are missing), while Should-* and
+ X-* are soft dependencies (only taken into account if scripts
+ providing them are present).</li>
</ul>
-<h2>What to list as dependencies I</h2>
+<h2>What to list as dependencies (I)</h2>
<p>If your package used the default update-rc.d settings before, this
is the header for you:</p>
<p>$remote_fs is needed by all scripts using files in /usr/. $syslog
is needed only by scripts starting services logging to syslog.</p>
+<h2>Virtual facilities (I)</h2>
+
+<p>Linux Software Base version 3.2 defines these virtual
+facilities:</p>
+
+<dl>
+
+<dt>$local_fs
+<dd>all local file systems are mounted. (In Debian, / and /var/ is available)
+
+<dt>$network
+<dd>basic networking support is available. Example: a server program
+could listen on a socket. (In Debian, network interfaces are up)
+
+<dt>$portmap
+<dd>daemons providing the SunRPC/ONCRPC portmapping service as defined
+in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
+running.
+
+<dt>$remote_fs
+<dd>all remote file systems are available. In some configurations,
+file systems such as /usr may be remote. Many applications that
+require $local_fs will probably also require $remote_fs. (In Debian,
+/usr/ and NFS directories are guaranteed to be mounted)
+</dl>
+
+<h2>Virtual facilities (II)</h2>
+
+<dl>
+
+<dt>$time
+<dd>the system time has been set, for example by using a network-based
+time program such as ntp or rdate, or via the hardware Real Time
+Clock.
+
+<dt>$syslog
+<dd>the system logger is operational.
+
+<dt>$named
+<dd>IP name-to-address translation, using the interfaces described in
+this specification, are available to the level the system normally
+provides them. Example: if a DNS query daemon normally provides this
+facility, then that daemon has been started.
+
+</dl>
+
+<p>All of these represent points in time during boot and shutdown.
+Virtual facilities are defined in <tt>/etc/insserv.conf</tt> and
+<tt>/etc/insserv.conf.d/</tt>
+
+<h2>What to list as dependencies (II)</h2>
+
+<p>Normally, the start and stop dependencies are the same.
-<h2>Status of dependency based boot</h2>
+<p>Virtual dependencies are preferred over specific dependencies.
+
+<p>When using specific dependencies, use the string listed in the
+ provides header of the scripts you depend on.
+
+<p>Scripts started in rcS.d/ need extra care.
+
+<p>All scripts not started in rcS.d/ should depend on $remote_fs.
+This make sure <tt>/usr/</tt> is available during start and that it is
+stopped before <tt>sendsigs</tt> kills all processes during
+shutdown.</p>
+
+<p>A sysadmin can provide overrides in
+<tt>/etc/insserv/overrides/scriptname</tt> if the script settings are
+wrong. The insserv package provides overrides in
+<tt>/usr/share/insserv/overrides/<tt> for packages currently missing
+headers.</p>
+
+<h2>Handling alternatives</h2>
+
+<p>Not quite tested yet.
+
+<p>Should perhaps be handled using virtual facilities in
+/etc/insserv.conf.d/.
+
+<p>Do not listing identical provides in several scripts - break
+installation.
+
+<h2>Status of the dependency based boot system</h2>
<img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
-<p>Release goal for Debian Lenny.
+<p>Release goal for Lenny.
+<br>76% of packages got LSB headers.
+<br>Unsolved in BTS: ~85
+<br>Without BTS reports: ~150
+<br>Last package will be fixed 2008-06-15 at the current rate.
+<br>Needs better documentation for maintainers.
+<br>Should update Debian policy to reflect dependency based boot
+ sequencing.
+
+<p>Two insserv bugs left to fix:
+<ul>
+<li>Might add unwanted stop symlinks (<a href="http://bugs.debian.org/464017">#464017</a>).
+<li>Will fail on "fake" loops as well as real loops. (<a href="http://bugs.debian.org/458582">#458582</a>)
+</ul></p>
+
+<h2>Tracking status</h2>
-<p>Packages with LSB header: 654 of 866 (76%)
-<br>Unsolved BTS reports: XXX
-<br>Packages without BTS reports: ~150
+<p>The current status is tracked in the svn repository for insserv. Run
+<br><tt><a href="status-overrides.txt">debian/rules missing-overrides</a></tt> or
+<br><tt><a href="status-popcon.txt">debian/rules missing-by-popcon</a></tt>
+<br>in the source directory to see the current status.
-<p>Need better documentation
+<p>See also wiki pages with documentation and status:
-<p>Need debian policy updates
+<ul>
+<li><a href="http://wiki.debian.org/LSBInitScripts">http://wiki.debian.org/LSBInitScripts</a>
+<li><a href="http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot">http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot</a>
+<li>Repository: <tt>svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv</tt>
+
+</ul>
-<p>Fix insserv bugs
+<p>There is also slides from my
+<a href="../200706-bootseq/200706-bootseq.html">talk from Debconf
+7</a>.<p>
-<p>
+<p>More man-power is needed to report BTS reports, NMU packages and
+discover dependency errors. Please test the system.
<h2>Thank you very much</h2>
projects / homepage.git / blobdiff