mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9   </head>
  10   <body>
  11
  12 <a href="../200706-bootseq/200706-bootseq.html">last related talk</a>
  13
  14 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering the Debian boot sequence for correctness and speed</a></h1>
  15
  16 <p>There are subtle bugs in the Debian boot and shutdown sequence.
  17 They are hard to find, as they normally only affect rare combination
  18 of packages, and harder to fix, as they normally require the combined
  19 work of several maintainers and changes in several packages.  This
  20 talk is about the release goal for Lenny to solve them, and gain a few
  21 advantages on the way.</p>
  22
  23 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  24   <br>pere@hungry.com
  25   <br>FOSDEM 2008, 2008-02-26</div>
  26
  27 <!--
  28
  29 There are subtle bugs in the debian boot and shutdown sequence. They
  30 are hard to find, as they normally only affect rare combination of
  31 packages, and harder to fix, as they normally require the combined
  32 work of several maintainers and changes in several packages.
  33
  34 One way to find these bugs is to document the dependencies of all
  35 init.d scripts, and use this information to check that the ordering is
  36 correct. When such information is available, it is also possible to
  37 reorder the boot and shutdown sequence to make sure all dependencies
  38 are fulfilled.
  39
  40 It is also possible to run scripts in parallel, to speed up the boot,
  41 when the order they need to run in is known.
  42
  43 This talk is about how all of this can be done with Debian.
  44 -->
  45
  46 <h2>Outline</h2>
  47
  48 <!--
  49
  50  - how do sysvinit boot work
  51    - runlevels
  52    - inittab
  53    - rc*.d/
  54  - ordering problem
  55  - how does it work in debian
  56  - effect on the boot
  57  - insserv
  58    - how to test it
  59  - speed change?
  60  - concurrency
  61
  62  - how to write lsb headers
  63    - provides
  64    - deps
  65    - runlevel list
  66
  67  - status in debian
  68    - need better doc
  69    - fix insserv bugs
  70
  71  - what is needed to convert
  72    - add LSB header to packages and get the change into testing
  73    - update policy
  74    - more users to test headers and detect errors
  75    - install and activate insserv by default
  76
  77 -->
  78
  79 <h2>SysV init in Debian - Booting</h2>
  80
  81 <p>Note, this is the stuff going on after the initrd part is done.
  82 The very early boot is done before hard drive partitions is mounted.</p>
  83
  84 <ol>
  85
  86 <li>/sbin/init start, which look at /etc/inittab and decides what to
  87   do.</li>
  88
  89 <li>The scripts in /etc/rcS.d/ is executed in sequence by
  90   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
  91
  92 <li>Depending on the runlevel, the scripts for the given runlevel is
  93   executed (normally the ones in /etc/rc2.d/) are executed in
  94   sequence, first the stop scripts with <tt>stop</tt> as their
  95   argument, next the start scripts with <tt>start</tt> as their
  96   argument.  The rc*.d/ directories contain symlinks the files in to
  97   /etc/init.d/.</li>
  98
  99 <li>The ordering is important.</li>
 100
 101 <li>The single-user runlevel will present a login prompt after rcS.d/
 102   was executed.  Runlevel 1 is not the single user runlevel, but it
 103   behaves as a better single user.</li>
 104
 105 </ol>
 106
 107 <h2>SysV init in Debian - Switching runlevels</h2>
 108
 109 <ul>
 110
 111 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 112   1, 2, 3, 4, 5, 6.</li>
 113
 114 <li>init runs in sequence all stop scripts in /etc/rcX.d/ that also
 115    has a start symlink in the previous runlevel /etc/rcY.d/, with
 116    <tt>stop</tt> as their argument.</li>
 117
 118 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 119   <tt>start</tt> as their argument.</li>
 120
 121 </ul>
 122
 123 <p>Note that switching to runlevel S will not run the scripts in
 124 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 125 1.  It will (should) kill all services and prepare the machine for
 126 maintenance.</p>
 127
 128 <h2>SysV init in Debian - Shutting down</h2>
 129
 130 <p>This is equivalent to switching to runlevel 0 (halt) or 6 (reboot),
 131   with a minor exception.  All scripts (both start and stop) are
 132   executed with the <tt>stop</tt> argument.
 133
 134 <h2>The ordering problem</h2>
 135
 136 <p>Script ordering is vital for this to work.  And how are the scripts
 137 ordered?  By numbers.
 138
 139 <p>And the numbers are picked using skills, knowledge and negotiation.
 140 Getting it right is often hard.
 141
 142 <p>The current Debian default is wrong.  Stop sequence should by
 143 default be the reverse of the start sequence.  It isn't.
 144
 145 <p>Reordering is hard and require cooperation between maintainers of
 146 all packages involved.
 147
 148 <h2>The ordering problem - an example</h2>
 149
 150 <p>Given two packages with two scripts inserted with the default
 151 settings in Debian:
 152
 153 <p>Package A: script_a sequence 20 (start and stop)
 154 <br>Package B: script_b sequence 20 (start and stop)
 155
 156 <p>Along come script C, which should run before script_a and after
 157 script_b.  Current solution is to change packages A and C or packages
 158 B and C to get something like this:
 159
 160 <p>Package A: script_a start seq. 22, stop seq. 18
 161 <br>Package B: script_b sequence 20 (start and stop)
 162 <br>Package C: script_c start seq 21, stop seq 19
 163
 164 <p>If other scripts depend on the old order of script_a, they will
 165 have to change their sequence number too.  Only way to discover this
 166 is by a lot of testing, or documenting script dependencies.
 167
 168 <h2>An ordering solution</h2>
 169
 170 <p>Let each script document its dependency, and generate sequence
 171 numbers using this dependency information.  Example:
 172
 173 <p>Package A: script_a depend on nothing
 174 <br>Package B: script_b depend on nothing
 175 <br>Package C: script_c depend on script_b, a dependency of script_a
 176
 177 <p>Generated sequence:
 178
 179 <p>script_b start seq 1, stop seq 3
 180 <br>script_c start seq 2, stop seq 2
 181 <br>script_a start seq 3, stop seq 1
 182
 183 <p>An implementation of this system is the dependency based boot
 184 sequencing, provided in the insserv package.  Uses format specified in
 185 Linux Software Base to document dependencies.</p>
 186
 187
 188
 189 <h2>LSB headers for insserv</h2>
 190
 191 <ul>
 192   <li>"Provides" header, list the Facility/service provided by the
 193     script.  Do not list virtual facilities (like $time)</li>
 194
 195   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 196     what runlevels to activate for this script
 197
 198   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 199     Should-Stop, X-Start-Before, X-Stop-After), list the
 200     facilities/services needed by this script.  It will start after
 201     its start dependencies and stop before its stop dependencies.  The
 202     X-* entreis are reverse dependencies.  Required-* are hard
 203     dependencies (will install even if they are missing), while
 204     Should-* and X-* are soft dependencies (only taken into account if
 205     scripts providing them are present).</li>
 206
 207 </ul>
 208
 209 <h2>What to list as dependencies I</h2>
 210
 211 <p>If your package used the default update-rc.d settings before, this
 212 is the header for you:</p>
 213 <pre>
 214 ### BEGIN INIT INFO
 215 # Provides:          scriptname
 216 # Required-Start:    $remote_fs $syslog
 217 # Required-Stop:     $remote_fs $syslog
 218 # Default-Start:     2 3 4 5
 219 # Default-Stop:      0 1 6
 220 ### END INIT INFO
 221 </pre>
 222
 223 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 224 is needed only by scripts starting services logging to syslog.</p>
 225
 226 <h2>What to list as dependencies II</h2>
 227
 228 <p>In the common case, the start and stop dependencies are identical.
 229
 230 <p>Prefer virtual dependencies over specific dependencies
 231
 232 XXX More
 233
 234 <h2>Virtual facilities</h2>
 235
 236 <p>Linux Software Base version 3.2 define these virtual facilities:
 237
 238 <dl>
 239
 240 <dt>$local_fs
 241 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 242
 243 <dt>$network
 244 <dd>basic networking support is available. Example: a server program
 245 could listen on a socket. (In Debian, network interfaces are up)
 246
 247 <dt>$portmap
 248 <dd>daemons providing SunRPC/ONCRPC portmapping service as defined in
 249 RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 250 running.
 251
 252 <dt>$remote_fs
 253 <dd>all remote file systems are available. In some configurations,
 254 file systems such as /usr may be remote. Many applications that
 255 require $local_fs will probably also require $remote_fs. (In Debian,
 256 /usr/ and NFS directories are guaranteed to be mounted)
 257
 258 <dt>$time
 259 <dd>the system time has been set, for example by using a network-based
 260 time program such as ntp or rdate, or via the hardware Real Time
 261 Clock.
 262
 263 <dt>$syslog
 264 <dd>system logger is operational.
 265
 266 <dt>$named
 267 <dd>IP name-to-address translation, using the interfaces described in
 268 this specification, are available to the level the system normally
 269 provides them. Example: if a DNS query daemon normally provides this
 270 facility, then that daemon has been started.
 271
 272 </dl>
 273
 274 <p>All of these represent points in time during boot and shutdown.
 275
 276 <h2>Status of dependency based boot</h2>
 277
 278 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 279
 280 <p>Release goal for Debian Lenny.
 281
 282 <p>Packages with LSB header (in Sid): 654 of 866 (76%)
 283 <br>Unsolved BTS reports: XXX
 284 <br>Packages without BTS reports: ~150
 285 <br>Last package projected fixed 2008-07-19 with the current rate
 286
 287 <p>Need better documentation
 288
 289 <p>Need debian policy updates
 290
 291 <p>Fix insserv bugs
 292
 293 <p>
 294
 295 <h2>References</h2>
 296
 297 http://wiki.debian.org/LSBInitScripts
 298 http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot
 299
 300     <h2>Thank you very much</h2>
 301
 302     <h3>Questions?</h3>
 303
 304     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 305
 306   </body>
 307 </html>