mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9 <style type="text/css">
  10 PRE {
  11   font-size: smaller
  12 }
  13 </style>
  14   </head>
  15   <body>
  16
  17 <a href="../200706-bootseq/200706-bootseq.html">insserv talk during Debconf 7</a>
  18
  19 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering
  20 the Debian boot sequence for correctness and speed</a></h1>
  21
  22 <p>There are subtle bugs in the Debian boot and shutdown sequences.
  23 They are hard to find, as they normally only affect rare combination
  24 of packages.  They are harder to fix, as they normally require the
  25 combined work of several maintainers and changes in several packages.
  26 This talk is about the release goal for Lenny to solve them, and gain
  27 a few advantages on the way.</p>
  28
  29 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  30   <br>pere@hungry.com
  31   <br>FOSDEM 2008, 2008-02-26</div>
  32
  33 <!--
  34
  35 There are subtle bugs in the debian boot and shutdown sequence. They
  36 are hard to find, as they normally only affect rare combination of
  37 packages, and harder to fix, as they normally require the combined
  38 work of several maintainers and changes in several packages.
  39
  40 One way to find these bugs is to document the dependencies of all
  41 init.d scripts, and use this information to check that the ordering is
  42 correct. When such information is available, it is also possible to
  43 reorder the boot and shutdown sequence to make sure all dependencies
  44 are fulfilled.
  45
  46 It is also possible to run scripts in parallel, to speed up the boot,
  47 when the order they need to run in is known.
  48
  49 This talk is about how all of this can be done with Debian.
  50 -->
  51
  52 <h2>Outline</h2>
  53
  54 <ul>
  55 <li>Quick overview of the SysV init boot system
  56 <li>The problem with ordering
  57 <li>Using LSB init.d script dependencies to solve it.
  58 <li>Status of dependency based boot sequencing.
  59 <li>How to write LSB headers.
  60 <li>What is left to do.
  61 </ul>
  62
  63 <!--
  64
  65  - how do sysvinit boot work
  66    - runlevels
  67    - inittab
  68    - rc*.d/
  69  - ordering problem
  70  - how does it work in debian
  71  - effect on the boot
  72  - insserv
  73    - how to verify the boot sequence
  74      - check script
  75      - dependency graph using dotty
  76    - how to test it
  77    - overriding included headers
  78  - speed change?
  79  - concurrency
  80
  81  - how to write lsb headers
  82    - provides
  83    - deps
  84    - runlevel list
  85
  86  - status in debian
  87    - need better doc
  88    - fix insserv bugs
  89
  90  - what is needed to convert
  91    - add LSB header to packages and get the change into testing
  92    - update policy
  93    - more users to test headers and detect errors
  94    - install and activate insserv by default
  95
  96 -->
  97
  98 <h2>SysV init in Debian - Booting</h2>
  99
 100 <p>Note, this is the stuff going on after the initrd part is done.
 101 The very early boot is done before hard drive partitions are
 102 mounted.</p>
 103
 104 <ol>
 105
 106 <li>/sbin/init start, which looks at /etc/inittab and decides what to
 107   do.</li>
 108
 109 <li>The scripts in /etc/rcS.d/  are executed in sequence by
 110   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
 111
 112 <li>Depending on the runlevel, the scripts for the given runlevel are
 113   executed (normally the ones in /etc/rc2.d/) are executed in
 114   sequence, first the stop scripts with <tt>stop</tt> as their
 115   argument, next the start scripts with <tt>start</tt> as their
 116   argument.  The rc*.d/ directories contain symlinks the files in to
 117   /etc/init.d/.</li>
 118
 119 <li>The ordering is important.</li>
 120
 121 <li>Runlevel 1 is not the single user runlevel.  The single-user
 122   runlevel will present a login prompt after rcS.d/ was executed.
 123   Runlevel 1 is not the single user runlevel, but it behaves as a
 124   better single user.</li>
 125
 126 </ol>
 127
 128 <p>Note that because the Linux kernel is becoming more and more event
 129 based, the boot sequence is no longer sequencial.<p>
 130
 131 <h2>SysV init in Debian - Switching runlevels</h2>
 132
 133 <ul>
 134
 135 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 136   1, 2, 3, 4, 5, 6.</li>
 137
 138 <li>init runs all stop scripts in /etc/rcX.d/ that also have a start
 139    symlink in the previous runlevel /etc/rcY.d/ in sequence, with
 140    <tt>stop</tt> as their argument.</li>
 141
 142 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 143   <tt>start</tt> as their argument.</li>
 144
 145 <li>Runlevels 0 and 6 are not behaving as you would expect.
 146
 147 </ul>
 148
 149 <p>Note that switching to runlevel S will not run the scripts in
 150 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 151 1.  It will (should) kill all services and prepare the machine for
 152 maintenance.</p>
 153
 154 <h2>SysV init in Debian - Shutting down</h2>
 155
 156 <p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
 157 (reboot).</p>
 158
 159 <p>Minor exception: all scripts (both start and stop) are executed
 160   with the <tt>stop</tt> argument, ignoring their start and stop
 161   settings and confusing script writers.</p>
 162
 163 <p>Only stop scripts for services started in the previous runlevel are
 164   executed.</p>
 165
 166 <h2>The ordering problem</h2>
 167
 168 <p>Script ordering is vital for this to work.  And how are the scripts
 169 ordered?  By numbers 01-99!
 170
 171 <p>And the numbers are picked using skills, knowledge and negotiation.
 172 Getting it right is often hard.
 173
 174 <p>The current Debian default is wrong.  The stop sequence should by
 175 default be the reverse of the start sequence.  It isn't.  The default
 176 uses '20' for both.
 177
 178 <p>Reordering is hard and sometimes requires cooperation between
 179 maintainers of all packages involved.
 180
 181 <h2>The ordering problem - an example</h2>
 182
 183 <p>Given two packages with two scripts inserted with the default
 184 settings in Debian:
 185
 186 <p>Package A: script_a sequence 20 (start and stop)
 187 <br>Package B: script_b sequence 20 (start and stop)
 188
 189 <p>Along comes script C, which should run before script_a and after
 190 script_b.  Current solution is to change packages A and C or packages
 191 B and C to get something like this:
 192
 193 <p>Package A: script_a start seq. 22, stop seq. 18
 194 <br>Package B: script_b sequence 20 (start and stop)
 195 <br>Package C: script_c start seq 21, stop seq 19
 196
 197 <p>If other scripts depend on the old order of script_a, they will
 198 have to change their sequence number too.  The only way to discover
 199 this is by a lot of testing, or documenting script dependencies.
 200
 201 <h2>An ordering solution</h2>
 202
 203 <p>Let each script document its dependencies, and generate sequence
 204 numbers using this dependency information.  Example:
 205
 206 <p>Package A: script_a depend on nothing
 207 <br>Package B: script_b depend on nothing
 208 <br>Package C: script_c depend on script_b, a dependency of script_a
 209
 210 <p>Generated sequence:
 211
 212 <p>script_b start seq 1, stop seq 3
 213 <br>script_c start seq 2, stop seq 2
 214 <br>script_a start seq 3, stop seq 1
 215
 216 <p>An implementation of this system is the dependency based boot
 217 sequencing, provided by the insserv package.  It uses the format
 218 specified in Linux Software Base to document init.d script
 219 dependencies.</p>
 220
 221 <h2>Checking the current boot sequence</h2>
 222
 223 Two options are available with the insserv package:
 224
 225 <dl>
 226 <dt>Static checking of current headers:
 227 <tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt>
 228 <dd>report mismatch in current ordering.
 229
 230 <dt>Graph of the dependencies:
 231 <a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a>
 232 <dd>for reviewing dependencies with dotty from graphviz.
 233
 234 </dl>
 235
 236 <h2>Enabling dependency based boot sequencing I</h2>
 237
 238 <p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
 239 <ul>
 240   <li>Use the insserv package in unstable or testing
 241   <li>Low priority debconf question.
 242   <li>Only enabled after verfiying that it is safe to enable
 243   <li>Converts all start symlinks in rc0.d/ and rc6.d to stop symlinks.
 244   <li>Replaces update-rc.d with a wrapper calling insserv
 245   <li>Finally reorder symlinks in rc*.d/ using dependencies
 246 </ul>
 247
 248 <h2>Enabling dependency based boot sequencing II</h2>
 249
 250 <p><pre size="-1">
 251 # aptitude install insserv
 252 # dpkg-reconfigure insserv
 253 info: Checking if it is safe to convert to dependency based boot.
 254 info: Backing up existing boot scripts in \
 255   /var/lib/insserv/bootscripts-20080223T0742.tar.gz
 256 info: Reordering boot system, log to \
 257   /var/lib/insserv/run-20080223T0742.log
 258 info: Recording new boot sequence in \
 259   /var/lib/insserv/bootscripts-20080223T0742-after.list
 260 info: Use '/usr/sbin/update-bootsystem-insserv \
 261   restore' to restore the old boot sequence.
 262 Adding `diversion of /usr/sbin/update-rc.d to \
 263   /usr/sbin/update-rc.d.distrib by insserv'
 264 success: Boot system successfully converted
 265 # <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
 266   /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
 267 [...]
 268 #
 269 </pre>
 270
 271 <h2>Using dependency based boot sequencing</h2>
 272
 273 <p>update-rc.d refuses to Insert scripts which create a loop.</p>
 274
 275 <p>update-rc.d requires scripts to be inserted in dependency order.</p>
 276
 277 <p>Incorrect dependencies give the wrong but predictable and stable
 278 (as in the same all the time) boot and shutdown order.</p>
 279
 280 <p>Every insserv upload are checked using test suite to make sure the
 281 generated boot sequence is correct, and that previously detected bugs
 282 do not show up again.</p>
 283
 284 <p>It is possible to enable concurrent booting, running boot scripts
 285   in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
 286
 287 <p>Might even speed up boot and shutdown (initial benchmark shows
 288   speedup by just reordering based on dependencies).</p>
 289
 290 <h2>Disabling dependency based boot sequencing</h2>
 291
 292 <p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
 293
 294 <p>It is always possible to disable just after it was enabled, before
 295 any new packages are installed.</p>
 296
 297 <p>When disabling it, a backup of the old boot sequence is restored if no
 298 changes have been made to the boot sequence since it was enabled.
 299
 300 <p>If restore is not possible, all postinst scripts for packages with
 301 init.d scripts will be executed again to make them call update-rc.d
 302 and add the boot scripts again.
 303
 304 <p>This is guaranteed to work if no packages have been added since it
 305 was enabled, and most often works if packages have been added.  So if
 306 you change your mind, do it quickly.
 307
 308 <h2>LSB headers for insserv</h2>
 309
 310 <ul>
 311   <li>"Provides" header, list the Facility/service provided by the
 312     script.  Do not list virtual facilities (like $time)</li>
 313
 314   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 315     what runlevels to activate for this script
 316
 317   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 318     Should-Stop, X-Start-Before, X-Stop-After), list the
 319     facilities/services needed by this script.  The script will start
 320     after its start dependencies and stop before its stop
 321     dependencies.  The X-* entries are reverse dependencies.
 322     Required-* are hard dependencies (update-rc.d will refuse to
 323     install the script if any of them are missing), while Should-* and
 324     X-* are soft dependencies (only taken into account if scripts
 325     providing them are present).</li>
 326
 327 </ul>
 328
 329 <h2>What to list as dependencies (I)</h2>
 330
 331 <p>If your package used the default update-rc.d settings before, this
 332 is the header for you:</p>
 333 <pre>
 334 ### BEGIN INIT INFO
 335 # Provides:          scriptname
 336 # Required-Start:    $remote_fs $syslog
 337 # Required-Stop:     $remote_fs $syslog
 338 # Default-Start:     2 3 4 5
 339 # Default-Stop:      0 1 6
 340 ### END INIT INFO
 341 </pre>
 342
 343 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 344 is needed only by scripts starting services logging to syslog.</p>
 345
 346 <h2>Virtual facilities (I)</h2>
 347
 348 <p>Linux Software Base version 3.2 defines these virtual
 349 facilities:</p>
 350
 351 <dl>
 352
 353 <dt>$local_fs
 354 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 355
 356 <dt>$network
 357 <dd>basic networking support is available. Example: a server program
 358 could listen on a socket. (In Debian, network interfaces are up)
 359
 360 <dt>$portmap
 361 <dd>daemons providing the SunRPC/ONCRPC portmapping service as defined
 362 in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 363 running.
 364
 365 <dt>$remote_fs
 366 <dd>all remote file systems are available. In some configurations,
 367 file systems such as /usr may be remote. Many applications that
 368 require $local_fs will probably also require $remote_fs. (In Debian,
 369 /usr/ and NFS directories are guaranteed to be mounted)
 370 </dl>
 371
 372 <h2>Virtual facilities (II)</h2>
 373
 374 <dl>
 375
 376 <dt>$time
 377 <dd>the system time has been set, for example by using a network-based
 378 time program such as ntp or rdate, or via the hardware Real Time
 379 Clock.
 380
 381 <dt>$syslog
 382 <dd>the system logger is operational.
 383
 384 <dt>$named
 385 <dd>IP name-to-address translation, using the interfaces described in
 386 this specification, are available to the level the system normally
 387 provides them. Example: if a DNS query daemon normally provides this
 388 facility, then that daemon has been started.
 389
 390 </dl>
 391
 392 <p>All of these represent points in time during boot and shutdown.
 393 Virtual facilities are defined in <tt>/etc/insserv.conf</tt> and
 394 <tt>/etc/insserv.conf.d/</tt>
 395
 396 <h2>What to list as dependencies (II)</h2>
 397
 398 <p>Normally, the start and stop dependencies are the same.
 399
 400 <p>Virtual dependencies are preferred over specific dependencies.
 401
 402 <p>When using specific dependencies, use the string listed in the
 403   provides header of the scripts you depend on.
 404
 405 <p>Scripts started in rcS.d/ need extra care.
 406
 407 <p>All scripts not started in rcS.d/ should depend on $remote_fs.
 408 This make sure <tt>/usr/</tt> is available during start and that it is
 409 stopped before <tt>sendsigs</tt> kills all processes during
 410 shutdown.</p>
 411
 412 <p>A sysadmin can provide overrides in
 413 <tt>/etc/insserv/overrides/scriptname</tt> if the script settings are
 414 wrong.  The insserv package provides overrides in
 415 <tt>/usr/share/insserv/overrides/<tt> for packages currently missing
 416 headers.</p>
 417
 418 <h2>Status of the dependency based boot system</h2>
 419
 420 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 421
 422 <p>Release goal for Debian Lenny.
 423 <br>76% of packages got LSB headers.
 424 <br>Unsolved in BTS: ~85
 425 <br>Without BTS reports: ~150
 426 <br>Last package will be fixed 2008-06-13 at the current rate.
 427 <br>Needs better documentation for maintainers.
 428 <br>Should update Debian policy to reflect dependency based boot
 429   sequencing.
 430
 431 <p>Two insserv bugs left to fix:
 432 <ul>
 433 <li>Will sometimes add  unwanted stop symlinks (<a href="http://bugs.debian.org/464017">#464017</a>).
 434 <li>Will fail on "fake" loops as well as real loops. (<a href="http://bugs.debian.org/458582">#458582</a>)
 435 </ul></p>
 436
 437 <h2>Tracking status</h2>
 438
 439 <p>The current status is tracked in the svn repository for insserv.  Run
 440 <br><tt><a href="status-overrides.txt">debian/rules missing-overrides</a></tt> or
 441 <br><tt><a href="status-popcon.txt">debian/rules missing-by-popcon</a></tt>
 442 <br>in the source directory to see the current status.
 443
 444 <p>See also wiki pages with documentation and status:
 445
 446 <ul>
 447 <li>http://wiki.debian.org/LSBInitScripts
 448 <li>http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot
 449 <li>Repository: <tt>svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv</tt>
 450 </ul>
 451
 452     <h2>Thank you very much</h2>
 453
 454     <h3>Questions?</h3>
 455
 456     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 457
 458   </body>
 459 </html>