mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9 <style type="text/css">
  10 PRE {
  11   font-size: smaller
  12 }
  13 </style>
  14   </head>
  15   <body>
  16
  17 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering
  18 the Debian boot sequence for correctness and speed</a></h1>
  19
  20 <p>There are subtle bugs in the Debian boot and shutdown sequences.
  21 They are hard to find, as they normally only affect rare combination
  22 of packages.  They are harder to fix, as they normally require the
  23 combined work of several maintainers and changes in several packages.
  24 This talk is about the release goal for Lenny to solve them, and gain
  25 a few advantages on the way.</p>
  26
  27 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  28   <br>pere@hungry.com
  29   <br>FOSDEM 2008, 2008-02-24</div>
  30
  31 <!--
  32
  33 There are subtle bugs in the debian boot and shutdown sequence. They
  34 are hard to find, as they normally only affect rare combination of
  35 packages, and harder to fix, as they normally require the combined
  36 work of several maintainers and changes in several packages.
  37
  38 One way to find these bugs is to document the dependencies of all
  39 init.d scripts, and use this information to check that the ordering is
  40 correct. When such information is available, it is also possible to
  41 reorder the boot and shutdown sequence to make sure all dependencies
  42 are fulfilled.
  43
  44 It is also possible to run scripts in parallel, to speed up the boot,
  45 when the order they need to run in is known.
  46
  47 This talk is about how all of this can be done with Debian.
  48 -->
  49
  50 <h2>Outline</h2>
  51
  52 <ul>
  53 <li>Quick overview of the SysV init boot system
  54 <li>The problem with ordering
  55 <li>Using LSB init.d script dependencies to solve it.
  56 <li>Status of dependency based boot sequencing.
  57 <li>How to write LSB headers.
  58 <li>What is left to do.
  59 </ul>
  60
  61 <!--
  62
  63  - how do sysvinit boot work
  64    - runlevels
  65    - inittab
  66    - rc*.d/
  67  - ordering problem
  68  - how does it work in debian
  69  - effect on the boot
  70  - insserv
  71    - how to verify the boot sequence
  72      - check script
  73      - dependency graph using dotty
  74    - how to test it
  75    - overriding included headers
  76  - speed change?
  77  - concurrency
  78
  79  - how to write lsb headers
  80    - provides
  81    - deps
  82    - runlevel list
  83
  84  - status in debian
  85    - need better doc
  86    - fix insserv bugs
  87
  88  - what is needed to convert
  89    - add LSB header to packages and get the change into testing
  90    - update policy
  91    - more users to test headers and detect errors
  92    - install and activate insserv by default
  93
  94 -->
  95
  96 <h2>SysV init in Debian - Booting</h2>
  97
  98 <p>Note, this is the stuff going on after the initrd part is done.
  99 The very early boot is done before hard drive partitions are
 100 mounted.</p>
 101
 102 <ol>
 103
 104 <li>/sbin/init stats and looks at /etc/inittab to decide what to
 105   do.</li>
 106
 107 <li>The scripts in /etc/rcS.d/  are executed in sequence by
 108   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
 109
 110 <li>Depending on the runlevel, the scripts for the given runlevel are
 111   executed (normally the ones in /etc/rc2.d/) are executed in
 112   sequence, first the stop scripts with <tt>stop</tt> as their
 113   argument, next the start scripts with <tt>start</tt> as their
 114   argument.  The rc*.d/ directories contain symlinks the files in to
 115   /etc/init.d/.</li>
 116
 117 <li>The ordering is important.</li>
 118
 119 <li>Runlevel 1 is not the single user runlevel.  The single-user
 120   runlevel will present a login prompt after rcS.d/ was executed.
 121   Runlevel 1 is not the single user runlevel, but it behaves as a
 122   better single user.</li>
 123
 124 </ol>
 125
 126 <p>Note that because the Linux kernel is becoming more and more event
 127 based, the boot sequence is no longer sequencial.<p>
 128
 129 <h2>SysV init in Debian - Switching runlevels</h2>
 130
 131 <ul>
 132
 133 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 134   1, 2, 3, 4, 5, 6.</li>
 135
 136 <li>init runs all stop scripts in /etc/rcX.d/ that also have a start
 137    symlink in the previous runlevel /etc/rcY.d/ in sequence, with
 138    <tt>stop</tt> as their argument.</li>
 139
 140 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 141   <tt>start</tt> as their argument.</li>
 142
 143 <li>Runlevels 0 and 6 are not behaving as you would expect.
 144
 145 </ul>
 146
 147 <p>Note that switching to runlevel S will not run the scripts in
 148 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 149 1.  It will (should) kill all services and prepare the machine for
 150 maintenance.</p>
 151
 152 <h2>SysV init in Debian - Shutting down</h2>
 153
 154 <p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
 155 (reboot).</p>
 156
 157 <p>Minor exception: all scripts (both start and stop) are executed
 158   with the <tt>stop</tt> argument, ignoring their start and stop
 159   settings and confusing script writers.</p>
 160
 161 <p>Only stop scripts for services started in the previous runlevel are
 162   executed.</p>
 163
 164 <h2>The ordering problem</h2>
 165
 166 <p>Script ordering is vital for this to work.  And how are the scripts
 167 ordered?  By numbers 01-99!
 168
 169 <p>And the numbers are picked using skills, knowledge and negotiation.
 170 Getting it right is often hard.
 171
 172 <p>The current Debian default is wrong.  The stop sequence should by
 173 default be the reverse of the start sequence.  It isn't.  The default
 174 uses '20' for both.
 175
 176 <p>Reordering is hard and sometimes requires cooperation between
 177 maintainers of all packages involved.
 178
 179 <h2>The ordering problem - an example</h2>
 180
 181 <p>Given two packages with two scripts inserted with the default
 182 settings in Debian:
 183
 184 <p>Package A: script_a sequence 20 (start and stop)
 185 <br>Package B: script_b sequence 20 (start and stop)
 186
 187 <p>Along comes script C, which should run before script_a and after
 188 script_b.  Current solution is to change packages A and C or packages
 189 B and C to get something like this:
 190
 191 <p>Package A: script_a start seq. 22, stop seq. 18
 192 <br>Package B: script_b sequence 20 (start and stop)
 193 <br>Package C: script_c start seq 21, stop seq 19
 194
 195 <p>If other scripts depend on the old order of script_a, they will
 196 have to change their sequence number too.  The only way to discover
 197 this is by a lot of testing, or documenting script dependencies.
 198
 199 <h2>An ordering solution</h2>
 200
 201 <p>Let each script document its dependencies, and generate sequence
 202 numbers using this dependency information.  Example:
 203
 204 <p>Package A: script_a depend on nothing
 205 <br>Package B: script_b depend on nothing
 206 <br>Package C: script_c depend on script_b, a dependency of script_a
 207
 208 <p>Generated sequence:
 209
 210 <p>script_b start seq 1, stop seq 3
 211 <br>script_c start seq 2, stop seq 2
 212 <br>script_a start seq 3, stop seq 1
 213
 214 <p>An implementation of this system is the dependency based boot
 215 sequencing, provided by the insserv package.  It uses the format
 216 specified in Linux Software Base to document init.d script
 217 dependencies.</p>
 218
 219 <h2>Checking the current boot sequence</h2>
 220
 221 Two options are available with the insserv package:
 222
 223 <dl>
 224 <dt>Static checking of current headers:
 225 <br><tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt>
 226 <dd>report mismatch in current ordering.
 227
 228 <dt>Graph of the dependencies:
 229 <br><a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a>
 230 <dd>for reviewing dependencies with dotty from graphviz.
 231
 232 </dl>
 233
 234 <h2>Enabling dependency based boot sequencing I</h2>
 235
 236 <p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
 237 <ul>
 238   <li>Use the insserv package in unstable or testing
 239   <li>Low priority debconf question.
 240   <li>Only enabled after verfiying that it is safe to enable
 241   <li>Converts all start symlinks in rc0.d/ and rc6.d to stop symlinks.
 242   <li>Replaces update-rc.d with a wrapper calling insserv
 243   <li>Finally reorder symlinks in rc*.d/ using dependencies
 244 </ul>
 245
 246 <p>Will refuse to enable when obsolete init.d scripts, loops,
 247 duplicate provides etc is detected</p>
 248
 249 <h2>Enabling dependency based boot sequencing II</h2>
 250
 251 <p><pre>
 252 # aptitude install insserv
 253 # dpkg-reconfigure insserv
 254 info: Checking if it is safe to convert to dependency based boot.
 255 info: Backing up existing boot scripts in \
 256   /var/lib/insserv/bootscripts-20080223T0742.tar.gz
 257 info: Reordering boot system, log to \
 258   /var/lib/insserv/run-20080223T0742.log
 259 info: Recording new boot sequence in \
 260   /var/lib/insserv/bootscripts-20080223T0742-after.list
 261 info: Use '/usr/sbin/update-bootsystem-insserv \
 262   restore' to restore the old boot sequence.
 263 Adding `diversion of /usr/sbin/update-rc.d to \
 264   /usr/sbin/update-rc.d.distrib by insserv'
 265 success: Boot system successfully converted
 266 # <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
 267   /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
 268 [...]
 269 #
 270 </pre>
 271
 272 <h2>Using dependency based boot sequencing</h2>
 273
 274 <p>update-rc.d refuses to Insert scripts which create a loop.</p>
 275
 276 <p>update-rc.d requires scripts to be inserted in dependency order.</p>
 277
 278 <p>Incorrect dependencies give the wrong but predictable and stable
 279 (as in the same all the time) boot and shutdown order.</p>
 280
 281 <p>Every insserv upload are checked using test suite to make sure the
 282 generated boot sequence is correct, and that previously detected bugs
 283 do not show up again.</p>
 284
 285 <p>It is possible to enable concurrent booting, running boot scripts
 286   in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
 287
 288 <p>Might even speed up boot and shutdown (initial benchmark shows
 289   speedup by just reordering based on dependencies).</p>
 290
 291 <h2>Disabling dependency based boot sequencing</h2>
 292
 293 <p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
 294
 295 <p>It is always possible to disable just after it was enabled, before
 296 any new packages are installed.</p>
 297
 298 <p>When disabling it, a backup of the old boot sequence is restored if no
 299 changes have been made to the boot sequence since it was enabled.
 300
 301 <p>If restore is not possible, all postinst scripts for packages with
 302 init.d scripts will be executed again to make them call update-rc.d
 303 and add the boot scripts again.
 304
 305 <p>This is guaranteed to work if no packages have been added since it
 306 was enabled, and most often works if packages have been added.  So if
 307 you change your mind, do it quickly.
 308
 309 <h2>LSB headers for insserv</h2>
 310
 311 <ul>
 312   <li>"Provides" header, list the Facility/service provided by the
 313     script.  Do not list virtual facilities (like $time)</li>
 314
 315   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 316     what runlevels to activate for this script
 317
 318   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 319     Should-Stop, X-Start-Before, X-Stop-After), list the
 320     facilities/services needed by this script.  The script will start
 321     after its start dependencies and stop before its stop
 322     dependencies.  The X-* entries are reverse dependencies.
 323     Required-* are hard dependencies (update-rc.d will refuse to
 324     install the script if any of them are missing), while Should-* and
 325     X-* are soft dependencies (only taken into account if scripts
 326     providing them are present).</li>
 327
 328 </ul>
 329
 330 <h2>What to list as dependencies (I)</h2>
 331
 332 <p>If your package used the default update-rc.d settings before, this
 333 is the header for you:</p>
 334 <pre>
 335 ### BEGIN INIT INFO
 336 # Provides:          scriptname
 337 # Required-Start:    $remote_fs $syslog
 338 # Required-Stop:     $remote_fs $syslog
 339 # Default-Start:     2 3 4 5
 340 # Default-Stop:      0 1 6
 341 ### END INIT INFO
 342 </pre>
 343
 344 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 345 is needed only by scripts starting services logging to syslog.</p>
 346
 347 <h2>Virtual facilities (I)</h2>
 348
 349 <p>Linux Software Base version 3.2 defines these virtual
 350 facilities:</p>
 351
 352 <dl>
 353
 354 <dt>$local_fs
 355 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 356
 357 <dt>$network
 358 <dd>basic networking support is available. Example: a server program
 359 could listen on a socket. (In Debian, network interfaces are up)
 360
 361 <dt>$portmap
 362 <dd>daemons providing the SunRPC/ONCRPC portmapping service as defined
 363 in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 364 running.
 365
 366 <dt>$remote_fs
 367 <dd>all remote file systems are available. In some configurations,
 368 file systems such as /usr may be remote. Many applications that
 369 require $local_fs will probably also require $remote_fs. (In Debian,
 370 /usr/ and NFS directories are guaranteed to be mounted)
 371 </dl>
 372
 373 <h2>Virtual facilities (II)</h2>
 374
 375 <dl>
 376
 377 <dt>$time
 378 <dd>the system time has been set, for example by using a network-based
 379 time program such as ntp or rdate, or via the hardware Real Time
 380 Clock.
 381
 382 <dt>$syslog
 383 <dd>the system logger is operational.
 384
 385 <dt>$named
 386 <dd>IP name-to-address translation, using the interfaces described in
 387 this specification, are available to the level the system normally
 388 provides them. Example: if a DNS query daemon normally provides this
 389 facility, then that daemon has been started.
 390
 391 </dl>
 392
 393 <p>All of these represent points in time during boot and shutdown.
 394 Virtual facilities are defined in <tt>/etc/insserv.conf</tt> and
 395 <tt>/etc/insserv.conf.d/</tt>
 396
 397 <h2>What to list as dependencies (II)</h2>
 398
 399 <p>Normally, the start and stop dependencies are the same.
 400
 401 <p>Virtual dependencies are preferred over specific dependencies.
 402
 403 <p>When using specific dependencies, use the string listed in the
 404   provides header of the scripts you depend on.
 405
 406 <p>Scripts started in rcS.d/ need extra care.
 407
 408 <p>All scripts not started in rcS.d/ should depend on $remote_fs.
 409 This make sure <tt>/usr/</tt> is available during start and that it is
 410 stopped before <tt>sendsigs</tt> kills all processes during
 411 shutdown.</p>
 412
 413 <p>A sysadmin can provide overrides in
 414 <tt>/etc/insserv/overrides/scriptname</tt> if the script settings are
 415 wrong.  The insserv package provides overrides in
 416 <tt>/usr/share/insserv/overrides/<tt> for packages currently missing
 417 headers.</p>
 418
 419 <h2>Handling alternatives</h2>
 420
 421 <p>Not quite tested yet.
 422
 423 <p>Should perhaps be handled using virtual facilities in
 424 /etc/insserv.conf.d/.
 425
 426 <p>Do not listing identical provides in several scripts - break
 427 installation.
 428
 429 <h2>Status of the dependency based boot system</h2>
 430
 431 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 432
 433 <p>Release goal for Lenny.
 434 <br>76% of packages got LSB headers.
 435 <br>Unsolved in BTS: ~85
 436 <br>Without BTS reports: ~150
 437 <br>Last package will be fixed 2008-06-15 at the current rate.
 438 <br>Needs better documentation for maintainers.
 439 <br>Should update Debian policy to reflect dependency based boot
 440   sequencing.
 441
 442 <p>Two insserv bugs left to fix:
 443 <ul>
 444 <li>Might add unwanted stop symlinks (<a href="http://bugs.debian.org/464017">#464017</a>).
 445 <li>Will fail on "fake" loops as well as real loops. (<a href="http://bugs.debian.org/458582">#458582</a>)
 446 </ul></p>
 447
 448 <h2>Tracking status</h2>
 449
 450 <p>The current status is tracked in the svn repository for insserv.  Run
 451 <br><tt><a href="status-overrides.txt">debian/rules missing-overrides</a></tt> or
 452 <br><tt><a href="status-popcon.txt">debian/rules missing-by-popcon</a></tt>
 453 <br>in the source directory to see the current status.
 454
 455 <p>See also wiki pages with documentation and status:
 456
 457 <ul>
 458 <li><a href="http://wiki.debian.org/LSBInitScripts">http://wiki.debian.org/LSBInitScripts</a>
 459 <li><a href="http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot">http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot</a>
 460 <li>Repository: <tt>svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv</tt>
 461
 462 </ul>
 463
 464 <p>There is also slides from my
 465 <a href="../200706-bootseq/200706-bootseq.html">talk from Debconf
 466 7</a>.<p>
 467
 468 <p>More man-power is needed to report BTS reports, NMU packages and
 469 discover dependency errors.  Please test the system.
 470
 471     <h2>Thank you very much</h2>
 472
 473     <h3>Questions?</h3>
 474
 475     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 476
 477   </body>
 478 </html>