mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9 <style type="text/css">
  10 PRE {
  11   font-size: smaller
  12 }
  13 </style>
  14   </head>
  15   <body>
  16
  17 <a href="../200706-bootseq/200706-bootseq.html">insserv talk during Debconf 7</a>
  18
  19 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering
  20 the Debian boot sequence for correctness and speed</a></h1>
  21
  22 <p>There are subtle bugs in the Debian boot and shutdown sequences.
  23 They are hard to find, as they normally only affect rare combination
  24 of packages.  They are harder to fix, as they normally require the
  25 combined work of several maintainers and changes in several packages.
  26 This talk is about the release goal for Lenny to solve them, and gain
  27 a few advantages on the way.</p>
  28
  29 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  30   <br>pere@hungry.com
  31   <br>FOSDEM 2008, 2008-02-26</div>
  32
  33 <!--
  34
  35 There are subtle bugs in the debian boot and shutdown sequence. They
  36 are hard to find, as they normally only affect rare combination of
  37 packages, and harder to fix, as they normally require the combined
  38 work of several maintainers and changes in several packages.
  39
  40 One way to find these bugs is to document the dependencies of all
  41 init.d scripts, and use this information to check that the ordering is
  42 correct. When such information is available, it is also possible to
  43 reorder the boot and shutdown sequence to make sure all dependencies
  44 are fulfilled.
  45
  46 It is also possible to run scripts in parallel, to speed up the boot,
  47 when the order they need to run in is known.
  48
  49 This talk is about how all of this can be done with Debian.
  50 -->
  51
  52 <h2>Outline</h2>
  53
  54 <ul>
  55 <li>Quick overview of the SysV init boot system
  56 <li>The problem with ordering
  57 <li>Using LSB init.d script dependencies to solve it.
  58 <li>Status of dependency based boot sequencing in Debian.
  59 <li>How to write LSB headers.
  60 <li>What is left to do in Debian.
  61 </ul>
  62
  63 <!--
  64
  65  - how do sysvinit boot work
  66    - runlevels
  67    - inittab
  68    - rc*.d/
  69  - ordering problem
  70  - how does it work in debian
  71  - effect on the boot
  72  - insserv
  73    - how to verify the boot sequence
  74      - check script
  75      - dependency graph using dotty
  76    - how to test it
  77    - overriding included headers
  78  - speed change?
  79  - concurrency
  80
  81  - how to write lsb headers
  82    - provides
  83    - deps
  84    - runlevel list
  85
  86  - status in debian
  87    - need better doc
  88    - fix insserv bugs
  89
  90  - what is needed to convert
  91    - add LSB header to packages and get the change into testing
  92    - update policy
  93    - more users to test headers and detect errors
  94    - install and activate insserv by default
  95
  96 -->
  97
  98 <h2>SysV init in Debian - Booting</h2>
  99
 100 <p>Note, this is the stuff going on after the initrd part is done.
 101 The very early boot is done before hard drive partitions are
 102 mounted.</p>
 103
 104 <ol>is
 105
 106 <li>/sbin/init start, which looks at /etc/inittab and decides what to
 107   do.</li>
 108
 109 <li>The scripts in /etc/rcS.d/  are executed in sequence by
 110   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
 111
 112 <li>Depending on the runlevel, the scripts for the given runlevel are
 113   executed (normally the ones in /etc/rc2.d/) are executed in
 114   sequence, first the stop scripts with <tt>stop</tt> as their
 115   argument, next the start scripts with <tt>start</tt> as their
 116   argument.  The rc*.d/ directories contain symlinks the files in to
 117   /etc/init.d/.</li>
 118
 119 <li>The ordering is important.</li>
 120
 121 <li>Runlevel 1 is not the single user runlevel.  The single-user
 122   runlevel will present a login prompt after rcS.d/ was executed.
 123   Runlevel 1 is not the single user runlevel, but it behaves as a
 124   better single user.</li>
 125
 126 </ol>
 127
 128 <p>Note that because the Linux kernel is becoming more and more event
 129 based, the boot sequence is no longer sequencial.<p>
 130
 131 <h2>SysV init in Debian - Switching runlevels</h2>
 132
 133 <ul>
 134
 135 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 136   1, 2, 3, 4, 5, 6.</li>
 137
 138 <li>init runs all stop scripts in /etc/rcX.d/ that also have a start
 139    symlink in the previous runlevel /etc/rcY.d/ in sequence, with
 140    <tt>stop</tt> as their argument.</li>
 141
 142 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 143   <tt>start</tt> as their argument.</li>
 144
 145 <li>Runlevels 0 and 6 are not behaving as you expect.
 146
 147 </ul>
 148
 149 <p>Note that switching to runlevel S will not run the scripts in
 150 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 151 1.  It will (should) kill all services and prepare the machine for
 152 maintenance.</p>
 153
 154 <h2>SysV init in Debian - Shutting down</h2>
 155
 156 <p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
 157 (reboot).</p>
 158
 159 <p>Minor exception: all scripts (both start and stop) are executed
 160   with the <tt>stop</tt> argument, ignoring their start and stop
 161   settings and confusing script writers.</p>
 162
 163 <p>Only stop scripts for services started in the previous runlevel are
 164   executed.</p>
 165
 166 <h2>The ordering problem</h2>
 167
 168 <p>Script ordering is vital for this to work.  And how are the scripts
 169 ordered?  By numbers!
 170
 171 <p>And the numbers are picked using skills, knowledge and negotiation.
 172 Getting it right is often hard.
 173
 174 <p>The current Debian default is wrong.  The stop sequence should by
 175 default be the reverse of the start sequence.  It isn't.
 176
 177 <p>Reordering is hard and sometimes requires cooperation between
 178 maintainers of all packages involved.
 179
 180 <h2>The ordering problem - an example</h2>
 181
 182 <p>Given two packages with two scripts inserted with the default
 183 settings in Debian:
 184
 185 <p>Package A: script_a sequence 20 (start and stop)
 186 <br>Package B: script_b sequence 20 (start and stop)
 187
 188 <p>Along comes script C, which should run before script_a and after
 189 script_b.  Current solution is to change packages A and C or packages
 190 B and C to get something like this:
 191
 192 <p>Package A: script_a start seq. 22, stop seq. 18
 193 <br>Package B: script_b sequence 20 (start and stop)
 194 <br>Package C: script_c start seq 21, stop seq 19
 195
 196 <p>If other scripts depend on the old order of script_a, they will
 197 have to change their sequence number too.  The only way to discover
 198 this is by a lot of testing, or documenting script dependencies.
 199
 200 <h2>An ordering solution</h2>
 201
 202 <p>Let each script document its dependencies, and generate sequence
 203 numbers using this dependency information.  Example:
 204
 205 <p>Package A: script_a depend on nothing
 206 <br>Package B: script_b depend on nothing
 207 <br>Package C: script_c depend on script_b, a dependency of script_a
 208
 209 <p>Generated sequence:
 210
 211 <p>script_b start seq 1, stop seq 3
 212 <br>script_c start seq 2, stop seq 2
 213 <br>script_a start seq 3, stop seq 1
 214
 215 <p>An implementation of this system is the dependency based boot
 216 sequencing, provided by the insserv package.  It uses the format
 217 specified in Linux Software Base to document init.d script
 218 dependencies.</p>
 219
 220 <h2>Checking the current boot sequence</h2>
 221
 222 Two options are available with the insserv package:
 223
 224 <dl>
 225 <dt>Static checking of current headers:
 226 <tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt>
 227 <dd>report mismatch in current ordering.
 228
 229 <dt>Graph of the dependencies:
 230 <a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a>
 231 <dd>for reviewing dependencies
 232
 233 </dl>
 234
 235 <h2>Enabling dependency based boot sequencing I</h2>
 236
 237 <p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
 238 <ul>
 239   <li>Use the insserv package in unstable or testing
 240   <li>Low priority debconf question.
 241   <li>Only enabled after verfiying that it is safe to enable
 242   <li>Converts all start symlinks in rc0.d/ and rc6.d to stop symlinks.
 243   <li>Replaces update-rc.d with a wrapper calling insserv
 244   <li>Finally reorder symlinks in rc*.d/ using dependencies
 245 </ul>
 246
 247 <h2>Enabling dependency based boot sequencing II</h2>
 248
 249 <p><pre size="-1">
 250 # aptitude install insserv
 251 # dpkg-reconfigure insserv
 252 info: Checking if it is safe to convert to dependency based boot.
 253 info: Backing up existing boot scripts in \
 254   /var/lib/insserv/bootscripts-20080223T0742.tar.gz
 255 info: Reordering boot system, log to \
 256   /var/lib/insserv/run-20080223T0742.log
 257 info: Recording new boot sequence in \
 258   /var/lib/insserv/bootscripts-20080223T0742-after.list
 259 info: Use '/usr/sbin/update-bootsystem-insserv \
 260   restore' to restore the old boot sequence.
 261 Adding `diversion of /usr/sbin/update-rc.d to \
 262   /usr/sbin/update-rc.d.distrib by insserv'
 263 success: Boot system successfully converted
 264 # <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
 265   /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
 266 [...]
 267 #
 268 </pre>
 269
 270 <h2>Using dependency based boot sequencing</h2>
 271
 272 <p>update-rc.d refuses to Insert scripts which create a loop.</p>
 273
 274 <p>update-rc.d requires scripts to be inserted in dependency order.</p>
 275
 276 <p>Incorrect dependencies give the wrong but predictable and stable
 277 (as in the same all the time) boot and shutdown order.</p>
 278
 279 <p>It is possible to enable concurrent booting, running boot scripts
 280   in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
 281
 282 <p>Might even speed up the boot and shutdown (initial benchmark show
 283   speedup only by reordering based on dependencies)</p>
 284
 285 <h2>Disabling dependency based boot sequencing</h2>
 286
 287 <p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
 288
 289 <p>It is always possible to disable just after it was enabled, before
 290 any new packages are installed.</p>
 291
 292 <p>When disabling it, a backup of the old boot sequence is restored if no
 293 changes have been made to the boot sequence since it was enabled.
 294
 295 <p>If restore is not possible, all postinst scripts for packages with
 296 init.d scripts will be executed again to make them call update-rc.d
 297 and add the boot scripts again.
 298
 299 <p>This is guaranteed to work if no packages have been added since it
 300 was enabled, and most often works if packages have been added.  So if
 301 you change your mind, do it quickly.
 302
 303 <h2>LSB headers for insserv</h2>
 304
 305 <ul>
 306   <li>"Provides" header, list the Facility/service provided by the
 307     script.  Do not list virtual facilities (like $time)</li>
 308
 309   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 310     what runlevels to activate for this script
 311
 312   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 313     Should-Stop, X-Start-Before, X-Stop-After), list the
 314     facilities/services needed by this script.  The script will start
 315     after its start dependencies and stop before its stop
 316     dependencies.  The X-* entries are reverse dependencies.
 317     Required-* are hard dependencies (update-rc.d will refuse to
 318     install the script if any of them are missing), while Should-* and
 319     X-* are soft dependencies (only taken into account if scripts
 320     providing them are present).</li>
 321
 322 </ul>
 323
 324 <h2>What to list as dependencies (I)</h2>
 325
 326 <p>If your package used the default update-rc.d settings before, this
 327 is the header for you:</p>
 328 <pre>
 329 ### BEGIN INIT INFO
 330 # Provides:          scriptname
 331 # Required-Start:    $remote_fs $syslog
 332 # Required-Stop:     $remote_fs $syslog
 333 # Default-Start:     2 3 4 5
 334 # Default-Stop:      0 1 6
 335 ### END INIT INFO
 336 </pre>
 337
 338 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 339 is needed only by scripts starting services logging to syslog.</p>
 340
 341 <h2>Virtual facilities</h2>
 342
 343 <p>Linux Software Base version 3.2 defines these virtual
 344 facilities:</p>
 345
 346 <dl>
 347
 348 <dt>$local_fs
 349 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 350
 351 <dt>$network
 352 <dd>basic networking support is available. Example: a server program
 353 could listen on a socket. (In Debian, network interfaces are up)
 354
 355 <dt>$portmap
 356 <dd>daemons providing the SunRPC/ONCRPC portmapping service as defined
 357 in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 358 running.
 359
 360 <dt>$remote_fs
 361 <dd>all remote file systems are available. In some configurations,
 362 file systems such as /usr may be remote. Many applications that
 363 require $local_fs will probably also require $remote_fs. (In Debian,
 364 /usr/ and NFS directories are guaranteed to be mounted)
 365
 366 <dt>$time
 367 <dd>the system time has been set, for example by using a network-based
 368 time program such as ntp or rdate, or via the hardware Real Time
 369 Clock.
 370
 371 <dt>$syslog
 372 <dd>the system logger is operational.
 373
 374 <dt>$named
 375 <dd>IP name-to-address translation, using the interfaces described in
 376 this specification, are available to the level the system normally
 377 provides them. Example: if a DNS query daemon normally provides this
 378 facility, then that daemon has been started.
 379
 380 </dl>
 381
 382 <p>All of these represent points in time during boot and shutdown.
 383
 384 <h2>What to list as dependencies (II)</h2>
 385
 386 <p>Normally, the start and stop dependencies are the same.
 387
 388 <p>Virtual dependencies are preferred over specific dependencies.
 389
 390 <p>When using specific dependencies, use the string listed in the
 391   provides header of the scripts you depend on.
 392
 393 <p>Scripts started in rcS.d/ need extra care.
 394
 395 <p>All scripts not started in rcS.d/ should depend on $remote_fs.
 396 This make sure <tt>/usr/</tt> is available during start and that it is
 397 stopped before <tt>sendsigs</tt> kills all processes during
 398 shutdown.</p>
 399
 400 XXX More
 401
 402 <h2>Status of the dependency based boot system</h2>
 403
 404 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 405
 406 <p>Release goal for Debian Lenny.
 407
 408 <p>Packages with LSB headers (in Sid): 662 of 866 (76%)
 409 <br>Unsolved BTS reports: around 85
 410 <br>Packages without BTS reports: around 150
 411 <br>Last package will be fixed 2008-06-13 at the current rate.
 412
 413 <p>Needs better documentation
 414
 415 <p>Should update Debian policy to reflect dependency based boot
 416   sequencing.
 417
 418 <p>Two insserv bugs to fix:
 419 <ul>
 420 <li>Will sometimes add  unwanted stop symlinks (#XXX).
 421 <li>Will fail fail on "fake" loops. (#XXX)
 422 </ul></p>
 423
 424 <p>
 425
 426 <h2>Tracking status</h2>
 427
 428 debian/rules missing-overrides
 429 debian/rules missing-by-popcon
 430
 431 <h2>References</h2>
 432
 433 http://wiki.debian.org/LSBInitScripts
 434 http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot
 435 svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv
 436
 437     <h2>Thank you very much</h2>
 438
 439     <h3>Questions?</h3>
 440
 441     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 442
 443   </body>
 444 </html>