mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9 <style type="text/css">
  10 PRE {
  11   font-size: smaller
  12 }
  13 </style>
  14   </head>
  15   <body>
  16
  17 <a href="../200706-bootseq/200706-bootseq.html">insserv talk during Debconf 7</a>
  18
  19 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering
  20 the Debian boot sequence for correctness and speed</a></h1>
  21
  22 <p>There are subtle bugs in the Debian boot and shutdown sequences.
  23 They are hard to find, as they normally only affect rare combination
  24 of packages.  They are harder to fix, as they normally require the
  25 combined work of several maintainers and changes in several packages.
  26 This talk is about the release goal for Lenny to solve them, and gain
  27 a few advantages on the way.</p>
  28
  29 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  30   <br>pere@hungry.com
  31   <br>FOSDEM 2008, 2008-02-26</div>
  32
  33 <!--
  34
  35 There are subtle bugs in the debian boot and shutdown sequence. They
  36 are hard to find, as they normally only affect rare combination of
  37 packages, and harder to fix, as they normally require the combined
  38 work of several maintainers and changes in several packages.
  39
  40 One way to find these bugs is to document the dependencies of all
  41 init.d scripts, and use this information to check that the ordering is
  42 correct. When such information is available, it is also possible to
  43 reorder the boot and shutdown sequence to make sure all dependencies
  44 are fulfilled.
  45
  46 It is also possible to run scripts in parallel, to speed up the boot,
  47 when the order they need to run in is known.
  48
  49 This talk is about how all of this can be done with Debian.
  50 -->
  51
  52 <h2>Outline</h2>
  53
  54 <ul>
  55 <li>Quick overview of the SysV init boot system
  56 <li>The problem with ordering
  57 <li>Using LSB init.d script dependencies to solve it.
  58 <li>Status of dependency based boot sequencing in Debian.
  59 <li>How to write LSB headers.
  60 <li>What is left to do in Debian.
  61 </ul>
  62
  63 <!--
  64
  65  - how do sysvinit boot work
  66    - runlevels
  67    - inittab
  68    - rc*.d/
  69  - ordering problem
  70  - how does it work in debian
  71  - effect on the boot
  72  - insserv
  73    - how to verify the boot sequence
  74      - check script
  75      - dependency graph using dotty
  76    - how to test it
  77    - overriding included headers
  78  - speed change?
  79  - concurrency
  80
  81  - how to write lsb headers
  82    - provides
  83    - deps
  84    - runlevel list
  85
  86  - status in debian
  87    - need better doc
  88    - fix insserv bugs
  89
  90  - what is needed to convert
  91    - add LSB header to packages and get the change into testing
  92    - update policy
  93    - more users to test headers and detect errors
  94    - install and activate insserv by default
  95
  96 -->
  97
  98 <h2>SysV init in Debian - Booting</h2>
  99
 100 <p>Note, this is the stuff going on after the initrd part is done.
 101 The very early boot is done before hard drive partitions are
 102 mounted.</p>
 103
 104 <ol>is
 105
 106 <li>/sbin/init start, which looks at /etc/inittab and decides what to
 107   do.</li>
 108
 109 <li>The scripts in /etc/rcS.d/  are executed in sequence by
 110   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
 111
 112 <li>Depending on the runlevel, the scripts for the given runlevel are
 113   executed (normally the ones in /etc/rc2.d/) are executed in
 114   sequence, first the stop scripts with <tt>stop</tt> as their
 115   argument, next the start scripts with <tt>start</tt> as their
 116   argument.  The rc*.d/ directories contain symlinks the files in to
 117   /etc/init.d/.</li>
 118
 119 <li>The ordering is important.</li>
 120
 121 <li>Runlevel 1 is not the single user runlevel.  The single-user
 122   runlevel will present a login prompt after rcS.d/ was executed.
 123   Runlevel 1 is not the single user runlevel, but it behaves as a
 124   better single user.</li>
 125
 126 </ol>
 127
 128 <p>Note that because the Linux kernel is becoming more and more event
 129 based, the boot sequence is no longer sequencial.<p>
 130
 131 <h2>SysV init in Debian - Switching runlevels</h2>
 132
 133 <ul>
 134
 135 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 136   1, 2, 3, 4, 5, 6.</li>
 137
 138 <li>init runs all stop scripts in /etc/rcX.d/ that also have a start
 139    symlink in the previous runlevel /etc/rcY.d/ in sequence, with
 140    <tt>stop</tt> as their argument.</li>
 141
 142 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 143   <tt>start</tt> as their argument.</li>
 144
 145 <li>Runlevels 0 and 6 are not behaving as you expect.
 146
 147 </ul>
 148
 149 <p>Note that switching to runlevel S will not run the scripts in
 150 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 151 1.  It will (should) kill all services and prepare the machine for
 152 maintenance.</p>
 153
 154 <h2>SysV init in Debian - Shutting down</h2>
 155
 156 <p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
 157 (reboot).</p>
 158
 159 <p>Minor exception: all scripts (both start and stop) are executed
 160   with the <tt>stop</tt> argument, ignoring their start and stop
 161   settings and confusing script writers.</p>
 162
 163 <p>Only stop scripts for services started in the previous runlevel are
 164   executed.</p>
 165
 166 <h2>The ordering problem</h2>
 167
 168 <p>Script ordering is vital for this to work.  And how are the scripts
 169 ordered?  By numbers!
 170
 171 <p>And the numbers are picked using skills, knowledge and negotiation.
 172 Getting it right is often hard.
 173
 174 <p>The current Debian default is wrong.  The stop sequence should by
 175 default be the reverse of the start sequence.  It isn't.
 176
 177 <p>Reordering is hard and sometimes requires cooperation between
 178 maintainers of all packages involved.
 179
 180 <h2>The ordering problem - an example</h2>
 181
 182 <p>Given two packages with two scripts inserted with the default
 183 settings in Debian:
 184
 185 <p>Package A: script_a sequence 20 (start and stop)
 186 <br>Package B: script_b sequence 20 (start and stop)
 187
 188 <p>Along comes script C, which should run before script_a and after
 189 script_b.  Current solution is to change packages A and C or packages
 190 B and C to get something like this:
 191
 192 <p>Package A: script_a start seq. 22, stop seq. 18
 193 <br>Package B: script_b sequence 20 (start and stop)
 194 <br>Package C: script_c start seq 21, stop seq 19
 195
 196 <p>If other scripts depend on the old order of script_a, they will
 197 have to change their sequence number too.  The only way to discover
 198 this is by a lot of testing, or documenting script dependencies.
 199
 200 <h2>An ordering solution</h2>
 201
 202 <p>Let each script document its dependencies, and generate sequence
 203 numbers using this dependency information.  Example:
 204
 205 <p>Package A: script_a depend on nothing
 206 <br>Package B: script_b depend on nothing
 207 <br>Package C: script_c depend on script_b, a dependency of script_a
 208
 209 <p>Generated sequence:
 210
 211 <p>script_b start seq 1, stop seq 3
 212 <br>script_c start seq 2, stop seq 2
 213 <br>script_a start seq 3, stop seq 1
 214
 215 <p>An implementation of this system is the dependency based boot
 216 sequencing, provided by the insserv package.  It uses the format
 217 specified in Linux Software Base to document init.d script
 218 dependencies.</p>
 219
 220 <h2>Checking the current boot sequence</h2>
 221
 222 Two options are available with the insserv package:
 223
 224 <dl>
 225 <dt>Static checking of current headers:
 226 <tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt>
 227 <dd>report mismatch in current ordering.
 228
 229 <dt>Graph of the dependencies:
 230 <a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a>
 231 <dd>for reviewing dependencies
 232
 233 </dl>
 234
 235 <h2>Enabling dependency based boot sequencing I</h2>
 236
 237 <p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
 238 <ul>
 239   <li>Use the insserv package in unstable or testing
 240   <li>Low priority debconf question.
 241   <li>Only enabled after verfiying that it is safe to enable
 242   <li>Converts all start symlinks in rc0.d/ and rc6.d to stop symlinks.
 243   <li>Replaces update-rc.d with a wrapper calling insserv
 244   <li>Finally reorder symlinks in rc*.d/ using dependencies
 245 </ul>
 246
 247 <h2>Enabling dependency based boot sequencing II</h2>
 248
 249 <p><pre size="-1">
 250 # aptitude install insserv
 251 # dpkg-reconfigure insserv
 252 info: Checking if it is safe to convert to dependency based boot.
 253 info: Backing up existing boot scripts in \
 254   /var/lib/insserv/bootscripts-20080223T0742.tar.gz
 255 info: Reordering boot system, log to \
 256   /var/lib/insserv/run-20080223T0742.log
 257 info: Recording new boot sequence in \
 258   /var/lib/insserv/bootscripts-20080223T0742-after.list
 259 info: Use '/usr/sbin/update-bootsystem-insserv \
 260   restore' to restore the old boot sequence.
 261 Adding `diversion of /usr/sbin/update-rc.d to \
 262   /usr/sbin/update-rc.d.distrib by insserv'
 263 success: Boot system successfully converted
 264 # <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
 265   /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
 266 [...]
 267 #
 268 </pre>
 269
 270 <h2>Using dependency based boot sequencing</h2>
 271
 272 <p>update-rc.d refuses to Insert scripts which create a loop.</p>
 273
 274 <p>update-rc.d requires scripts to be inserted in dependency order.</p>
 275
 276 <p>Incorrect dependencies give the wrong boot and shutdown order.</p>
 277
 278 <p>It is possible to enable concurrent booting, running boot scripts
 279   in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
 280
 281 <p>Might even speed up the boot and shutdown (initial benchmark show
 282
 283 <h2>Disabling dependency based boot sequencing</h2>
 284
 285 <p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
 286
 287 <p>It is always possible to disable just after it was enabled, before
 288 any new packages are installed.</p>
 289
 290 <p>When disabling it, a backup of the old boot sequence is restored if no
 291 changes have been made to the boot sequence since it was enabled.
 292
 293 <p>If restore is not possible, all postinst scripts for packages with
 294 init.d scripts will be executed again to make them call update-rc.d
 295 and add the boot scripts again.
 296
 297 <p>This is guaranteed to work if no packages have been added since it
 298 was enabled, and most often works if packages have been added.  So if
 299 you change your mind, do it quickly.
 300
 301 <h2>LSB headers for insserv</h2>
 302
 303 <ul>
 304   <li>"Provides" header, list the Facility/service provided by the
 305     script.  Do not list virtual facilities (like $time)</li>
 306
 307   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 308     what runlevels to activate for this script
 309
 310   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 311     Should-Stop, X-Start-Before, X-Stop-After), list the
 312     facilities/services needed by this script.  The script will start
 313     after its start dependencies and stop before its stop
 314     dependencies.  The X-* entries are reverse dependencies.
 315     Required-* are hard dependencies (update-rc.d will refuse to
 316     install the script if any of them are missing), while Should-* and
 317     X-* are soft dependencies (only taken into account if scripts
 318     providing them are present).</li>
 319
 320 </ul>
 321
 322 <h2>What to list as dependencies (I)</h2>
 323
 324 <p>If your package used the default update-rc.d settings before, this
 325 is the header for you:</p>
 326 <pre>
 327 ### BEGIN INIT INFO
 328 # Provides:          scriptname
 329 # Required-Start:    $remote_fs $syslog
 330 # Required-Stop:     $remote_fs $syslog
 331 # Default-Start:     2 3 4 5
 332 # Default-Stop:      0 1 6
 333 ### END INIT INFO
 334 </pre>
 335
 336 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 337 is needed only by scripts starting services logging to syslog.</p>
 338
 339 <h2>Virtual facilities</h2>
 340
 341 <p>Linux Software Base version 3.2 defines these virtual
 342 facilities:</p>
 343
 344 <dl>
 345
 346 <dt>$local_fs
 347 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 348
 349 <dt>$network
 350 <dd>basic networking support is available. Example: a server program
 351 could listen on a socket. (In Debian, network interfaces are up)
 352
 353 <dt>$portmap
 354 <dd>daemons providing the SunRPC/ONCRPC portmapping service as defined
 355 in RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 356 running.
 357
 358 <dt>$remote_fs
 359 <dd>all remote file systems are available. In some configurations,
 360 file systems such as /usr may be remote. Many applications that
 361 require $local_fs will probably also require $remote_fs. (In Debian,
 362 /usr/ and NFS directories are guaranteed to be mounted)
 363
 364 <dt>$time
 365 <dd>the system time has been set, for example by using a network-based
 366 time program such as ntp or rdate, or via the hardware Real Time
 367 Clock.
 368
 369 <dt>$syslog
 370 <dd>the system logger is operational.
 371
 372 <dt>$named
 373 <dd>IP name-to-address translation, using the interfaces described in
 374 this specification, are available to the level the system normally
 375 provides them. Example: if a DNS query daemon normally provides this
 376 facility, then that daemon has been started.
 377
 378 </dl>
 379
 380 <p>All of these represent points in time during boot and shutdown.
 381
 382 <h2>What to list as dependencies (II)</h2>
 383
 384 <p>Normally, the start and stop dependencies are the same.
 385
 386 <p>Virtual dependencies are preferred over specific dependencies.
 387
 388 <p>When using specific dependencies, use the string listed in the
 389   provides header of the scripts you depend on.
 390
 391 <p>Scripts started in rcS.d/ need extra care.
 392
 393 <p>All scripts not started in rcS.d/ should depend on $remote_fs.
 394 This make sure <tt>/usr/</tt> is available during start and that it is
 395 stopped before <tt>sendsigs</tt> kills all processes during
 396 shutdown.</p>
 397
 398 XXX More
 399
 400 <h2>Status of the dependency based boot system</h2>
 401
 402 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 403
 404 <p>Release goal for Debian Lenny.
 405
 406 <p>Packages with LSB headers (in Sid): 662 of 866 (76%)
 407 <br>Unsolved BTS reports: around 85
 408 <br>Packages without BTS reports: around 150
 409 <br>Last package will be fixed 2008-06-13 at the current rate.
 410
 411 <p>Needs better documentation
 412
 413 <p>Should update Debian policy to reflect dependency based boot
 414   sequencing.
 415
 416 <p>Two insserv bugs to fix:
 417 <ul>
 418 <li>Will sometimes add  unwanted stop symlinks (#XXX).
 419 <li>Will fail fail on "fake" loops. (#XXX)
 420 </ul></p>
 421
 422 <p>
 423
 424 <h2>Tracking status</h2>
 425
 426 debian/rules missing-overrides
 427 debian/rules missing-by-popcon
 428
 429 <h2>References</h2>
 430
 431 http://wiki.debian.org/LSBInitScripts
 432 http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot
 433 svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv
 434
 435     <h2>Thank you very much</h2>
 436
 437     <h3>Questions?</h3>
 438
 439     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 440
 441   </body>
 442 </html>