mypapers/200802-bootsequence/200802-bootsequence.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   2 <html>
   3   <head>
   4   <title>Talk: Reordering the Debian boot sequence for correctness and speed</title>
   5     <link rel="stylesheet" href="../mrtg-td/slides.css" type="text/css">
   6     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   7     <meta name="Language" content="en">
   8     <meta name="Author" content="Petter Reinholdtsen">
   9 <style type="text/css">
  10 PRE {
  11   font-size: smaller
  12 }
  13 </style>
  14   </head>
  15   <body>
  16
  17 <a href="../200706-bootseq/200706-bootseq.html">last related talk</a>
  18
  19 <h1><A href="http://www.fosdem.org/2008/schedule/events/debian_boot">Reordering the Debian boot sequence for correctness and speed</a></h1>
  20
  21 <p>There are subtle bugs in the Debian boot and shutdown sequence.
  22 They are hard to find, as they normally only affect rare combination
  23 of packages, and harder to fix, as they normally require the combined
  24 work of several maintainers and changes in several packages.  This
  25 talk is about the release goal for Lenny to solve them, and gain a few
  26 advantages on the way.</p>
  27
  28 <div class="presenter">Petter Reinholdtsen - one of the sysvinit maintainers
  29   <br>pere@hungry.com
  30   <br>FOSDEM 2008, 2008-02-26</div>
  31
  32 <!--
  33
  34 There are subtle bugs in the debian boot and shutdown sequence. They
  35 are hard to find, as they normally only affect rare combination of
  36 packages, and harder to fix, as they normally require the combined
  37 work of several maintainers and changes in several packages.
  38
  39 One way to find these bugs is to document the dependencies of all
  40 init.d scripts, and use this information to check that the ordering is
  41 correct. When such information is available, it is also possible to
  42 reorder the boot and shutdown sequence to make sure all dependencies
  43 are fulfilled.
  44
  45 It is also possible to run scripts in parallel, to speed up the boot,
  46 when the order they need to run in is known.
  47
  48 This talk is about how all of this can be done with Debian.
  49 -->
  50
  51 <h2>Outline</h2>
  52
  53 <!--
  54
  55  - how do sysvinit boot work
  56    - runlevels
  57    - inittab
  58    - rc*.d/
  59  - ordering problem
  60  - how does it work in debian
  61  - effect on the boot
  62  - insserv
  63    - how to verify the boot sequence
  64      - check script
  65      - dependency graph using dotty
  66    - how to test it
  67    - overriding included headers
  68  - speed change?
  69  - concurrency
  70
  71  - how to write lsb headers
  72    - provides
  73    - deps
  74    - runlevel list
  75
  76  - status in debian
  77    - need better doc
  78    - fix insserv bugs
  79
  80  - what is needed to convert
  81    - add LSB header to packages and get the change into testing
  82    - update policy
  83    - more users to test headers and detect errors
  84    - install and activate insserv by default
  85
  86 -->
  87
  88 <h2>SysV init in Debian - Booting</h2>
  89
  90 <p>Note, this is the stuff going on after the initrd part is done.
  91 The very early boot is done before hard drive partitions is mounted.</p>
  92
  93 <ol>
  94
  95 <li>/sbin/init start, which look at /etc/inittab and decides what to
  96   do.</li>
  97
  98 <li>The scripts in /etc/rcS.d/ is executed in sequence by
  99   /etc/init.d/rc, with <tt>start</tt> as the argument.</li>
 100
 101 <li>Depending on the runlevel, the scripts for the given runlevel is
 102   executed (normally the ones in /etc/rc2.d/) are executed in
 103   sequence, first the stop scripts with <tt>stop</tt> as their
 104   argument, next the start scripts with <tt>start</tt> as their
 105   argument.  The rc*.d/ directories contain symlinks the files in to
 106   /etc/init.d/.</li>
 107
 108 <li>The ordering is important.</li>
 109
 110 <li>The single-user runlevel will present a login prompt after rcS.d/
 111   was executed.  Runlevel 1 is not the single user runlevel, but it
 112   behaves as a better single user.</li>
 113
 114 </ol>
 115
 116 <p>Note that because the Linux kernel is becoming more and more event
 117 based, the boot sequence is no longer sequencial.<p>
 118
 119 <h2>SysV init in Debian - Switching runlevels</h2>
 120
 121 <ul>
 122
 123 <li>Call <tt>telinit X</tt> where X is the new runlevel, one of S, 0,
 124   1, 2, 3, 4, 5, 6.</li>
 125
 126 <li>init runs in sequence all stop scripts in /etc/rcX.d/ that also
 127    has a start symlink in the previous runlevel /etc/rcY.d/, with
 128    <tt>stop</tt> as their argument.</li>
 129
 130 <li>init then run in sequence all start scripts in /etc/rcX.d/, with
 131   <tt>start</tt> as their argument.</li>
 132
 133 </ul>
 134
 135 <p>Note that switching to runlevel S will not run the scripts in
 136 /etc/rcS.d/.  To get a similar effect after boot, switch to runlevel
 137 1.  It will (should) kill all services and prepare the machine for
 138 maintenance.</p>
 139
 140 <h2>SysV init in Debian - Shutting down</h2>
 141
 142 <p>This is roughtly equivalent to switching to runlevel 0 (halt) or 6
 143 (reboot).</p>
 144
 145 <p>Minor exception: all scripts (both start and stop) are executed
 146   with the <tt>stop</tt> argument, ignoring their start and stop
 147   setting and confusing script writers.</p>
 148
 149 <p>Only stop scripts for services started in the previous runlevel is
 150   executed.</p>
 151
 152 <h2>The ordering problem</h2>
 153
 154 <p>Script ordering is vital for this to work.  And how are the scripts
 155 ordered?  By numbers.
 156
 157 <p>And the numbers are picked using skills, knowledge and negotiation.
 158 Getting it right is often hard.
 159
 160 <p>The current Debian default is wrong.  Stop sequence should by
 161 default be the reverse of the start sequence.  It isn't.
 162
 163 <p>Reordering is hard and require cooperation between maintainers of
 164 all packages involved.
 165
 166 <h2>The ordering problem - an example</h2>
 167
 168 <p>Given two packages with two scripts inserted with the default
 169 settings in Debian:
 170
 171 <p>Package A: script_a sequence 20 (start and stop)
 172 <br>Package B: script_b sequence 20 (start and stop)
 173
 174 <p>Along come script C, which should run before script_a and after
 175 script_b.  Current solution is to change packages A and C or packages
 176 B and C to get something like this:
 177
 178 <p>Package A: script_a start seq. 22, stop seq. 18
 179 <br>Package B: script_b sequence 20 (start and stop)
 180 <br>Package C: script_c start seq 21, stop seq 19
 181
 182 <p>If other scripts depend on the old order of script_a, they will
 183 have to change their sequence number too.  Only way to discover this
 184 is by a lot of testing, or documenting script dependencies.
 185
 186 <h2>An ordering solution</h2>
 187
 188 <p>Let each script document its dependency, and generate sequence
 189 numbers using this dependency information.  Example:
 190
 191 <p>Package A: script_a depend on nothing
 192 <br>Package B: script_b depend on nothing
 193 <br>Package C: script_c depend on script_b, a dependency of script_a
 194
 195 <p>Generated sequence:
 196
 197 <p>script_b start seq 1, stop seq 3
 198 <br>script_c start seq 2, stop seq 2
 199 <br>script_a start seq 3, stop seq 1
 200
 201 <p>An implementation of this system is the dependency based boot
 202 sequencing, provided in the insserv package.  Uses format specified in
 203 Linux Software Base to document dependencies.</p>
 204
 205 <h2>Checking the current boot sequence</h2>
 206
 207 <p>Static checking of current headers:
 208 <tt>/usr/share/insserv/check-initd-order [-o] [-k]</tt> - report mismatch
 209 in current ordering.
 210
 211 <p>Graph of the dependencies:
 212 <a href="sid-base-boot-20080220.dot"><tt>/usr/share/insserv/check-initd-order -g</tt></a> - for reviewing dependencies
 213
 214 <h2>Enabling dependency based boot sequencing I</h2>
 215
 216 <p><a href="insserv-enable.png"><img align="right" src="insserv-enable.png" width="50%"></a>
 217 <ul>
 218   <li>Use the insserv package in unstable and testing
 219   <li>Low priority debconf question.
 220   <li>Only enabled after verfiying that it is safe to enable
 221   <li>Converts all start symlinks in rc0.d/ and rc6.d with stop symlinks.
 222   <li>Replaces update-rc.d with wrapper calling insserv
 223   <li>All of rc*.d/ is reordered
 224 </ul>
 225
 226 <h2>Enabling dependency based boot sequencing II</h2>
 227
 228 <p><pre size="-1">
 229 # aptitude install insserv
 230 # dpkg-reconfigure insserv
 231 info: Checking if it is safe to convert to dependency based boot.
 232 info: Backing up existing boot scripts in \
 233   /var/lib/insserv/bootscripts-20080223T0742.tar.gz
 234 info: Reordering boot system, log to \
 235   /var/lib/insserv/run-20080223T0742.log
 236 info: Recording new boot sequence in \
 237   /var/lib/insserv/bootscripts-20080223T0742-after.list
 238 info: Use '/usr/sbin/update-bootsystem-insserv \
 239   restore' to restore the old boot sequence.
 240 Adding `diversion of /usr/sbin/update-rc.d to \
 241   /usr/sbin/update-rc.d.distrib by insserv'
 242 success: Boot system successfully converted
 243 # <a href="sid-base-seqchanges.txt">/var/lib/insserv/insserv-seq-changes \
 244   /var/lib/insserv/bootscripts-20080223T0742.tar.gz</a>
 245 #
 246 </pre>
 247
 248 <h2>With dependency based boot sequencing</h2>
 249
 250 <p>update-rc.d refuse to Insert scripts which create a loop.</p>
 251
 252 <p>update-rc.d require scripts to be inserted in dependency order.</p>
 253
 254 <p>Incorrect dependencies give the wrong boot and shutdown order.</p>
 255
 256 <p>It is possible to enable concurrent booting, running boot scripts
 257   in parallel (CONCURRENCY=startpar in /etc/default/rcS)</p>
 258
 259 <h2>Disabling dependency based boot sequencing</h2>
 260
 261 <p>Run <tt>dpkg-reconfigure insserv</tt> and disable it.
 262
 263 <p>It is always possible to disable just after it was enabled, before
 264 any new packages are installed.</p>
 265
 266 <p>To disable it, a backup of the old boot sequence is restored if no
 267 changes has been done to the boot sequence since it was enabled.
 268
 269 <p>If restoring is not possible, All postinst scripts for packages
 270 with init.d scripts will be executed again to get them to call
 271 update-rc.d and add the boot scripts again.
 272
 273 <h2>LSB headers for insserv</h2>
 274
 275 <ul>
 276   <li>"Provides" header, list the Facility/service provided by the
 277     script.  Do not list virtual facilities (like $time)</li>
 278
 279   <li>Runlevel entries (Default-Start and Default-Stop headers), list
 280     what runlevels to activate for this script
 281
 282   <li>Dependency entries (Required-Start, Required-Stop, Should-Start,
 283     Should-Stop, X-Start-Before, X-Stop-After), list the
 284     facilities/services needed by this script.  It will start after
 285     its start dependencies and stop before its stop dependencies.  The
 286     X-* entreis are reverse dependencies.  Required-* are hard
 287     dependencies (will install even if they are missing), while
 288     Should-* and X-* are soft dependencies (only taken into account if
 289     scripts providing them are present).</li>
 290
 291 </ul>
 292
 293 <h2>What to list as dependencies I</h2>
 294
 295 <p>If your package used the default update-rc.d settings before, this
 296 is the header for you:</p>
 297 <pre>
 298 ### BEGIN INIT INFO
 299 # Provides:          scriptname
 300 # Required-Start:    $remote_fs $syslog
 301 # Required-Stop:     $remote_fs $syslog
 302 # Default-Start:     2 3 4 5
 303 # Default-Stop:      0 1 6
 304 ### END INIT INFO
 305 </pre>
 306
 307 <p>$remote_fs is needed by all scripts using files in /usr/.  $syslog
 308 is needed only by scripts starting services logging to syslog.</p>
 309
 310 <h2>Virtual facilities</h2>
 311
 312 <p>Linux Software Base version 3.2 define these virtual facilities:
 313
 314 <dl>
 315
 316 <dt>$local_fs
 317 <dd>all local file systems are mounted. (In Debian, / and /var/ is available)
 318
 319 <dt>$network
 320 <dd>basic networking support is available. Example: a server program
 321 could listen on a socket. (In Debian, network interfaces are up)
 322
 323 <dt>$portmap
 324 <dd>daemons providing SunRPC/ONCRPC portmapping service as defined in
 325 RFC 1833: Binding Protocols for ONC RPC Version 2 (if present) are
 326 running.
 327
 328 <dt>$remote_fs
 329 <dd>all remote file systems are available. In some configurations,
 330 file systems such as /usr may be remote. Many applications that
 331 require $local_fs will probably also require $remote_fs. (In Debian,
 332 /usr/ and NFS directories are guaranteed to be mounted)
 333
 334 <dt>$time
 335 <dd>the system time has been set, for example by using a network-based
 336 time program such as ntp or rdate, or via the hardware Real Time
 337 Clock.
 338
 339 <dt>$syslog
 340 <dd>system logger is operational.
 341
 342 <dt>$named
 343 <dd>IP name-to-address translation, using the interfaces described in
 344 this specification, are available to the level the system normally
 345 provides them. Example: if a DNS query daemon normally provides this
 346 facility, then that daemon has been started.
 347
 348 </dl>
 349
 350 <p>All of these represent points in time during boot and shutdown.
 351
 352 <h2>What to list as dependencies II</h2>
 353
 354 <p>Normally, the start and stop dependencies are the same.
 355
 356 <p>Prefer virtual dependencies over specific dependencies.
 357
 358 <p>When using specific dependencies, use the string listed in the
 359   provides header of the scripts you depend on.
 360
 361 <p>Scripts started in rcS.d/ need extra care.
 362
 363 <p>Depend on $remote_fs unless started in rcS.d/.  Make sure /usr/ is
 364 available during start and that it is stopped before sendsigs kill all
 365 processes during shutdown.</p>
 366
 367 XXX More
 368
 369 <h2>Status of dependency based boot</h2>
 370
 371 <img alt="LSB header progress graph" src="lsb-header-progress.png" width="50%" align="right">
 372
 373 <p>Release goal for Debian Lenny.
 374
 375 <p>Packages with LSB header (in Sid): 654 of 866 (76%)
 376 <br>Unsolved BTS reports: 85
 377 <br>Packages without BTS reports: ~150
 378 <br>Last package projected fixed 2008-07-19 with the current rate
 379
 380 <p>Need better documentation
 381
 382 <p>Should update  Debian policy to reflect this new feature.
 383
 384 <p>Two insserv bugs to fix.
 385 <ul>
 386 <li>Avoid adding unwanted stop symlinks.
 387 <li>Do not fail on "fake" loops. (minor)
 388 </ul></p>
 389
 390 <p>
 391
 392 <h2>Tracking status</h2>
 393 debian/rules missing-overrides
 394 debian/rules missing-by-popcon
 395
 396 <h2>References</h2>
 397
 398 http://wiki.debian.org/LSBInitScripts
 399 http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot
 400 svn+ssh://svn.debian.org/svn/initscripts-ng/trunk/src/insserv
 401
 402     <h2>Thank you very much</h2>
 403
 404     <h3>Questions?</h3>
 405
 406     <p><a href="http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html">http://www.hungry.com/~pere/mypapers/200802-bootsequence/200802-bootsequence.html</a></p>
 407
 408   </body>
 409 </html>