blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Speeding up the Debian installer using eatmydata and dpkg-divert</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">Speeding up the Debian installer using eatmydata and dpkg-divert</div>
  24       <div class="date">16th September 2014</div>
  25       <div class="body"><p>The <a href="https://www.debian.org/">Debian</a> installer could be
  26 a lot quicker.  When we install more than 2000 packages in
  27 <a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using
  28 tasksel in the installer, unpacking the binary packages take forever.
  29 A part of the slow I/O issue was discussed in
  30 <a href="https://bugs.debian.org/613428">bug #613428</a> about too
  31 much file system sync-ing done by dpkg, which is the package
  32 responsible for unpacking the binary packages.  Other parts (like code
  33 executed by postinst scripts) might also sync to disk during
  34 installation.  All this sync-ing to disk do not really make sense to
  35 me.  If the machine crash half-way through, I start over, I do not try
  36 to salvage the half installed system.  So the failure sync-ing is
  37 supposed to protect against, hardware or system crash, is not really
  38 relevant while the installer is running.</p>
  39
  40 <p>A few days ago, I thought of a way to get rid of all the file
  41 system sync()-ing in a fairly non-intrusive way, without the need to
  42 change the code in several packages.  The idea is not new, but I have
  43 not heard anyone propose the approach using dpkg-divert before.  It
  44 depend on the small and clever package
  45 <a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which
  46 uses LD_PRELOAD to replace the system functions for syncing data to
  47 disk with functions doing nothing, thus allowing programs to live
  48 dangerous while speeding up disk I/O significantly.  Instead of
  49 modifying the implementation of dpkg, apt and tasksel (which are the
  50 packages responsible for selecting, fetching and installing packages),
  51 it occurred to me that we could just divert the programs away, replace
  52 them with a simple shell wrapper calling
  53 "eatmydata&nbsp;$program&nbsp;$@", to get the same effect.
  54 Two days ago I decided to test the idea, and wrapped up a simple
  55 implementation for the Debian Edu udeb.</p>
  56
  57 <p>The effect was stunning.  In my first test it reduced the running
  58 time of the pkgsel step (installing tasks) from 64 to less than 44
  59 minutes (20 minutes shaved off the installation) on an old Dell
  60 Latitude D505 machine.  I am not quite sure what the optimised time
  61 would have been, as I messed up the testing a bit, causing the debconf
  62 priority to get low enough for two questions to pop up during
  63 installation.  As soon as I saw the questions I moved the installation
  64 along, but do not know how long the question were holding up the
  65 installation.  I did some more measurements using Debian Edu Jessie,
  66 and got these results.  The time measured is the time stamp in
  67 /var/log/syslog between the "pkgsel: starting tasksel" and the
  68 "pkgsel: finishing up" lines, if you want to do the same measurement
  69 yourself.  In Debian Edu, the tasksel dialog do not show up, and the
  70 timing thus do not depend on how quickly the user handle the tasksel
  71 dialog.</p>
  72
  73 <p><table>
  74
  75 <tr>
  76 <th>Machine/setup</th>
  77 <th>Original tasksel</th>
  78 <th>Optimised tasksel</th>
  79 <th>Reduction</th>
  80 </tr>
  81
  82 <tr>
  83 <td>Latitude D505 Main+LTSP LXDE</td>
  84 <td>64 min (07:46-08:50)</td>
  85 <td><44 min (11:27-12:11)</td>
  86 <td>>20 min 18%</td>
  87 </tr>
  88
  89 <tr>
  90 <td>Latitude D505 Roaming LXDE</td>
  91 <td>57 min (08:48-09:45)</td>
  92 <td>34 min (07:43-08:17)</td>
  93 <td>23 min 40%</td>
  94 </tr>
  95
  96 <tr>
  97 <td>Latitude D505 Minimal</td>
  98 <td>22 min (10:37-10:59)</td>
  99 <td>11 min (11:16-11:27)</td>
 100 <td>11 min 50%</td>
 101 </tr>
 102
 103 <tr>
 104 <td>Thinkpad X200 Minimal</td>
 105 <td>6 min (08:19-08:25)</td>
 106 <td>4 min (08:04-08:08)</td>
 107 <td>2 min 33%</td>
 108 </tr>
 109
 110 <tr>
 111 <td>Thinkpad X200 Roaming KDE</td>
 112 <td>19 min (09:21-09:40)</td>
 113 <td>15 min (10:25-10:40)</td>
 114 <td>4 min 21%</td>
 115 </tr>
 116
 117 </table></p>
 118
 119 <p>The test is done using a netinst ISO on a USB stick, so some of the
 120 time is spent downloading packages.  The connection to the Internet
 121 was 100Mbit/s during testing, so downloading should not be a
 122 significant factor in the measurement.  Download typically took a few
 123 seconds to a few minutes, depending on the amount of packages being
 124 installed.</p>
 125
 126 <p>The speedup is implemented by using two hooks in
 127 <a href="https://www.debian.org/devel/debian-installer/">Debian
 128 Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the
 129 finish-install.d hook to remove the divert at the end of the
 130 installation.  I picked the pre-pkgsel.d hook instead of the
 131 post-base-installer.d hook because I test using an ISO without the
 132 eatmydata package included, and the post-base-installer.d hook in
 133 Debian Edu can only operate on packages included in the ISO.  The
 134 negative effect of this is that I am unable to activate this
 135 optimization for the kernel installation step in d-i.  If the code is
 136 moved to the post-base-installer.d hook, the speedup would be larger
 137 for the entire installation.</p>
 138
 139 <p>I've implemented this in the
 140 <a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a>
 141 git repository, and plan to provide the optimization as part of the
 142 Debian Edu installation.  If you want to test this yourself, you can
 143 create two files in the installer (or in an udeb).  One shell script
 144 need do go into /usr/lib/pre-pkgsel.d/, with content like this:</p>
 145
 146 <p><blockquote><pre>
 147 #!/bin/sh
 148 set -e
 149 . /usr/share/debconf/confmodule
 150 info() {
 151     logger -t my-pkgsel "info: $*"
 152 }
 153 error() {
 154     logger -t my-pkgsel "error: $*"
 155 }
 156 override_install() {
 157     apt-install eatmydata || true
 158     if [ -x /target/usr/bin/eatmydata ] ; then
 159         for bin in dpkg apt-get aptitude tasksel ; do
 160             file=/usr/bin/$bin
 161             # Test that the file exist and have not been diverted already.
 162             if [ -f /target$file ] ; then
 163                 info "diverting $file using eatmydata"
 164                 printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
 165                     > /target$file.edu
 166                 chmod 755 /target$file.edu
 167                 in-target dpkg-divert --package debian-edu-config \
 168                     --rename --quiet --add $file
 169                 ln -sf ./$bin.edu /target$file
 170             else
 171                 error "unable to divert $file, as it is missing."
 172             fi
 173         done
 174     else
 175         error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
 176     fi
 177 }
 178
 179 override_install
 180 </pre></blockquote></p>
 181
 182 <p>To clean up, another shell script should go into
 183 /usr/lib/finish-install.d/ with code like this:
 184
 185 <p><blockquote><pre>
 186 #! /bin/sh -e
 187 . /usr/share/debconf/confmodule
 188 error() {
 189     logger -t my-finish-install "error: $@"
 190 }
 191 remove_install_override() {
 192     for bin in dpkg apt-get aptitude tasksel ; do
 193         file=/usr/bin/$bin
 194         if [ -x /target$file.edu ] ; then
 195             rm /target$file
 196             in-target dpkg-divert --package debian-edu-config \
 197                 --rename --quiet --remove $file
 198             rm /target$file.edu
 199         else
 200             error "Missing divert for $file."
 201         fi
 202     done
 203     sync # Flush file buffers before continuing
 204 }
 205
 206 remove_install_override
 207 </pre></blockquote></p>
 208
 209 <p>In Debian Edu, I placed both code fragments in a separate script
 210 edu-eatmydata-install and call it from the pre-pkgsel.d and
 211 finish-install.d scripts.</p>
 212
 213 <p>By now you might ask if this change should get into the normal
 214 Debian installer too?  I suspect it should, but am not sure the
 215 current debian-installer coordinators find it useful enough.  It also
 216 depend on the side effects of the change.  I'm not aware of any, but I
 217 guess we will see if the change is safe after some more testing.
 218 Perhaps there is some package in Debian depending on sync() and
 219 fsync() having effect?  Perhaps it should go into its own udeb, to
 220 allow those of us wanting to enable it to do so without affecting
 221 everyone.</p>
 222
 223 <p>Update 2014-09-24: Since a few days ago, enabling this optimization
 224 will break installation of all programs using gnutls because of
 225 <a href="https://bugs.debian.org/702711">bug #702711</a>.  An updated
 226 eatmydata package in Debian will solve it.</p>
 227
 228 <p>Update 2014-10-17: The bug mentioned above is fixed in testing and
 229 the optimization work again.  And I have discovered that the
 230 dpkg-divert trick is not really needed and implemented a slightly
 231 simpler approach as part of the debian-edu-install package.  See
 232 tools/edu-eatmydata-install in the source package.</p>
 233
 234 <p>Update 2014-11-11: Unfortunately, a new
 235 <a href="http://bugs.debian.org/765738">bug #765738</a> in eatmydata only
 236 triggering on i386 made it into testing, and broke this installation
 237 optimization again.  If <a href="http://bugs.debian.org/768893">unblock
 238 request 768893</a> is accepted, it should be working again.</p>
 239 </div>
 240
 241       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.</div>
 242
 243
 244     </div>
 245
 246
 247
 248
 249     <div id="sidebar">
 250
 251
 252
 253 <h2>Archive</h2>
 254 <ul>
 255
 256 <li>2015
 257 <ul>
 258
 259 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 260
 261 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 262
 263 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 264
 265 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 266
 267 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 268
 269 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 270
 271 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 272
 273 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 274
 275 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 276
 277 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (4)</a></li>
 278
 279 </ul></li>
 280
 281 <li>2014
 282 <ul>
 283
 284 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 285
 286 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 287
 288 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 289
 290 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 291
 292 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 293
 294 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 295
 296 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 297
 298 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 299
 300 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 301
 302 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 303
 304 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 305
 306 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 307
 308 </ul></li>
 309
 310 <li>2013
 311 <ul>
 312
 313 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 314
 315 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 316
 317 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 318
 319 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 320
 321 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 322
 323 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 324
 325 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 326
 327 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 328
 329 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 330
 331 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 332
 333 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 334
 335 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 336
 337 </ul></li>
 338
 339 <li>2012
 340 <ul>
 341
 342 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 343
 344 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 345
 346 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 347
 348 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 349
 350 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 351
 352 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 353
 354 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 355
 356 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 357
 358 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 359
 360 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 361
 362 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 363
 364 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 365
 366 </ul></li>
 367
 368 <li>2011
 369 <ul>
 370
 371 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 372
 373 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 374
 375 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 376
 377 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 378
 379 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 380
 381 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 382
 383 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 384
 385 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 386
 387 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 388
 389 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 390
 391 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 392
 393 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 394
 395 </ul></li>
 396
 397 <li>2010
 398 <ul>
 399
 400 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 401
 402 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 403
 404 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 405
 406 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 407
 408 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 409
 410 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 411
 412 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 413
 414 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 415
 416 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 417
 418 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 419
 420 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 421
 422 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 423
 424 </ul></li>
 425
 426 <li>2009
 427 <ul>
 428
 429 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 430
 431 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 432
 433 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 434
 435 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 436
 437 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 438
 439 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 440
 441 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 442
 443 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 444
 445 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 446
 447 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 448
 449 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 450
 451 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 452
 453 </ul></li>
 454
 455 <li>2008
 456 <ul>
 457
 458 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 459
 460 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 461
 462 </ul></li>
 463
 464 </ul>
 465
 466
 467
 468 <h2>Tags</h2>
 469 <ul>
 470
 471  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 472
 473  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 474
 475  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 476
 477  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 478
 479  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (8)</a></li>
 480
 481  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (15)</a></li>
 482
 483  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 484
 485  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 486
 487  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (112)</a></li>
 488
 489  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (153)</a></li>
 490
 491  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 492
 493  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (15)</a></li>
 494
 495  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (18)</a></li>
 496
 497  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 498
 499  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (291)</a></li>
 500
 501  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 502
 503  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 504
 505  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (20)</a></li>
 506
 507  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 508
 509  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (16)</a></li>
 510
 511  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 512
 513  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 514
 515  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (10)</a></li>
 516
 517  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (19)</a></li>
 518
 519  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 520
 521  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 522
 523  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 524
 525  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 526
 527  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 528
 529  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (36)</a></li>
 530
 531  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (265)</a></li>
 532
 533  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (177)</a></li>
 534
 535  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (20)</a></li>
 536
 537  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 538
 539  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (54)</a></li>
 540
 541  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (88)</a></li>
 542
 543  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 544
 545  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 546
 547  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 548
 549  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 550
 551  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (9)</a></li>
 552
 553  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 554
 555  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 556
 557  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 558
 559  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (43)</a></li>
 560
 561  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 562
 563  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 564
 565  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (48)</a></li>
 566
 567  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 568
 569  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (10)</a></li>
 570
 571  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (35)</a></li>
 572
 573  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (2)</a></li>
 574
 575  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 576
 577  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
 578
 579  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (54)</a></li>
 580
 581  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 582
 583  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (37)</a></li>
 584
 585 </ul>
 586
 587
 588     </div>
 589     <p style="text-align: right">
 590  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 591 </p>
 592
 593   </body>
 594 </html>