blog/archive/2017/03/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: entries from March 2017</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9     <link rel="alternate" title="RSS Feed" href="03.rss" type="application/rss+xml" />
  10   </head>
  11   <body>
  12     <!-- XML FEED -->
  13     <div class="title">
  14  <h1>
  15      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <h3>Entries from March 2017.</h3>
  23
  24     <div class="entry">
  25       <div class="title">
  26         <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a>
  27       </div>
  28       <div class="date">
  29          9th March 2017
  30       </div>
  31       <div class="body">
  32         <p>Over the years, administrating thousand of NFS mounting linux
  33 computers at the time, I often needed a way to detect if the machine
  34 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
  35 file or directory affected by the hang, the process (and possibly the
  36 shell) will hang too.  So you want to be able to detect this without
  37 risking the detection process getting stuck too.  It has not been
  38 obvious how to do this.  When the hang has lasted a while, it is
  39 possible to find messages like these in dmesg:</p>
  40
  41 <p><blockquote>
  42 nfs: server nfsserver not responding, still trying
  43 <br>nfs: server nfsserver OK
  44 </blockquote></p>
  45
  46 <p>It is hard to know if the hang is still going on, and it is hard to
  47 be sure looking in dmesg is going to work.  If there are lots of other
  48 messages in dmesg the lines might have rotated out of site before they
  49 are noticed.</p>
  50
  51 <p>While reading through the nfs client implementation in linux kernel
  52 code, I came across some statistics that seem to give a way to detect
  53 it.  The om_timeouts sunrpc value in the kernel will increase every
  54 time the above log entry is inserted into dmesg.  And after digging a
  55 bit further, I discovered that this value show up in
  56 /proc/self/mountstats on Linux.</p>
  57
  58 <p>The mountstats content seem to be shared between files using the
  59 same file system context, so it is enough to check one of the
  60 mountstats files to get the state of the mount point for the machine.
  61 I assume this will not show lazy umounted NFS points, nor NFS mount
  62 points in a different process context (ie with a different filesystem
  63 view), but that does not worry me.</p>
  64
  65 <p>The content for a NFS mount point look similar to this:</p>
  66
  67 <p><blockquote><pre>
  68 [...]
  69 device /dev/mapper/Debian-var mounted on /var with fstype ext3
  70 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
  71         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
  72         age:    7863311
  73         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
  74         sec:    flavor=1,pseudoflavor=1
  75         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
  76         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
  77         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
  78         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
  79         per-op statistics
  80                 NULL: 0 0 0 0 0 0 0 0
  81              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
  82              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
  83               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
  84               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
  85             READLINK: 125 125 0 20472 18620 0 1112 1118
  86                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
  87                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
  88               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
  89                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
  90              SYMLINK: 903 903 0 233428 245488 6 5865 5917
  91                MKNOD: 80 80 0 20148 21760 0 299 304
  92               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
  93                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
  94               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
  95                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
  96              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
  97          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
  98               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
  99               FSINFO: 2 2 0 232 328 0 1 1
 100             PATHCONF: 1 1 0 116 140 0 0 0
 101               COMMIT: 0 0 0 0 0 0 0 0
 102
 103 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
 104 [...]
 105 </pre></blockquote></p>
 106
 107 <p>The key number to look at is the third number in the per-op list.
 108 It is the number of NFS timeouts experiences per file system
 109 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 110 numbers are increasing, I believe the machine is experiencing NFS
 111 hang.  Unfortunately the timeout value do not start to increase right
 112 away.  The NFS operations need to time out first, and this can take a
 113 while.  The exact timeout value depend on the setup.  For example the
 114 defaults for TCP and UDP mount points are quite different, and the
 115 timeout value is affected by the soft, hard, timeo and retrans NFS
 116 mount options.</p>
 117
 118 <p>The only way I have been able to get working on Debian and RedHat
 119 Enterprise Linux for getting the timeout count is to peek in /proc/.
 120 But according to
 121 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 122 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 123 command can be used to get these timeout values.  But this do not work
 124 on Linux, as far as I can tell.  I
 125 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 126 but have not seen any replies yet.</p>
 127
 128 <p>Is there a better way to figure out if a Linux NFS client is
 129 experiencing NFS hangs?  Is there a way to detect which processes are
 130 affected?  Is there a way to get the NFS mount going quickly once the
 131 network problem causing the NFS hang has been cleared?  I would very
 132 much welcome some clues, as we regularly run into NFS hangs.</p>
 133
 134       </div>
 135       <div class="tags">
 136
 137
 138         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 139
 140
 141       </div>
 142     </div>
 143     <div class="padding"></div>
 144
 145     <div class="entry">
 146       <div class="title">
 147         <a href="http://people.skolelinux.org/pere/blog/How_does_it_feel_to_be_wiretapped__when_you_should_be_doing_the_wiretapping___.html">How does it feel to be wiretapped, when you should be doing the wiretapping...</a>
 148       </div>
 149       <div class="date">
 150          8th March 2017
 151       </div>
 152       <div class="body">
 153         <p>So the new president in the United States of America claim to be
 154 surprised to discover that he was wiretapped during the election
 155 before he was elected president.  He even claim this must be illegal.
 156 Well, doh, if it is one thing the confirmations from Snowden
 157 documented, it is that the entire population in USA is wiretapped, one
 158 way or another.  Of course the president candidates were wiretapped,
 159 alongside the senators, judges and the rest of the people in USA.</p>
 160
 161 <p>Next, the Federal Bureau of Investigation ask the Department of
 162 Justice to go public rejecting the claims that Donald Trump was
 163 wiretapped illegally.  I fail to see the relevance, given that I am
 164 sure the surveillance industry in USA believe they have all the legal
 165 backing they need to conduct mass surveillance on the entire
 166 world.</p>
 167
 168 <p>There is even the director of the FBI stating that he never saw an
 169 order requesting wiretapping of Donald Trump.  That is not very
 170 surprising, given how the FISA court work, with all its activity being
 171 secret.  Perhaps he only heard about it?</p>
 172
 173 <p>What I find most sad in this story is how Norwegian journalists
 174 present it.  In a news reports the other day in the radio from the
 175 Norwegian National broadcasting Company (NRK), I heard the journalist
 176 claim that 'the FBI denies any wiretapping', while the reality is that
 177 'the FBI denies any illegal wiretapping'.  There is a fundamental and
 178 important difference, and it make me sad that the journalists are
 179 unable to grasp it.</p>
 180
 181       </div>
 182       <div class="tags">
 183
 184
 185         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
 186
 187
 188       </div>
 189     </div>
 190     <div class="padding"></div>
 191
 192     <div class="entry">
 193       <div class="title">
 194         <a href="http://people.skolelinux.org/pere/blog/Norwegian_Bokm_l_translation_of_The_Debian_Administrator_s_Handbook_complete__proofreading_in_progress.html">Norwegian Bokmål translation of The Debian Administrator's Handbook complete, proofreading in progress</a>
 195       </div>
 196       <div class="date">
 197          3rd March 2017
 198       </div>
 199       <div class="body">
 200         <p>For almost a year now, we have been working on making a Norwegian
 201 Bokmål edition of <a href="https://debian-handbook.info/">The Debian
 202 Administrator's Handbook</a>.  Now, thanks to the tireless effort of
 203 Ole-Erik, Ingrid and Andreas, the initial translation is complete, and
 204 we are working on the proof reading to ensure consistent language and
 205 use of correct computer science terms.  The plan is to make the book
 206 available on paper, as well as in electronic form.  For that to
 207 happen, the proof reading must be completed and all the figures need
 208 to be translated.  If you want to help out, get in touch.</p>
 209
 210 <p><a href="http://people.skolelinux.org/pere/debian-handbook/debian-handbook-nb-NO.pdf">A
 211
 212 fresh PDF edition</a> in A4 format (the final book will have smaller
 213 pages) of the book created every morning is available for
 214 proofreading.  If you find any errors, please
 215 <a href="https://hosted.weblate.org/projects/debian-handbook/">visit
 216 Weblate and correct the error</a>.  The
 217 <a href="http://l.github.io/debian-handbook/stat/nb-NO/index.html">state
 218 of the translation including figures</a> is a useful source for those
 219 provide Norwegian bokmål screen shots and figures.</p>
 220
 221       </div>
 222       <div class="tags">
 223
 224
 225         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
 226
 227
 228       </div>
 229     </div>
 230     <div class="padding"></div>
 231
 232     <div class="entry">
 233       <div class="title">
 234         <a href="http://people.skolelinux.org/pere/blog/Unlimited_randomness_with_the_ChaosKey_.html">Unlimited randomness with the ChaosKey?</a>
 235       </div>
 236       <div class="date">
 237          1st March 2017
 238       </div>
 239       <div class="body">
 240         <p>A few days ago I ordered a small batch of
 241 <a href="http://altusmetrum.org/ChaosKey/">the ChaosKey</a>, a small
 242 USB dongle for generating entropy created by Bdale Garbee and Keith
 243 Packard.  Yesterday it arrived, and I am very happy to report that it
 244 work great!  According to its designers, to get it to work out of the
 245 box, you need the Linux kernel version 4.1 or later.  I tested on a
 246 Debian Stretch machine (kernel version 4.9), and there it worked just
 247 fine, increasing the available entropy very quickly.  I wrote a small
 248 test oneliner to test.  It first print the current entropy level,
 249 drain /dev/random, and then print the entropy level for five seconds.
 250 Here is the situation without the ChaosKey inserted:</p>
 251
 252 <blockquote><pre>
 253 % cat /proc/sys/kernel/random/entropy_avail; \
 254   dd bs=1M if=/dev/random of=/dev/null count=1; \
 255   for n in $(seq 1 5); do \
 256      cat /proc/sys/kernel/random/entropy_avail; \
 257      sleep 1; \
 258   done
 259 300
 260 0+1 oppføringer inn
 261 0+1 oppføringer ut
 262 28 byte kopiert, 0,000264565 s, 106 kB/s
 263 4
 264 8
 265 12
 266 17
 267 21
 268 %
 269 </pre></blockquote>
 270
 271 <p>The entropy level increases by 3-4 every second.  In such case any
 272 application requiring random bits (like a HTTPS enabled web server)
 273 will halt and wait for more entrpy.  And here is the situation with
 274 the ChaosKey inserted:</p>
 275
 276 <blockquote><pre>
 277 % cat /proc/sys/kernel/random/entropy_avail; \
 278   dd bs=1M if=/dev/random of=/dev/null count=1; \
 279   for n in $(seq 1 5); do \
 280      cat /proc/sys/kernel/random/entropy_avail; \
 281      sleep 1; \
 282   done
 283 1079
 284 0+1 oppføringer inn
 285 0+1 oppføringer ut
 286 104 byte kopiert, 0,000487647 s, 213 kB/s
 287 433
 288 1028
 289 1031
 290 1035
 291 1038
 292 %
 293 </pre></blockquote>
 294
 295 <p>Quite the difference. :) I bought a few more than I need, in case
 296 someone want to buy one here in Norway. :)</p>
 297
 298 <p>Update: The dongle was presented at Debconf last year.  You might
 299 find <a href="https://debconf16.debconf.org/talks/94/">the talk
 300 recording illuminating</a>.  It explains exactly what the source of
 301 randomness is, if you are unable to spot it from the schema drawing
 302 available from the ChaosKey web site linked at the start of this blog
 303 post.</p>
 304
 305       </div>
 306       <div class="tags">
 307
 308
 309         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
 310
 311
 312       </div>
 313     </div>
 314     <div class="padding"></div>
 315
 316     <p style="text-align: right;"><a href="03.rss"><img src="http://people.skolelinux.org/pere/blog/xml.gif" alt="RSS Feed" width="36" height="14" /></a></p>
 317     <div id="sidebar">
 318
 319
 320
 321 <h2>Archive</h2>
 322 <ul>
 323
 324 <li>2017
 325 <ul>
 326
 327 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 328
 329 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 330
 331 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (4)</a></li>
 332
 333 </ul></li>
 334
 335 <li>2016
 336 <ul>
 337
 338 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 339
 340 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 341
 342 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 343
 344 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 345
 346 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 347
 348 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 349
 350 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 351
 352 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 353
 354 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 355
 356 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 357
 358 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 359
 360 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 361
 362 </ul></li>
 363
 364 <li>2015
 365 <ul>
 366
 367 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 368
 369 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 370
 371 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 372
 373 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 374
 375 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 376
 377 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 378
 379 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 380
 381 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 382
 383 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 384
 385 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 386
 387 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 388
 389 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 390
 391 </ul></li>
 392
 393 <li>2014
 394 <ul>
 395
 396 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 397
 398 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 399
 400 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 401
 402 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 403
 404 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 405
 406 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 407
 408 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 409
 410 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 411
 412 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 413
 414 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 415
 416 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 417
 418 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 419
 420 </ul></li>
 421
 422 <li>2013
 423 <ul>
 424
 425 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 426
 427 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 428
 429 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 430
 431 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 432
 433 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 434
 435 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 436
 437 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 438
 439 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 440
 441 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 442
 443 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 444
 445 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 446
 447 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 448
 449 </ul></li>
 450
 451 <li>2012
 452 <ul>
 453
 454 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 455
 456 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 457
 458 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 459
 460 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 461
 462 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 463
 464 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 465
 466 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 467
 468 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 469
 470 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 471
 472 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 473
 474 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 475
 476 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 477
 478 </ul></li>
 479
 480 <li>2011
 481 <ul>
 482
 483 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 484
 485 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 486
 487 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 488
 489 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 490
 491 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 492
 493 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 494
 495 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 496
 497 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 498
 499 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 500
 501 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 502
 503 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 504
 505 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 506
 507 </ul></li>
 508
 509 <li>2010
 510 <ul>
 511
 512 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 513
 514 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 515
 516 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 517
 518 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 519
 520 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 521
 522 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 523
 524 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 525
 526 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 527
 528 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 529
 530 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 531
 532 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 533
 534 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 535
 536 </ul></li>
 537
 538 <li>2009
 539 <ul>
 540
 541 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 542
 543 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 544
 545 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 546
 547 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 548
 549 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 550
 551 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 552
 553 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 554
 555 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 556
 557 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 558
 559 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 560
 561 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 562
 563 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 564
 565 </ul></li>
 566
 567 <li>2008
 568 <ul>
 569
 570 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 571
 572 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 573
 574 </ul></li>
 575
 576 </ul>
 577
 578
 579
 580 <h2>Tags</h2>
 581 <ul>
 582
 583  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 584
 585  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 586
 587  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 588
 589  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 590
 591  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
 592
 593  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
 594
 595  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 596
 597  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 598
 599  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (148)</a></li>
 600
 601  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
 602
 603  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (3)</a></li>
 604
 605  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 606
 607  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (16)</a></li>
 608
 609  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (23)</a></li>
 610
 611  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 612
 613  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (345)</a></li>
 614
 615  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 616
 617  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 618
 619  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (29)</a></li>
 620
 621  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 622
 623  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
 624
 625  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 626
 627  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 628
 629  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
 630
 631  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (20)</a></li>
 632
 633  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 634
 635  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 636
 637  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 638
 639  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 640
 641  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 642
 643  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
 644
 645  <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
 646
 647  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (287)</a></li>
 648
 649  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (187)</a></li>
 650
 651  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (28)</a></li>
 652
 653  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 654
 655  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (64)</a></li>
 656
 657  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (99)</a></li>
 658
 659  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 660
 661  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 662
 663  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 664
 665  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 666
 667  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
 668
 669  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 670
 671  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
 672
 673  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 674
 675  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (52)</a></li>
 676
 677  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 678
 679  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 680
 681  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (51)</a></li>
 682
 683  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (5)</a></li>
 684
 685  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (11)</a></li>
 686
 687  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (48)</a></li>
 688
 689  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (3)</a></li>
 690
 691  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 692
 693  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
 694
 695  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
 696
 697  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 698
 699  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (40)</a></li>
 700
 701 </ul>
 702
 703
 704     </div>
 705     <p style="text-align: right">
 706  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 707 </p>
 708
 709   </body>
 710 </html>