blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Detecting NFS hangs on Linux without hanging yourself...</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="https://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="https://people.skolelinux.org/pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="https://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">Detecting NFS hangs on Linux without hanging yourself...</div>
  24       <div class="date"> 9th March 2017</div>
  25       <div class="body"><p>Over the years, administrating thousand of NFS mounting linux
  26 computers at the time, I often needed a way to detect if the machine
  27 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
  28 file or directory affected by the hang, the process (and possibly the
  29 shell) will hang too.  So you want to be able to detect this without
  30 risking the detection process getting stuck too.  It has not been
  31 obvious how to do this.  When the hang has lasted a while, it is
  32 possible to find messages like these in dmesg:</p>
  33
  34 <p><blockquote>
  35 nfs: server nfsserver not responding, still trying
  36 <br>nfs: server nfsserver OK
  37 </blockquote></p>
  38
  39 <p>It is hard to know if the hang is still going on, and it is hard to
  40 be sure looking in dmesg is going to work.  If there are lots of other
  41 messages in dmesg the lines might have rotated out of site before they
  42 are noticed.</p>
  43
  44 <p>While reading through the nfs client implementation in linux kernel
  45 code, I came across some statistics that seem to give a way to detect
  46 it.  The om_timeouts sunrpc value in the kernel will increase every
  47 time the above log entry is inserted into dmesg.  And after digging a
  48 bit further, I discovered that this value show up in
  49 /proc/self/mountstats on Linux.</p>
  50
  51 <p>The mountstats content seem to be shared between files using the
  52 same file system context, so it is enough to check one of the
  53 mountstats files to get the state of the mount point for the machine.
  54 I assume this will not show lazy umounted NFS points, nor NFS mount
  55 points in a different process context (ie with a different filesystem
  56 view), but that does not worry me.</p>
  57
  58 <p>The content for a NFS mount point look similar to this:</p>
  59
  60 <p><blockquote><pre>
  61 [...]
  62 device /dev/mapper/Debian-var mounted on /var with fstype ext3
  63 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
  64         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
  65         age:    7863311
  66         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
  67         sec:    flavor=1,pseudoflavor=1
  68         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
  69         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
  70         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
  71         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
  72         per-op statistics
  73                 NULL: 0 0 0 0 0 0 0 0
  74              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
  75              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
  76               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
  77               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
  78             READLINK: 125 125 0 20472 18620 0 1112 1118
  79                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
  80                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
  81               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
  82                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
  83              SYMLINK: 903 903 0 233428 245488 6 5865 5917
  84                MKNOD: 80 80 0 20148 21760 0 299 304
  85               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
  86                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
  87               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
  88                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
  89              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
  90          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
  91               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
  92               FSINFO: 2 2 0 232 328 0 1 1
  93             PATHCONF: 1 1 0 116 140 0 0 0
  94               COMMIT: 0 0 0 0 0 0 0 0
  95
  96 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
  97 [...]
  98 </pre></blockquote></p>
  99
 100 <p>The key number to look at is the third number in the per-op list.
 101 It is the number of NFS timeouts experiences per file system
 102 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 103 numbers are increasing, I believe the machine is experiencing NFS
 104 hang.  Unfortunately the timeout value do not start to increase right
 105 away.  The NFS operations need to time out first, and this can take a
 106 while.  The exact timeout value depend on the setup.  For example the
 107 defaults for TCP and UDP mount points are quite different, and the
 108 timeout value is affected by the soft, hard, timeo and retrans NFS
 109 mount options.</p>
 110
 111 <p>The only way I have been able to get working on Debian and RedHat
 112 Enterprise Linux for getting the timeout count is to peek in /proc/.
 113 But according to
 114 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 115 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 116 command can be used to get these timeout values.  But this do not work
 117 on Linux, as far as I can tell.  I
 118 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 119 but have not seen any replies yet.</p>
 120
 121 <p>Is there a better way to figure out if a Linux NFS client is
 122 experiencing NFS hangs?  Is there a way to detect which processes are
 123 affected?  Is there a way to get the NFS mount going quickly once the
 124 network problem causing the NFS hang has been cleared?  I would very
 125 much welcome some clues, as we regularly run into NFS hangs.</p>
 126 </div>
 127
 128       <div class="tags">Tags: <a href="https://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="https://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="https://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.</div>
 129
 130
 131     </div>
 132
 133
 134
 135
 136     <div id="sidebar">
 137
 138
 139
 140 <h2>Archive</h2>
 141 <ul>
 142
 143 <li>2025
 144 <ul>
 145
 146 <li><a href="https://people.skolelinux.org/pere/blog/archive/2025/01/">January (1)</a></li>
 147
 148 </ul></li>
 149
 150 <li>2023
 151 <ul>
 152
 153 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/01/">January (3)</a></li>
 154
 155 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/02/">February (1)</a></li>
 156
 157 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/04/">April (2)</a></li>
 158
 159 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/05/">May (3)</a></li>
 160
 161 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/06/">June (1)</a></li>
 162
 163 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/08/">August (1)</a></li>
 164
 165 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/09/">September (1)</a></li>
 166
 167 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/10/">October (1)</a></li>
 168
 169 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/11/">November (4)</a></li>
 170
 171 <li><a href="https://people.skolelinux.org/pere/blog/archive/2023/12/">December (1)</a></li>
 172
 173 </ul></li>
 174
 175 <li>2022
 176 <ul>
 177
 178 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/02/">February (1)</a></li>
 179
 180 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/03/">March (3)</a></li>
 181
 182 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/04/">April (2)</a></li>
 183
 184 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/06/">June (2)</a></li>
 185
 186 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/07/">July (1)</a></li>
 187
 188 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/09/">September (1)</a></li>
 189
 190 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/10/">October (1)</a></li>
 191
 192 <li><a href="https://people.skolelinux.org/pere/blog/archive/2022/12/">December (1)</a></li>
 193
 194 </ul></li>
 195
 196 <li>2021
 197 <ul>
 198
 199 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/01/">January (2)</a></li>
 200
 201 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/02/">February (1)</a></li>
 202
 203 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/05/">May (1)</a></li>
 204
 205 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/06/">June (1)</a></li>
 206
 207 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/07/">July (3)</a></li>
 208
 209 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/08/">August (1)</a></li>
 210
 211 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/09/">September (1)</a></li>
 212
 213 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/10/">October (1)</a></li>
 214
 215 <li><a href="https://people.skolelinux.org/pere/blog/archive/2021/12/">December (1)</a></li>
 216
 217 </ul></li>
 218
 219 <li>2020
 220 <ul>
 221
 222 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/02/">February (2)</a></li>
 223
 224 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/03/">March (2)</a></li>
 225
 226 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/04/">April (2)</a></li>
 227
 228 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/05/">May (3)</a></li>
 229
 230 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/06/">June (2)</a></li>
 231
 232 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/07/">July (1)</a></li>
 233
 234 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/09/">September (1)</a></li>
 235
 236 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/10/">October (1)</a></li>
 237
 238 <li><a href="https://people.skolelinux.org/pere/blog/archive/2020/11/">November (1)</a></li>
 239
 240 </ul></li>
 241
 242 <li>2019
 243 <ul>
 244
 245 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/01/">January (4)</a></li>
 246
 247 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/02/">February (3)</a></li>
 248
 249 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/03/">March (3)</a></li>
 250
 251 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/05/">May (2)</a></li>
 252
 253 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/06/">June (5)</a></li>
 254
 255 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/07/">July (2)</a></li>
 256
 257 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/08/">August (1)</a></li>
 258
 259 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/09/">September (1)</a></li>
 260
 261 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/11/">November (1)</a></li>
 262
 263 <li><a href="https://people.skolelinux.org/pere/blog/archive/2019/12/">December (4)</a></li>
 264
 265 </ul></li>
 266
 267 <li>2018
 268 <ul>
 269
 270 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/01/">January (1)</a></li>
 271
 272 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/02/">February (5)</a></li>
 273
 274 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/03/">March (5)</a></li>
 275
 276 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/04/">April (3)</a></li>
 277
 278 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/06/">June (2)</a></li>
 279
 280 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/07/">July (5)</a></li>
 281
 282 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/08/">August (3)</a></li>
 283
 284 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/09/">September (3)</a></li>
 285
 286 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/10/">October (5)</a></li>
 287
 288 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/11/">November (2)</a></li>
 289
 290 <li><a href="https://people.skolelinux.org/pere/blog/archive/2018/12/">December (4)</a></li>
 291
 292 </ul></li>
 293
 294 <li>2017
 295 <ul>
 296
 297 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 298
 299 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 300
 301 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/03/">March (5)</a></li>
 302
 303 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
 304
 305 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
 306
 307 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
 308
 309 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
 310
 311 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
 312
 313 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
 314
 315 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/11/">November (3)</a></li>
 316
 317 <li><a href="https://people.skolelinux.org/pere/blog/archive/2017/12/">December (4)</a></li>
 318
 319 </ul></li>
 320
 321 <li>2016
 322 <ul>
 323
 324 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 325
 326 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 327
 328 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 329
 330 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 331
 332 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 333
 334 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 335
 336 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 337
 338 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 339
 340 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 341
 342 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 343
 344 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 345
 346 <li><a href="https://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 347
 348 </ul></li>
 349
 350 <li>2015
 351 <ul>
 352
 353 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 354
 355 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 356
 357 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 358
 359 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 360
 361 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 362
 363 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 364
 365 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 366
 367 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 368
 369 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 370
 371 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 372
 373 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 374
 375 <li><a href="https://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 376
 377 </ul></li>
 378
 379 <li>2014
 380 <ul>
 381
 382 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 383
 384 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 385
 386 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 387
 388 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 389
 390 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 391
 392 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 393
 394 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 395
 396 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 397
 398 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 399
 400 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 401
 402 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 403
 404 <li><a href="https://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 405
 406 </ul></li>
 407
 408 <li>2013
 409 <ul>
 410
 411 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 412
 413 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 414
 415 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 416
 417 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 418
 419 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 420
 421 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 422
 423 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 424
 425 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 426
 427 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 428
 429 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 430
 431 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 432
 433 <li><a href="https://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 434
 435 </ul></li>
 436
 437 <li>2012
 438 <ul>
 439
 440 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 441
 442 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 443
 444 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 445
 446 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 447
 448 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 449
 450 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 451
 452 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 453
 454 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 455
 456 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 457
 458 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 459
 460 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 461
 462 <li><a href="https://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 463
 464 </ul></li>
 465
 466 <li>2011
 467 <ul>
 468
 469 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 470
 471 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 472
 473 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 474
 475 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 476
 477 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 478
 479 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 480
 481 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 482
 483 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 484
 485 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 486
 487 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 488
 489 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 490
 491 <li><a href="https://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 492
 493 </ul></li>
 494
 495 <li>2010
 496 <ul>
 497
 498 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 499
 500 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 501
 502 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 503
 504 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 505
 506 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 507
 508 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 509
 510 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 511
 512 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 513
 514 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 515
 516 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 517
 518 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 519
 520 <li><a href="https://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 521
 522 </ul></li>
 523
 524 <li>2009
 525 <ul>
 526
 527 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 528
 529 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 530
 531 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 532
 533 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 534
 535 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 536
 537 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 538
 539 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 540
 541 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 542
 543 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 544
 545 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 546
 547 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 548
 549 <li><a href="https://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 550
 551 </ul></li>
 552
 553 <li>2008
 554 <ul>
 555
 556 <li><a href="https://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 557
 558 <li><a href="https://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 559
 560 </ul></li>
 561
 562 </ul>
 563
 564
 565
 566 <h2>Tags</h2>
 567 <ul>
 568
 569  <li><a href="https://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (19)</a></li>
 570
 571  <li><a href="https://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 572
 573  <li><a href="https://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 574
 575  <li><a href="https://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 576
 577  <li><a href="https://people.skolelinux.org/pere/blog/tags/betalkontant">betalkontant (9)</a></li>
 578
 579  <li><a href="https://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (12)</a></li>
 580
 581  <li><a href="https://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (17)</a></li>
 582
 583  <li><a href="https://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 584
 585  <li><a href="https://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (3)</a></li>
 586
 587  <li><a href="https://people.skolelinux.org/pere/blog/tags/debian">debian (197)</a></li>
 588
 589  <li><a href="https://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (159)</a></li>
 590
 591  <li><a href="https://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (9)</a></li>
 592
 593  <li><a href="https://people.skolelinux.org/pere/blog/tags/digistan">digistan (11)</a></li>
 594
 595  <li><a href="https://people.skolelinux.org/pere/blog/tags/dld">dld (18)</a></li>
 596
 597  <li><a href="https://people.skolelinux.org/pere/blog/tags/docbook">docbook (32)</a></li>
 598
 599  <li><a href="https://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 600
 601  <li><a href="https://people.skolelinux.org/pere/blog/tags/english">english (456)</a></li>
 602
 603  <li><a href="https://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 604
 605  <li><a href="https://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (14)</a></li>
 606
 607  <li><a href="https://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (34)</a></li>
 608
 609  <li><a href="https://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 610
 611  <li><a href="https://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (20)</a></li>
 612
 613  <li><a href="https://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 614
 615  <li><a href="https://people.skolelinux.org/pere/blog/tags/intervju">intervju (43)</a></li>
 616
 617  <li><a href="https://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (16)</a></li>
 618
 619  <li><a href="https://people.skolelinux.org/pere/blog/tags/kart">kart (23)</a></li>
 620
 621  <li><a href="https://people.skolelinux.org/pere/blog/tags/kodi">kodi (6)</a></li>
 622
 623  <li><a href="https://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 624
 625  <li><a href="https://people.skolelinux.org/pere/blog/tags/lego">lego (5)</a></li>
 626
 627  <li><a href="https://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 628
 629  <li><a href="https://people.skolelinux.org/pere/blog/tags/linuxcnc">linuxcnc (5)</a></li>
 630
 631  <li><a href="https://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 632
 633  <li><a href="https://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 634
 635  <li><a href="https://people.skolelinux.org/pere/blog/tags/madewithcc">madewithcc (3)</a></li>
 636
 637  <li><a href="https://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 638
 639  <li><a href="https://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (46)</a></li>
 640
 641  <li><a href="https://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (15)</a></li>
 642
 643  <li><a href="https://people.skolelinux.org/pere/blog/tags/noark5">noark5 (24)</a></li>
 644
 645  <li><a href="https://people.skolelinux.org/pere/blog/tags/norsk">norsk (323)</a></li>
 646
 647  <li><a href="https://people.skolelinux.org/pere/blog/tags/nuug">nuug (198)</a></li>
 648
 649  <li><a href="https://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (40)</a></li>
 650
 651  <li><a href="https://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 652
 653  <li><a href="https://people.skolelinux.org/pere/blog/tags/opensnitch">opensnitch (4)</a></li>
 654
 655  <li><a href="https://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (76)</a></li>
 656
 657  <li><a href="https://people.skolelinux.org/pere/blog/tags/personvern">personvern (114)</a></li>
 658
 659  <li><a href="https://people.skolelinux.org/pere/blog/tags/raid">raid (2)</a></li>
 660
 661  <li><a href="https://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 662
 663  <li><a href="https://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 664
 665  <li><a href="https://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 666
 667  <li><a href="https://people.skolelinux.org/pere/blog/tags/robot">robot (17)</a></li>
 668
 669  <li><a href="https://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 670
 671  <li><a href="https://people.skolelinux.org/pere/blog/tags/ruter">ruter (7)</a></li>
 672
 673  <li><a href="https://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 674
 675  <li><a href="https://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (60)</a></li>
 676
 677  <li><a href="https://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 678
 679  <li><a href="https://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 680
 681  <li><a href="https://people.skolelinux.org/pere/blog/tags/standard">standard (75)</a></li>
 682
 683  <li><a href="https://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (7)</a></li>
 684
 685  <li><a href="https://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (14)</a></li>
 686
 687  <li><a href="https://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (65)</a></li>
 688
 689  <li><a href="https://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (5)</a></li>
 690
 691  <li><a href="https://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 692
 693  <li><a href="https://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
 694
 695  <li><a href="https://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri (22)</a></li>
 696
 697  <li><a href="https://people.skolelinux.org/pere/blog/tags/video">video (80)</a></li>
 698
 699  <li><a href="https://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 700
 701  <li><a href="https://people.skolelinux.org/pere/blog/tags/web">web (42)</a></li>
 702
 703 </ul>
 704
 705
 706     </div>
 707     <p style="text-align: right">
 708  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 709 </p>
 710
 711   </body>
 712 </html>