blog/tags/sysadmin/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Entries Tagged sysadmin</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9     <link rel="alternate" title="RSS Feed" href="sysadmin.rss" type="application/rss+xml" />
  10   </head>
  11   <body>
  12     <div class="title">
  13  <h1>
  14      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  15
  16  </h1>
  17
  18 </div>
  19
  20
  21     <h3>Entries tagged "sysadmin".</h3>
  22
  23     <div class="entry">
  24       <div class="title">
  25         <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a>
  26       </div>
  27       <div class="date">
  28          9th March 2017
  29       </div>
  30       <div class="body">
  31         <p>Over the years, administrating thousand of NFS mounting linux
  32 computers at the time, I often needed a way to detect if the machine
  33 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
  34 file or directory affected by the hang, the process (and possibly the
  35 shell) will hang too.  So you want to be able to detect this without
  36 risking the detection process getting stuck too.  It has not been
  37 obvious how to do this.  When the hang has lasted a while, it is
  38 possible to find messages like these in dmesg:</p>
  39
  40 <p><blockquote>
  41 nfs: server nfsserver not responding, still trying
  42 <br>nfs: server nfsserver OK
  43 </blockquote></p>
  44
  45 <p>It is hard to know if the hang is still going on, and it is hard to
  46 be sure looking in dmesg is going to work.  If there are lots of other
  47 messages in dmesg the lines might have rotated out of site before they
  48 are noticed.</p>
  49
  50 <p>While reading through the nfs client implementation in linux kernel
  51 code, I came across some statistics that seem to give a way to detect
  52 it.  The om_timeouts sunrpc value in the kernel will increase every
  53 time the above log entry is inserted into dmesg.  And after digging a
  54 bit further, I discovered that this value show up in
  55 /proc/self/mountstats on Linux.</p>
  56
  57 <p>The mountstats content seem to be shared between files using the
  58 same file system context, so it is enough to check one of the
  59 mountstats files to get the state of the mount point for the machine.
  60 I assume this will not show lazy umounted NFS points, nor NFS mount
  61 points in a different process context (ie with a different filesystem
  62 view), but that does not worry me.</p>
  63
  64 <p>The content for a NFS mount point look similar to this:</p>
  65
  66 <p><blockquote><pre>
  67 [...]
  68 device /dev/mapper/Debian-var mounted on /var with fstype ext3
  69 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
  70         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
  71         age:    7863311
  72         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
  73         sec:    flavor=1,pseudoflavor=1
  74         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
  75         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
  76         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
  77         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
  78         per-op statistics
  79                 NULL: 0 0 0 0 0 0 0 0
  80              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
  81              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
  82               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
  83               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
  84             READLINK: 125 125 0 20472 18620 0 1112 1118
  85                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
  86                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
  87               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
  88                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
  89              SYMLINK: 903 903 0 233428 245488 6 5865 5917
  90                MKNOD: 80 80 0 20148 21760 0 299 304
  91               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
  92                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
  93               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
  94                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
  95              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
  96          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
  97               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
  98               FSINFO: 2 2 0 232 328 0 1 1
  99             PATHCONF: 1 1 0 116 140 0 0 0
 100               COMMIT: 0 0 0 0 0 0 0 0
 101
 102 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
 103 [...]
 104 </pre></blockquote></p>
 105
 106 <p>The key number to look at is the third number in the per-op list.
 107 It is the number of NFS timeouts experiences per file system
 108 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 109 numbers are increasing, I believe the machine is experiencing NFS
 110 hang.  Unfortunately the timeout value do not start to increase right
 111 away.  The NFS operations need to time out first, and this can take a
 112 while.  The exact timeout value depend on the setup.  For example the
 113 defaults for TCP and UDP mount points are quite different, and the
 114 timeout value is affected by the soft, hard, timeo and retrans NFS
 115 mount options.</p>
 116
 117 <p>The only way I have been able to get working on Debian and RedHat
 118 Enterprise Linux for getting the timeout count is to peek in /proc/.
 119 But according to
 120 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 121 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 122 command can be used to get these timeout values.  But this do not work
 123 on Linux, as far as I can tell.  I
 124 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 125 but have not seen any replies yet.</p>
 126
 127 <p>Is there a better way to figure out if a Linux NFS client is
 128 experiencing NFS hangs?  Is there a way to detect which processes are
 129 affected?  Is there a way to get the NFS mount going quickly once the
 130 network problem causing the NFS hang has been cleared?  I would very
 131 much welcome some clues, as we regularly run into NFS hangs.</p>
 132
 133       </div>
 134       <div class="tags">
 135
 136
 137         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 138
 139
 140       </div>
 141     </div>
 142     <div class="padding"></div>
 143
 144     <div class="entry">
 145       <div class="title">
 146         <a href="http://people.skolelinux.org/pere/blog/Debian_Jessie__PXE_and_automatic_firmware_installation.html">Debian Jessie, PXE and automatic firmware installation</a>
 147       </div>
 148       <div class="date">
 149         17th October 2014
 150       </div>
 151       <div class="body">
 152         <p>When PXE installing laptops with Debian, I often run into the
 153 problem that the WiFi card require some firmware to work properly.
 154 And it has been a pain to fix this using preseeding in Debian.
 155 Normally something more is needed.  But thanks to
 156 <a href="https://packages.qa.debian.org/i/isenkram.html">my isenkram
 157 package</a> and its recent tasksel extension, it has now become easy
 158 to do this using simple preseeding.</p>
 159
 160 <p>The isenkram-cli package provide tasksel tasks which will install
 161 firmware for the hardware found in the machine (actually, requested by
 162 the kernel modules for the hardware).  (It can also install user space
 163 programs supporting the hardware detected, but that is not the focus
 164 of this story.)</p>
 165
 166 <p>To get this working in the default installation, two preeseding
 167 values are needed.  First, the isenkram-cli package must be installed
 168 into the target chroot (aka the hard drive) before tasksel is executed
 169 in the pkgsel step of the debian-installer system.  This is done by
 170 preseeding the base-installer/includes debconf value to include the
 171 isenkram-cli package.  The package name is next passed to debootstrap
 172 for installation.  With the isenkram-cli package in place, tasksel
 173 will automatically use the isenkram tasks to detect hardware specific
 174 packages for the machine being installed and install them, because
 175 isenkram-cli contain tasksel tasks.</p>
 176
 177 <p>Second, one need to enable the non-free APT repository, because
 178 most firmware unfortunately is non-free.  This is done by preseeding
 179 the apt-mirror-setup step.  This is unfortunate, but for a lot of
 180 hardware it is the only option in Debian.</p>
 181
 182 <p>The end result is two lines needed in your preseeding file to get
 183 firmware installed automatically by the installer:</p>
 184
 185 <p><blockquote><pre>
 186 base-installer base-installer/includes string isenkram-cli
 187 apt-mirror-setup apt-setup/non-free boolean true
 188 </pre></blockquote></p>
 189
 190 <p>The current version of isenkram-cli in testing/jessie will install
 191 both firmware and user space packages when using this method.  It also
 192 do not work well, so use version 0.15 or later.  Installing both
 193 firmware and user space packages might give you a bit more than you
 194 want, so I decided to split the tasksel task in two, one for firmware
 195 and one for user space programs.  The firmware task is enabled by
 196 default, while the one for user space programs is not.  This split is
 197 implemented in the package currently in unstable.</p>
 198
 199 <p>If you decide to give this a go, please let me know (via email) how
 200 this recipe work for you. :)</p>
 201
 202 <p>So, I bet you are wondering, how can this work.  First and
 203 foremost, it work because tasksel is modular, and driven by whatever
 204 files it find in /usr/lib/tasksel/ and /usr/share/tasksel/.  So the
 205 isenkram-cli package place two files for tasksel to find.  First there
 206 is the task description file (/usr/share/tasksel/descs/isenkram.desc):</p>
 207
 208 <p><blockquote><pre>
 209 Task: isenkram-packages
 210 Section: hardware
 211 Description: Hardware specific packages (autodetected by isenkram)
 212  Based on the detected hardware various hardware specific packages are
 213  proposed.
 214 Test-new-install: show show
 215 Relevance: 8
 216 Packages: for-current-hardware
 217
 218 Task: isenkram-firmware
 219 Section: hardware
 220 Description: Hardware specific firmware packages (autodetected by isenkram)
 221  Based on the detected hardware various hardware specific firmware
 222  packages are proposed.
 223 Test-new-install: mark show
 224 Relevance: 8
 225 Packages: for-current-hardware-firmware
 226 </pre></blockquote></p>
 227
 228 <p>The key parts are Test-new-install which indicate how the task
 229 should be handled and the Packages line referencing to a script in
 230 /usr/lib/tasksel/packages/.  The scripts use other scripts to get a
 231 list of packages to install.  The for-current-hardware-firmware script
 232 look like this to list relevant firmware for the machine:
 233
 234 <p><blockquote><pre>
 235 #!/bin/sh
 236 #
 237 PATH=/usr/sbin:$PATH
 238 export PATH
 239 isenkram-autoinstall-firmware -l
 240 </pre></blockquote></p>
 241
 242 <p>With those two pieces in place, the firmware is installed by
 243 tasksel during the normal d-i run. :)</p>
 244
 245 <p>If you want to test what tasksel will install when isenkram-cli is
 246 installed, run <tt>DEBIAN_PRIORITY=critical tasksel --test
 247 --new-install</tt> to get the list of packages that tasksel would
 248 install.</p>
 249
 250 <p><a href="https://wiki.debian.org/DebianEdu/">Debian Edu</a> will be
 251 pilots in testing this feature, as isenkram is used there now to
 252 install firmware, replacing the earlier scripts.</p>
 253
 254       </div>
 255       <div class="tags">
 256
 257
 258         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 259
 260
 261       </div>
 262     </div>
 263     <div class="padding"></div>
 264
 265     <div class="entry">
 266       <div class="title">
 267         <a href="http://people.skolelinux.org/pere/blog/Scripting_the_Cerebrum_bofhd_user_administration_system_using_XML_RPC.html">Scripting the Cerebrum/bofhd user administration system using XML-RPC</a>
 268       </div>
 269       <div class="date">
 270          6th December 2012
 271       </div>
 272       <div class="body">
 273         <p>Where I work at the <a href="http://www.uio.no/">University of
 274 Oslo</a>, we use the
 275 <a href="http://sourceforge.net/projects/cerebrum/">Cerebrum user
 276 administration system</a> to maintain users, groups, DNS, DHCP, etc.
 277 I've known since the system was written that the server is providing
 278 an <a href="http://en.wikipedia.org/wiki/XML-RPC">XML-RPC</a> API, but
 279 I have never spent time to try to figure out how to use it, as we
 280 always use the bofh command line client at work.  Until today.  I want
 281 to script the updating of DNS and DHCP to make it easier to set up
 282 virtual machines.  Here are a few notes on how to use it with
 283 Python.</p>
 284
 285 <p>I started by looking at the source of the Java
 286 <a href="http://cerebrum.svn.sourceforge.net/viewvc/cerebrum/trunk/cerebrum/clients/jbofh/">bofh
 287 client</a>, to figure out how it connected to the API server.  I also
 288 googled for python examples on how to use XML-RPC, and found
 289 <a href="http://tldp.org/HOWTO/XML-RPC-HOWTO/xmlrpc-howto-python.html">a
 290 simple example in</a> the XML-RPC howto.</p>
 291
 292 <p>This simple example code show how to connect, get the list of
 293 commands (as a JSON dump), and how to get the information about the
 294 user currently logged in:</p>
 295
 296 <blockquote><pre>
 297 #!/usr/bin/env python
 298 import getpass
 299 import xmlrpclib
 300 server_url = 'https://cerebrum-uio.uio.no:8000';
 301 username = getpass.getuser()
 302 password = getpass.getpass()
 303 server = xmlrpclib.Server(server_url);
 304 #print server.get_commands(sessionid)
 305 sessionid = server.login(username, password)
 306 print server.run_command(sessionid, "user_info", username)
 307 result = server.logout(sessionid)
 308 print result
 309 </pre></blockquote>
 310
 311 <p>Armed with this knowledge I can now move forward and script the DNS
 312 and DHCP updates I wanted to do.</p>
 313
 314       </div>
 315       <div class="tags">
 316
 317
 318         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 319
 320
 321       </div>
 322     </div>
 323     <div class="padding"></div>
 324
 325     <p style="text-align: right;"><a href="sysadmin.rss"><img src="http://people.skolelinux.org/pere/blog/xml.gif" alt="RSS Feed" width="36" height="14" /></a></p>
 326     <div id="sidebar">
 327
 328
 329
 330 <h2>Archive</h2>
 331 <ul>
 332
 333 <li>2017
 334 <ul>
 335
 336 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 337
 338 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 339
 340 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (5)</a></li>
 341
 342 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
 343
 344 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
 345
 346 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
 347
 348 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
 349
 350 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
 351
 352 </ul></li>
 353
 354 <li>2016
 355 <ul>
 356
 357 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 358
 359 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 360
 361 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 362
 363 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 364
 365 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 366
 367 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 368
 369 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 370
 371 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 372
 373 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 374
 375 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 376
 377 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 378
 379 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 380
 381 </ul></li>
 382
 383 <li>2015
 384 <ul>
 385
 386 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 387
 388 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 389
 390 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 391
 392 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 393
 394 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 395
 396 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 397
 398 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 399
 400 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 401
 402 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 403
 404 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 405
 406 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 407
 408 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 409
 410 </ul></li>
 411
 412 <li>2014
 413 <ul>
 414
 415 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 416
 417 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 418
 419 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 420
 421 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 422
 423 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 424
 425 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 426
 427 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 428
 429 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 430
 431 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 432
 433 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 434
 435 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 436
 437 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 438
 439 </ul></li>
 440
 441 <li>2013
 442 <ul>
 443
 444 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 445
 446 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 447
 448 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 449
 450 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 451
 452 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 453
 454 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 455
 456 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 457
 458 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 459
 460 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 461
 462 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 463
 464 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 465
 466 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 467
 468 </ul></li>
 469
 470 <li>2012
 471 <ul>
 472
 473 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 474
 475 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 476
 477 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 478
 479 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 480
 481 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 482
 483 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 484
 485 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 486
 487 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 488
 489 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 490
 491 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 492
 493 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 494
 495 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 496
 497 </ul></li>
 498
 499 <li>2011
 500 <ul>
 501
 502 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 503
 504 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 505
 506 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 507
 508 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 509
 510 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 511
 512 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 513
 514 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 515
 516 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 517
 518 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 519
 520 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 521
 522 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 523
 524 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 525
 526 </ul></li>
 527
 528 <li>2010
 529 <ul>
 530
 531 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 532
 533 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 534
 535 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 536
 537 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 538
 539 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 540
 541 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 542
 543 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 544
 545 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 546
 547 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 548
 549 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 550
 551 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 552
 553 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 554
 555 </ul></li>
 556
 557 <li>2009
 558 <ul>
 559
 560 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 561
 562 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 563
 564 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 565
 566 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 567
 568 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 569
 570 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 571
 572 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 573
 574 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 575
 576 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 577
 578 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 579
 580 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 581
 582 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 583
 584 </ul></li>
 585
 586 <li>2008
 587 <ul>
 588
 589 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 590
 591 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 592
 593 </ul></li>
 594
 595 </ul>
 596
 597
 598
 599 <h2>Tags</h2>
 600 <ul>
 601
 602  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 603
 604  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 605
 606  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 607
 608  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 609
 610  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
 611
 612  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
 613
 614  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 615
 616  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 617
 618  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (153)</a></li>
 619
 620  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
 621
 622  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (4)</a></li>
 623
 624  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 625
 626  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (17)</a></li>
 627
 628  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (24)</a></li>
 629
 630  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 631
 632  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (353)</a></li>
 633
 634  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 635
 636  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 637
 638  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (30)</a></li>
 639
 640  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 641
 642  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
 643
 644  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 645
 646  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 647
 648  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
 649
 650  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (20)</a></li>
 651
 652  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 653
 654  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 655
 656  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 657
 658  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 659
 660  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 661
 662  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
 663
 664  <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
 665
 666  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (292)</a></li>
 667
 668  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (189)</a></li>
 669
 670  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (33)</a></li>
 671
 672  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 673
 674  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (64)</a></li>
 675
 676  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (104)</a></li>
 677
 678  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 679
 680  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 681
 682  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 683
 684  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 685
 686  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
 687
 688  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 689
 690  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
 691
 692  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 693
 694  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (53)</a></li>
 695
 696  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 697
 698  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 699
 700  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (55)</a></li>
 701
 702  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (6)</a></li>
 703
 704  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
 705
 706  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (52)</a></li>
 707
 708  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (3)</a></li>
 709
 710  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 711
 712  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
 713
 714  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
 715
 716  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 717
 718  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (40)</a></li>
 719
 720 </ul>
 721
 722
 723     </div>
 724     <p style="text-align: right">
 725  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 726 </p>
 727
 728   </body>
 729 </html>