blog/tags/sysadmin/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Entries Tagged sysadmin</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9     <link rel="alternate" title="RSS Feed" href="sysadmin.rss" type="application/rss+xml" />
  10   </head>
  11   <body>
  12     <div class="title">
  13  <h1>
  14      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  15
  16  </h1>
  17
  18 </div>
  19
  20
  21     <h3>Entries tagged "sysadmin".</h3>
  22
  23     <div class="entry">
  24       <div class="title">
  25         <a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a>
  26       </div>
  27       <div class="date">
  28          1st November 2017
  29       </div>
  30       <div class="body">
  31         <p>If you care about how fault tolerant your storage is, you might
  32 find these articles and papers interesting.  They have formed how I
  33 think of when designing a storage system.</p>
  34
  35 <ul>
  36
  37 <li>USENIX :login; <a
  38 href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
  39 Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
  40 Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
  41 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
  42 H. Arpaci-Dusseau</li>
  43
  44 <li>ZDNet
  45 <a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
  46 RAID 5 stops working in 2009</a> by Robin Harris</li>
  47
  48 <li>ZDNet
  49 <a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
  50 RAID 6 stops working in 2019</a> by Robin Harris</li>
  51
  52 <li>USENIX FAST'07
  53 <a href="http://research.google.com/archive/disk_failures.pdf">Failure
  54 Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
  55 Wolf-Dietrich Weber and Luiz André Barroso</li>
  56
  57 <li>USENIX ;login: <a
  58 href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
  59 Integrity.  Finding Truth in a World of Guesses and Lies</a> by Doug
  60 Hughes</li>
  61
  62 <li>USENIX FAST'08
  63 <a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
  64 cAnalysis of Data Corruption in the Storage Stack</a> -
  65 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
  66 Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
  67
  68 <li>USENIX FAST'07 <a
  69 href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
  70 failures in the real world: what does an MTTF of 1,000,000 hours mean
  71 to you?</a> by B. Schroeder and G. A. Gibson.</li>
  72
  73 <li>USENIX ;login: <a
  74 href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
  75 Disks the Dominant Contributor for Storage Failures?  A Comprehensive
  76 Study of Storage Subsystem Failure Characteristics</a> by Weihang
  77 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
  78
  79 <li>SIGMETRICS 2007
  80 <a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
  81 analysis of latent sector errors in disk drives</a> -
  82 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
  83
  84 </ul>
  85
  86 <p>Several of these research papers are based on data collected from
  87 hundred thousands or millions of disk, and their findings are eye
  88 opening.  The short story is simply do not implicitly trust RAID or
  89 redundant storage systems.  Details matter.  And unfortunately there
  90 are few options on Linux addressing all the identified issues.  Both
  91 ZFS and Btrfs are doing a fairly good job, but have legal and
  92 practical issues on their own.  I wonder how cluster file systems like
  93 Ceph do in this regard.</p>
  94
  95 <p>Just remember, in the end, it do not matter how redundant, or how
  96 fault tolerant your storage is, if you do not continuously monitor its
  97 status to detect and replace failed disks.</p>
  98
  99       </div>
 100       <div class="tags">
 101
 102
 103         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 104
 105
 106       </div>
 107     </div>
 108     <div class="padding"></div>
 109
 110     <div class="entry">
 111       <div class="title">
 112         <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a>
 113       </div>
 114       <div class="date">
 115          9th March 2017
 116       </div>
 117       <div class="body">
 118         <p>Over the years, administrating thousand of NFS mounting linux
 119 computers at the time, I often needed a way to detect if the machine
 120 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
 121 file or directory affected by the hang, the process (and possibly the
 122 shell) will hang too.  So you want to be able to detect this without
 123 risking the detection process getting stuck too.  It has not been
 124 obvious how to do this.  When the hang has lasted a while, it is
 125 possible to find messages like these in dmesg:</p>
 126
 127 <p><blockquote>
 128 nfs: server nfsserver not responding, still trying
 129 <br>nfs: server nfsserver OK
 130 </blockquote></p>
 131
 132 <p>It is hard to know if the hang is still going on, and it is hard to
 133 be sure looking in dmesg is going to work.  If there are lots of other
 134 messages in dmesg the lines might have rotated out of site before they
 135 are noticed.</p>
 136
 137 <p>While reading through the nfs client implementation in linux kernel
 138 code, I came across some statistics that seem to give a way to detect
 139 it.  The om_timeouts sunrpc value in the kernel will increase every
 140 time the above log entry is inserted into dmesg.  And after digging a
 141 bit further, I discovered that this value show up in
 142 /proc/self/mountstats on Linux.</p>
 143
 144 <p>The mountstats content seem to be shared between files using the
 145 same file system context, so it is enough to check one of the
 146 mountstats files to get the state of the mount point for the machine.
 147 I assume this will not show lazy umounted NFS points, nor NFS mount
 148 points in a different process context (ie with a different filesystem
 149 view), but that does not worry me.</p>
 150
 151 <p>The content for a NFS mount point look similar to this:</p>
 152
 153 <p><blockquote><pre>
 154 [...]
 155 device /dev/mapper/Debian-var mounted on /var with fstype ext3
 156 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
 157         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
 158         age:    7863311
 159         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
 160         sec:    flavor=1,pseudoflavor=1
 161         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
 162         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
 163         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
 164         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
 165         per-op statistics
 166                 NULL: 0 0 0 0 0 0 0 0
 167              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
 168              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
 169               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
 170               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
 171             READLINK: 125 125 0 20472 18620 0 1112 1118
 172                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
 173                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
 174               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
 175                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
 176              SYMLINK: 903 903 0 233428 245488 6 5865 5917
 177                MKNOD: 80 80 0 20148 21760 0 299 304
 178               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
 179                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
 180               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
 181                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
 182              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
 183          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
 184               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
 185               FSINFO: 2 2 0 232 328 0 1 1
 186             PATHCONF: 1 1 0 116 140 0 0 0
 187               COMMIT: 0 0 0 0 0 0 0 0
 188
 189 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
 190 [...]
 191 </pre></blockquote></p>
 192
 193 <p>The key number to look at is the third number in the per-op list.
 194 It is the number of NFS timeouts experiences per file system
 195 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 196 numbers are increasing, I believe the machine is experiencing NFS
 197 hang.  Unfortunately the timeout value do not start to increase right
 198 away.  The NFS operations need to time out first, and this can take a
 199 while.  The exact timeout value depend on the setup.  For example the
 200 defaults for TCP and UDP mount points are quite different, and the
 201 timeout value is affected by the soft, hard, timeo and retrans NFS
 202 mount options.</p>
 203
 204 <p>The only way I have been able to get working on Debian and RedHat
 205 Enterprise Linux for getting the timeout count is to peek in /proc/.
 206 But according to
 207 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 208 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 209 command can be used to get these timeout values.  But this do not work
 210 on Linux, as far as I can tell.  I
 211 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 212 but have not seen any replies yet.</p>
 213
 214 <p>Is there a better way to figure out if a Linux NFS client is
 215 experiencing NFS hangs?  Is there a way to detect which processes are
 216 affected?  Is there a way to get the NFS mount going quickly once the
 217 network problem causing the NFS hang has been cleared?  I would very
 218 much welcome some clues, as we regularly run into NFS hangs.</p>
 219
 220       </div>
 221       <div class="tags">
 222
 223
 224         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 225
 226
 227       </div>
 228     </div>
 229     <div class="padding"></div>
 230
 231     <div class="entry">
 232       <div class="title">
 233         <a href="http://people.skolelinux.org/pere/blog/Debian_Jessie__PXE_and_automatic_firmware_installation.html">Debian Jessie, PXE and automatic firmware installation</a>
 234       </div>
 235       <div class="date">
 236         17th October 2014
 237       </div>
 238       <div class="body">
 239         <p>When PXE installing laptops with Debian, I often run into the
 240 problem that the WiFi card require some firmware to work properly.
 241 And it has been a pain to fix this using preseeding in Debian.
 242 Normally something more is needed.  But thanks to
 243 <a href="https://packages.qa.debian.org/i/isenkram.html">my isenkram
 244 package</a> and its recent tasksel extension, it has now become easy
 245 to do this using simple preseeding.</p>
 246
 247 <p>The isenkram-cli package provide tasksel tasks which will install
 248 firmware for the hardware found in the machine (actually, requested by
 249 the kernel modules for the hardware).  (It can also install user space
 250 programs supporting the hardware detected, but that is not the focus
 251 of this story.)</p>
 252
 253 <p>To get this working in the default installation, two preeseding
 254 values are needed.  First, the isenkram-cli package must be installed
 255 into the target chroot (aka the hard drive) before tasksel is executed
 256 in the pkgsel step of the debian-installer system.  This is done by
 257 preseeding the base-installer/includes debconf value to include the
 258 isenkram-cli package.  The package name is next passed to debootstrap
 259 for installation.  With the isenkram-cli package in place, tasksel
 260 will automatically use the isenkram tasks to detect hardware specific
 261 packages for the machine being installed and install them, because
 262 isenkram-cli contain tasksel tasks.</p>
 263
 264 <p>Second, one need to enable the non-free APT repository, because
 265 most firmware unfortunately is non-free.  This is done by preseeding
 266 the apt-mirror-setup step.  This is unfortunate, but for a lot of
 267 hardware it is the only option in Debian.</p>
 268
 269 <p>The end result is two lines needed in your preseeding file to get
 270 firmware installed automatically by the installer:</p>
 271
 272 <p><blockquote><pre>
 273 base-installer base-installer/includes string isenkram-cli
 274 apt-mirror-setup apt-setup/non-free boolean true
 275 </pre></blockquote></p>
 276
 277 <p>The current version of isenkram-cli in testing/jessie will install
 278 both firmware and user space packages when using this method.  It also
 279 do not work well, so use version 0.15 or later.  Installing both
 280 firmware and user space packages might give you a bit more than you
 281 want, so I decided to split the tasksel task in two, one for firmware
 282 and one for user space programs.  The firmware task is enabled by
 283 default, while the one for user space programs is not.  This split is
 284 implemented in the package currently in unstable.</p>
 285
 286 <p>If you decide to give this a go, please let me know (via email) how
 287 this recipe work for you. :)</p>
 288
 289 <p>So, I bet you are wondering, how can this work.  First and
 290 foremost, it work because tasksel is modular, and driven by whatever
 291 files it find in /usr/lib/tasksel/ and /usr/share/tasksel/.  So the
 292 isenkram-cli package place two files for tasksel to find.  First there
 293 is the task description file (/usr/share/tasksel/descs/isenkram.desc):</p>
 294
 295 <p><blockquote><pre>
 296 Task: isenkram-packages
 297 Section: hardware
 298 Description: Hardware specific packages (autodetected by isenkram)
 299  Based on the detected hardware various hardware specific packages are
 300  proposed.
 301 Test-new-install: show show
 302 Relevance: 8
 303 Packages: for-current-hardware
 304
 305 Task: isenkram-firmware
 306 Section: hardware
 307 Description: Hardware specific firmware packages (autodetected by isenkram)
 308  Based on the detected hardware various hardware specific firmware
 309  packages are proposed.
 310 Test-new-install: mark show
 311 Relevance: 8
 312 Packages: for-current-hardware-firmware
 313 </pre></blockquote></p>
 314
 315 <p>The key parts are Test-new-install which indicate how the task
 316 should be handled and the Packages line referencing to a script in
 317 /usr/lib/tasksel/packages/.  The scripts use other scripts to get a
 318 list of packages to install.  The for-current-hardware-firmware script
 319 look like this to list relevant firmware for the machine:
 320
 321 <p><blockquote><pre>
 322 #!/bin/sh
 323 #
 324 PATH=/usr/sbin:$PATH
 325 export PATH
 326 isenkram-autoinstall-firmware -l
 327 </pre></blockquote></p>
 328
 329 <p>With those two pieces in place, the firmware is installed by
 330 tasksel during the normal d-i run. :)</p>
 331
 332 <p>If you want to test what tasksel will install when isenkram-cli is
 333 installed, run <tt>DEBIAN_PRIORITY=critical tasksel --test
 334 --new-install</tt> to get the list of packages that tasksel would
 335 install.</p>
 336
 337 <p><a href="https://wiki.debian.org/DebianEdu/">Debian Edu</a> will be
 338 pilots in testing this feature, as isenkram is used there now to
 339 install firmware, replacing the earlier scripts.</p>
 340
 341       </div>
 342       <div class="tags">
 343
 344
 345         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 346
 347
 348       </div>
 349     </div>
 350     <div class="padding"></div>
 351
 352     <div class="entry">
 353       <div class="title">
 354         <a href="http://people.skolelinux.org/pere/blog/Scripting_the_Cerebrum_bofhd_user_administration_system_using_XML_RPC.html">Scripting the Cerebrum/bofhd user administration system using XML-RPC</a>
 355       </div>
 356       <div class="date">
 357          6th December 2012
 358       </div>
 359       <div class="body">
 360         <p>Where I work at the <a href="http://www.uio.no/">University of
 361 Oslo</a>, we use the
 362 <a href="http://sourceforge.net/projects/cerebrum/">Cerebrum user
 363 administration system</a> to maintain users, groups, DNS, DHCP, etc.
 364 I've known since the system was written that the server is providing
 365 an <a href="http://en.wikipedia.org/wiki/XML-RPC">XML-RPC</a> API, but
 366 I have never spent time to try to figure out how to use it, as we
 367 always use the bofh command line client at work.  Until today.  I want
 368 to script the updating of DNS and DHCP to make it easier to set up
 369 virtual machines.  Here are a few notes on how to use it with
 370 Python.</p>
 371
 372 <p>I started by looking at the source of the Java
 373 <a href="http://cerebrum.svn.sourceforge.net/viewvc/cerebrum/trunk/cerebrum/clients/jbofh/">bofh
 374 client</a>, to figure out how it connected to the API server.  I also
 375 googled for python examples on how to use XML-RPC, and found
 376 <a href="http://tldp.org/HOWTO/XML-RPC-HOWTO/xmlrpc-howto-python.html">a
 377 simple example in</a> the XML-RPC howto.</p>
 378
 379 <p>This simple example code show how to connect, get the list of
 380 commands (as a JSON dump), and how to get the information about the
 381 user currently logged in:</p>
 382
 383 <blockquote><pre>
 384 #!/usr/bin/env python
 385 import getpass
 386 import xmlrpclib
 387 server_url = 'https://cerebrum-uio.uio.no:8000';
 388 username = getpass.getuser()
 389 password = getpass.getpass()
 390 server = xmlrpclib.Server(server_url);
 391 #print server.get_commands(sessionid)
 392 sessionid = server.login(username, password)
 393 print server.run_command(sessionid, "user_info", username)
 394 result = server.logout(sessionid)
 395 print result
 396 </pre></blockquote>
 397
 398 <p>Armed with this knowledge I can now move forward and script the DNS
 399 and DHCP updates I wanted to do.</p>
 400
 401       </div>
 402       <div class="tags">
 403
 404
 405         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 406
 407
 408       </div>
 409     </div>
 410     <div class="padding"></div>
 411
 412     <p style="text-align: right;"><a href="sysadmin.rss"><img src="http://people.skolelinux.org/pere/blog/xml.gif" alt="RSS Feed" width="36" height="14" /></a></p>
 413     <div id="sidebar">
 414
 415
 416
 417 <h2>Archive</h2>
 418 <ul>
 419
 420 <li>2017
 421 <ul>
 422
 423 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 424
 425 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 426
 427 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (5)</a></li>
 428
 429 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
 430
 431 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
 432
 433 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
 434
 435 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
 436
 437 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
 438
 439 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
 440
 441 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (1)</a></li>
 442
 443 </ul></li>
 444
 445 <li>2016
 446 <ul>
 447
 448 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 449
 450 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 451
 452 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 453
 454 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 455
 456 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 457
 458 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 459
 460 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 461
 462 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 463
 464 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 465
 466 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 467
 468 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 469
 470 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 471
 472 </ul></li>
 473
 474 <li>2015
 475 <ul>
 476
 477 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 478
 479 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 480
 481 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 482
 483 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 484
 485 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 486
 487 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 488
 489 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 490
 491 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 492
 493 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 494
 495 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 496
 497 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 498
 499 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 500
 501 </ul></li>
 502
 503 <li>2014
 504 <ul>
 505
 506 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 507
 508 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 509
 510 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 511
 512 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 513
 514 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 515
 516 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 517
 518 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 519
 520 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 521
 522 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 523
 524 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 525
 526 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 527
 528 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 529
 530 </ul></li>
 531
 532 <li>2013
 533 <ul>
 534
 535 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 536
 537 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 538
 539 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 540
 541 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 542
 543 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 544
 545 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 546
 547 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 548
 549 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 550
 551 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 552
 553 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 554
 555 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 556
 557 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 558
 559 </ul></li>
 560
 561 <li>2012
 562 <ul>
 563
 564 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 565
 566 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 567
 568 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 569
 570 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 571
 572 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 573
 574 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 575
 576 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 577
 578 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 579
 580 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 581
 582 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 583
 584 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 585
 586 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 587
 588 </ul></li>
 589
 590 <li>2011
 591 <ul>
 592
 593 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 594
 595 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 596
 597 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 598
 599 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 600
 601 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 602
 603 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 604
 605 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 606
 607 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 608
 609 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 610
 611 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 612
 613 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 614
 615 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 616
 617 </ul></li>
 618
 619 <li>2010
 620 <ul>
 621
 622 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 623
 624 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 625
 626 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 627
 628 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 629
 630 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 631
 632 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 633
 634 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 635
 636 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 637
 638 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 639
 640 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 641
 642 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 643
 644 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 645
 646 </ul></li>
 647
 648 <li>2009
 649 <ul>
 650
 651 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 652
 653 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 654
 655 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 656
 657 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 658
 659 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 660
 661 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 662
 663 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 664
 665 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 666
 667 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 668
 669 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 670
 671 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 672
 673 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 674
 675 </ul></li>
 676
 677 <li>2008
 678 <ul>
 679
 680 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 681
 682 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 683
 684 </ul></li>
 685
 686 </ul>
 687
 688
 689
 690 <h2>Tags</h2>
 691 <ul>
 692
 693  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (14)</a></li>
 694
 695  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 696
 697  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 698
 699  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 700
 701  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
 702
 703  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
 704
 705  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 706
 707  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 708
 709  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (154)</a></li>
 710
 711  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
 712
 713  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (4)</a></li>
 714
 715  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 716
 717  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (17)</a></li>
 718
 719  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (24)</a></li>
 720
 721  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 722
 723  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (358)</a></li>
 724
 725  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 726
 727  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 728
 729  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (30)</a></li>
 730
 731  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 732
 733  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
 734
 735  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 736
 737  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 738
 739  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
 740
 741  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (20)</a></li>
 742
 743  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 744
 745  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 746
 747  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 748
 749  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 750
 751  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 752
 753  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
 754
 755  <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
 756
 757  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (293)</a></li>
 758
 759  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (189)</a></li>
 760
 761  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (33)</a></li>
 762
 763  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 764
 765  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (65)</a></li>
 766
 767  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (104)</a></li>
 768
 769  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (2)</a></li>
 770
 771  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 772
 773  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 774
 775  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 776
 777  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
 778
 779  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 780
 781  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
 782
 783  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 784
 785  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (53)</a></li>
 786
 787  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 788
 789  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 790
 791  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (55)</a></li>
 792
 793  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (6)</a></li>
 794
 795  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
 796
 797  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (52)</a></li>
 798
 799  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (4)</a></li>
 800
 801  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 802
 803  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
 804
 805  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
 806
 807  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 808
 809  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (40)</a></li>
 810
 811 </ul>
 812
 813
 814     </div>
 815     <p style="text-align: right">
 816  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 817 </p>
 818
 819   </body>
 820 </html>