blog/tags/sysadmin/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Entries Tagged sysadmin</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9     <link rel="alternate" title="RSS Feed" href="sysadmin.rss" type="application/rss+xml" />
  10   </head>
  11   <body>
  12     <div class="title">
  13  <h1>
  14      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  15
  16  </h1>
  17
  18 </div>
  19
  20
  21     <h3>Entries tagged "sysadmin".</h3>
  22
  23     <div class="entry">
  24       <div class="title">
  25         <a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a>
  26       </div>
  27       <div class="date">
  28          1st November 2017
  29       </div>
  30       <div class="body">
  31         <p>If you care about how fault tolerant your storage is, you might
  32 find these articles and papers interesting.  They have formed how I
  33 think of when designing a storage system.</p>
  34
  35 <ul>
  36
  37 <li>USENIX :login; <a
  38 href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
  39 Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
  40 Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
  41 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
  42 H. Arpaci-Dusseau</li>
  43
  44 <li>ZDNet
  45 <a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
  46 RAID 5 stops working in 2009</a> by Robin Harris</li>
  47
  48 <li>ZDNet
  49 <a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
  50 RAID 6 stops working in 2019</a> by Robin Harris</li>
  51
  52 <li>USENIX FAST'07
  53 <a href="http://research.google.com/archive/disk_failures.pdf">Failure
  54 Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
  55 Wolf-Dietrich Weber and Luiz André Barroso</li>
  56
  57 <li>USENIX ;login: <a
  58 href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
  59 Integrity.  Finding Truth in a World of Guesses and Lies</a> by Doug
  60 Hughes</li>
  61
  62 <li>USENIX FAST'08
  63 <a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
  64 Analysis of Data Corruption in the Storage Stack</a> by
  65 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
  66 Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
  67
  68 <li>USENIX FAST'07 <a
  69 href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
  70 failures in the real world: what does an MTTF of 1,000,000 hours mean
  71 to you?</a> by B. Schroeder and G. A. Gibson.</li>
  72
  73 <li>USENIX ;login: <a
  74 href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
  75 Disks the Dominant Contributor for Storage Failures?  A Comprehensive
  76 Study of Storage Subsystem Failure Characteristics</a> by Weihang
  77 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
  78
  79 <li>SIGMETRICS 2007
  80 <a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
  81 analysis of latent sector errors in disk drives</a> by
  82 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
  83
  84 </ul>
  85
  86 <p>Several of these research papers are based on data collected from
  87 hundred thousands or millions of disk, and their findings are eye
  88 opening.  The short story is simply do not implicitly trust RAID or
  89 redundant storage systems.  Details matter.  And unfortunately there
  90 are few options on Linux addressing all the identified issues.  Both
  91 ZFS and Btrfs are doing a fairly good job, but have legal and
  92 practical issues on their own.  I wonder how cluster file systems like
  93 Ceph do in this regard.  After all, there is an old saying, you know
  94 you have a distributed system when the crash of a compyter you have
  95 never heard of stops you from getting any work done.  The same holds
  96 true if fault tolerance do not work.</p>
  97
  98 <p>Just remember, in the end, it do not matter how redundant, or how
  99 fault tolerant your storage is, if you do not continuously monitor its
 100 status to detect and replace failed disks.</p>
 101
 102       </div>
 103       <div class="tags">
 104
 105
 106         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 107
 108
 109       </div>
 110     </div>
 111     <div class="padding"></div>
 112
 113     <div class="entry">
 114       <div class="title">
 115         <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a>
 116       </div>
 117       <div class="date">
 118          9th March 2017
 119       </div>
 120       <div class="body">
 121         <p>Over the years, administrating thousand of NFS mounting linux
 122 computers at the time, I often needed a way to detect if the machine
 123 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
 124 file or directory affected by the hang, the process (and possibly the
 125 shell) will hang too.  So you want to be able to detect this without
 126 risking the detection process getting stuck too.  It has not been
 127 obvious how to do this.  When the hang has lasted a while, it is
 128 possible to find messages like these in dmesg:</p>
 129
 130 <p><blockquote>
 131 nfs: server nfsserver not responding, still trying
 132 <br>nfs: server nfsserver OK
 133 </blockquote></p>
 134
 135 <p>It is hard to know if the hang is still going on, and it is hard to
 136 be sure looking in dmesg is going to work.  If there are lots of other
 137 messages in dmesg the lines might have rotated out of site before they
 138 are noticed.</p>
 139
 140 <p>While reading through the nfs client implementation in linux kernel
 141 code, I came across some statistics that seem to give a way to detect
 142 it.  The om_timeouts sunrpc value in the kernel will increase every
 143 time the above log entry is inserted into dmesg.  And after digging a
 144 bit further, I discovered that this value show up in
 145 /proc/self/mountstats on Linux.</p>
 146
 147 <p>The mountstats content seem to be shared between files using the
 148 same file system context, so it is enough to check one of the
 149 mountstats files to get the state of the mount point for the machine.
 150 I assume this will not show lazy umounted NFS points, nor NFS mount
 151 points in a different process context (ie with a different filesystem
 152 view), but that does not worry me.</p>
 153
 154 <p>The content for a NFS mount point look similar to this:</p>
 155
 156 <p><blockquote><pre>
 157 [...]
 158 device /dev/mapper/Debian-var mounted on /var with fstype ext3
 159 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
 160         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
 161         age:    7863311
 162         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
 163         sec:    flavor=1,pseudoflavor=1
 164         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
 165         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
 166         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
 167         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
 168         per-op statistics
 169                 NULL: 0 0 0 0 0 0 0 0
 170              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
 171              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
 172               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
 173               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
 174             READLINK: 125 125 0 20472 18620 0 1112 1118
 175                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
 176                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
 177               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
 178                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
 179              SYMLINK: 903 903 0 233428 245488 6 5865 5917
 180                MKNOD: 80 80 0 20148 21760 0 299 304
 181               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
 182                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
 183               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
 184                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
 185              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
 186          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
 187               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
 188               FSINFO: 2 2 0 232 328 0 1 1
 189             PATHCONF: 1 1 0 116 140 0 0 0
 190               COMMIT: 0 0 0 0 0 0 0 0
 191
 192 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
 193 [...]
 194 </pre></blockquote></p>
 195
 196 <p>The key number to look at is the third number in the per-op list.
 197 It is the number of NFS timeouts experiences per file system
 198 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 199 numbers are increasing, I believe the machine is experiencing NFS
 200 hang.  Unfortunately the timeout value do not start to increase right
 201 away.  The NFS operations need to time out first, and this can take a
 202 while.  The exact timeout value depend on the setup.  For example the
 203 defaults for TCP and UDP mount points are quite different, and the
 204 timeout value is affected by the soft, hard, timeo and retrans NFS
 205 mount options.</p>
 206
 207 <p>The only way I have been able to get working on Debian and RedHat
 208 Enterprise Linux for getting the timeout count is to peek in /proc/.
 209 But according to
 210 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 211 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 212 command can be used to get these timeout values.  But this do not work
 213 on Linux, as far as I can tell.  I
 214 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 215 but have not seen any replies yet.</p>
 216
 217 <p>Is there a better way to figure out if a Linux NFS client is
 218 experiencing NFS hangs?  Is there a way to detect which processes are
 219 affected?  Is there a way to get the NFS mount going quickly once the
 220 network problem causing the NFS hang has been cleared?  I would very
 221 much welcome some clues, as we regularly run into NFS hangs.</p>
 222
 223       </div>
 224       <div class="tags">
 225
 226
 227         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 228
 229
 230       </div>
 231     </div>
 232     <div class="padding"></div>
 233
 234     <div class="entry">
 235       <div class="title">
 236         <a href="http://people.skolelinux.org/pere/blog/Debian_Jessie__PXE_and_automatic_firmware_installation.html">Debian Jessie, PXE and automatic firmware installation</a>
 237       </div>
 238       <div class="date">
 239         17th October 2014
 240       </div>
 241       <div class="body">
 242         <p>When PXE installing laptops with Debian, I often run into the
 243 problem that the WiFi card require some firmware to work properly.
 244 And it has been a pain to fix this using preseeding in Debian.
 245 Normally something more is needed.  But thanks to
 246 <a href="https://packages.qa.debian.org/i/isenkram.html">my isenkram
 247 package</a> and its recent tasksel extension, it has now become easy
 248 to do this using simple preseeding.</p>
 249
 250 <p>The isenkram-cli package provide tasksel tasks which will install
 251 firmware for the hardware found in the machine (actually, requested by
 252 the kernel modules for the hardware).  (It can also install user space
 253 programs supporting the hardware detected, but that is not the focus
 254 of this story.)</p>
 255
 256 <p>To get this working in the default installation, two preeseding
 257 values are needed.  First, the isenkram-cli package must be installed
 258 into the target chroot (aka the hard drive) before tasksel is executed
 259 in the pkgsel step of the debian-installer system.  This is done by
 260 preseeding the base-installer/includes debconf value to include the
 261 isenkram-cli package.  The package name is next passed to debootstrap
 262 for installation.  With the isenkram-cli package in place, tasksel
 263 will automatically use the isenkram tasks to detect hardware specific
 264 packages for the machine being installed and install them, because
 265 isenkram-cli contain tasksel tasks.</p>
 266
 267 <p>Second, one need to enable the non-free APT repository, because
 268 most firmware unfortunately is non-free.  This is done by preseeding
 269 the apt-mirror-setup step.  This is unfortunate, but for a lot of
 270 hardware it is the only option in Debian.</p>
 271
 272 <p>The end result is two lines needed in your preseeding file to get
 273 firmware installed automatically by the installer:</p>
 274
 275 <p><blockquote><pre>
 276 base-installer base-installer/includes string isenkram-cli
 277 apt-mirror-setup apt-setup/non-free boolean true
 278 </pre></blockquote></p>
 279
 280 <p>The current version of isenkram-cli in testing/jessie will install
 281 both firmware and user space packages when using this method.  It also
 282 do not work well, so use version 0.15 or later.  Installing both
 283 firmware and user space packages might give you a bit more than you
 284 want, so I decided to split the tasksel task in two, one for firmware
 285 and one for user space programs.  The firmware task is enabled by
 286 default, while the one for user space programs is not.  This split is
 287 implemented in the package currently in unstable.</p>
 288
 289 <p>If you decide to give this a go, please let me know (via email) how
 290 this recipe work for you. :)</p>
 291
 292 <p>So, I bet you are wondering, how can this work.  First and
 293 foremost, it work because tasksel is modular, and driven by whatever
 294 files it find in /usr/lib/tasksel/ and /usr/share/tasksel/.  So the
 295 isenkram-cli package place two files for tasksel to find.  First there
 296 is the task description file (/usr/share/tasksel/descs/isenkram.desc):</p>
 297
 298 <p><blockquote><pre>
 299 Task: isenkram-packages
 300 Section: hardware
 301 Description: Hardware specific packages (autodetected by isenkram)
 302  Based on the detected hardware various hardware specific packages are
 303  proposed.
 304 Test-new-install: show show
 305 Relevance: 8
 306 Packages: for-current-hardware
 307
 308 Task: isenkram-firmware
 309 Section: hardware
 310 Description: Hardware specific firmware packages (autodetected by isenkram)
 311  Based on the detected hardware various hardware specific firmware
 312  packages are proposed.
 313 Test-new-install: mark show
 314 Relevance: 8
 315 Packages: for-current-hardware-firmware
 316 </pre></blockquote></p>
 317
 318 <p>The key parts are Test-new-install which indicate how the task
 319 should be handled and the Packages line referencing to a script in
 320 /usr/lib/tasksel/packages/.  The scripts use other scripts to get a
 321 list of packages to install.  The for-current-hardware-firmware script
 322 look like this to list relevant firmware for the machine:
 323
 324 <p><blockquote><pre>
 325 #!/bin/sh
 326 #
 327 PATH=/usr/sbin:$PATH
 328 export PATH
 329 isenkram-autoinstall-firmware -l
 330 </pre></blockquote></p>
 331
 332 <p>With those two pieces in place, the firmware is installed by
 333 tasksel during the normal d-i run. :)</p>
 334
 335 <p>If you want to test what tasksel will install when isenkram-cli is
 336 installed, run <tt>DEBIAN_PRIORITY=critical tasksel --test
 337 --new-install</tt> to get the list of packages that tasksel would
 338 install.</p>
 339
 340 <p><a href="https://wiki.debian.org/DebianEdu/">Debian Edu</a> will be
 341 pilots in testing this feature, as isenkram is used there now to
 342 install firmware, replacing the earlier scripts.</p>
 343
 344       </div>
 345       <div class="tags">
 346
 347
 348         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 349
 350
 351       </div>
 352     </div>
 353     <div class="padding"></div>
 354
 355     <div class="entry">
 356       <div class="title">
 357         <a href="http://people.skolelinux.org/pere/blog/Scripting_the_Cerebrum_bofhd_user_administration_system_using_XML_RPC.html">Scripting the Cerebrum/bofhd user administration system using XML-RPC</a>
 358       </div>
 359       <div class="date">
 360          6th December 2012
 361       </div>
 362       <div class="body">
 363         <p>Where I work at the <a href="http://www.uio.no/">University of
 364 Oslo</a>, we use the
 365 <a href="http://sourceforge.net/projects/cerebrum/">Cerebrum user
 366 administration system</a> to maintain users, groups, DNS, DHCP, etc.
 367 I've known since the system was written that the server is providing
 368 an <a href="http://en.wikipedia.org/wiki/XML-RPC">XML-RPC</a> API, but
 369 I have never spent time to try to figure out how to use it, as we
 370 always use the bofh command line client at work.  Until today.  I want
 371 to script the updating of DNS and DHCP to make it easier to set up
 372 virtual machines.  Here are a few notes on how to use it with
 373 Python.</p>
 374
 375 <p>I started by looking at the source of the Java
 376 <a href="http://cerebrum.svn.sourceforge.net/viewvc/cerebrum/trunk/cerebrum/clients/jbofh/">bofh
 377 client</a>, to figure out how it connected to the API server.  I also
 378 googled for python examples on how to use XML-RPC, and found
 379 <a href="http://tldp.org/HOWTO/XML-RPC-HOWTO/xmlrpc-howto-python.html">a
 380 simple example in</a> the XML-RPC howto.</p>
 381
 382 <p>This simple example code show how to connect, get the list of
 383 commands (as a JSON dump), and how to get the information about the
 384 user currently logged in:</p>
 385
 386 <blockquote><pre>
 387 #!/usr/bin/env python
 388 import getpass
 389 import xmlrpclib
 390 server_url = 'https://cerebrum-uio.uio.no:8000';
 391 username = getpass.getuser()
 392 password = getpass.getpass()
 393 server = xmlrpclib.Server(server_url);
 394 #print server.get_commands(sessionid)
 395 sessionid = server.login(username, password)
 396 print server.run_command(sessionid, "user_info", username)
 397 result = server.logout(sessionid)
 398 print result
 399 </pre></blockquote>
 400
 401 <p>Armed with this knowledge I can now move forward and script the DNS
 402 and DHCP updates I wanted to do.</p>
 403
 404       </div>
 405       <div class="tags">
 406
 407
 408         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 409
 410
 411       </div>
 412     </div>
 413     <div class="padding"></div>
 414
 415     <p style="text-align: right;"><a href="sysadmin.rss"><img src="http://people.skolelinux.org/pere/blog/xml.gif" alt="RSS Feed" width="36" height="14" /></a></p>
 416     <div id="sidebar">
 417
 418
 419
 420 <h2>Archive</h2>
 421 <ul>
 422
 423 <li>2017
 424 <ul>
 425
 426 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 427
 428 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 429
 430 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (5)</a></li>
 431
 432 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
 433
 434 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
 435
 436 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
 437
 438 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
 439
 440 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
 441
 442 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
 443
 444 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (1)</a></li>
 445
 446 </ul></li>
 447
 448 <li>2016
 449 <ul>
 450
 451 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 452
 453 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 454
 455 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 456
 457 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 458
 459 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 460
 461 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 462
 463 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 464
 465 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 466
 467 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 468
 469 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 470
 471 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 472
 473 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 474
 475 </ul></li>
 476
 477 <li>2015
 478 <ul>
 479
 480 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 481
 482 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 483
 484 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 485
 486 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 487
 488 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 489
 490 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 491
 492 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 493
 494 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 495
 496 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 497
 498 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 499
 500 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 501
 502 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 503
 504 </ul></li>
 505
 506 <li>2014
 507 <ul>
 508
 509 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 510
 511 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 512
 513 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 514
 515 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 516
 517 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 518
 519 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 520
 521 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 522
 523 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 524
 525 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 526
 527 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 528
 529 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 530
 531 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 532
 533 </ul></li>
 534
 535 <li>2013
 536 <ul>
 537
 538 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 539
 540 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 541
 542 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 543
 544 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 545
 546 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 547
 548 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 549
 550 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 551
 552 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 553
 554 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 555
 556 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 557
 558 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 559
 560 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 561
 562 </ul></li>
 563
 564 <li>2012
 565 <ul>
 566
 567 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 568
 569 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 570
 571 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 572
 573 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 574
 575 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 576
 577 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 578
 579 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 580
 581 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 582
 583 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 584
 585 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 586
 587 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 588
 589 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 590
 591 </ul></li>
 592
 593 <li>2011
 594 <ul>
 595
 596 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 597
 598 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 599
 600 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 601
 602 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 603
 604 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 605
 606 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 607
 608 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 609
 610 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 611
 612 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 613
 614 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 615
 616 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 617
 618 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 619
 620 </ul></li>
 621
 622 <li>2010
 623 <ul>
 624
 625 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 626
 627 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 628
 629 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 630
 631 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 632
 633 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 634
 635 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 636
 637 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 638
 639 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 640
 641 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 642
 643 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 644
 645 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 646
 647 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 648
 649 </ul></li>
 650
 651 <li>2009
 652 <ul>
 653
 654 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 655
 656 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 657
 658 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 659
 660 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 661
 662 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 663
 664 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 665
 666 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 667
 668 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 669
 670 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 671
 672 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 673
 674 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 675
 676 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 677
 678 </ul></li>
 679
 680 <li>2008
 681 <ul>
 682
 683 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 684
 685 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 686
 687 </ul></li>
 688
 689 </ul>
 690
 691
 692
 693 <h2>Tags</h2>
 694 <ul>
 695
 696  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (14)</a></li>
 697
 698  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 699
 700  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 701
 702  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 703
 704  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
 705
 706  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
 707
 708  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 709
 710  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 711
 712  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (154)</a></li>
 713
 714  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
 715
 716  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (4)</a></li>
 717
 718  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 719
 720  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (17)</a></li>
 721
 722  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (24)</a></li>
 723
 724  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 725
 726  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (358)</a></li>
 727
 728  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 729
 730  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 731
 732  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (30)</a></li>
 733
 734  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 735
 736  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
 737
 738  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 739
 740  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 741
 742  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
 743
 744  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (20)</a></li>
 745
 746  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 747
 748  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 749
 750  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 751
 752  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 753
 754  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 755
 756  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
 757
 758  <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
 759
 760  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (293)</a></li>
 761
 762  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (189)</a></li>
 763
 764  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (33)</a></li>
 765
 766  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 767
 768  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (65)</a></li>
 769
 770  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (104)</a></li>
 771
 772  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (2)</a></li>
 773
 774  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 775
 776  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 777
 778  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 779
 780  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
 781
 782  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 783
 784  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
 785
 786  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 787
 788  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (53)</a></li>
 789
 790  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 791
 792  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 793
 794  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (55)</a></li>
 795
 796  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (6)</a></li>
 797
 798  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
 799
 800  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (52)</a></li>
 801
 802  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (4)</a></li>
 803
 804  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 805
 806  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
 807
 808  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
 809
 810  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 811
 812  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (40)</a></li>
 813
 814 </ul>
 815
 816
 817     </div>
 818     <p style="text-align: right">
 819  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 820 </p>
 821
 822   </body>
 823 </html>