blog/tags/sysadmin/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Entries Tagged sysadmin</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9     <link rel="alternate" title="RSS Feed" href="sysadmin.rss" type="application/rss+xml" />
  10   </head>
  11   <body>
  12     <div class="title">
  13  <h1>
  14      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  15
  16  </h1>
  17
  18 </div>
  19
  20
  21     <h3>Entries tagged "sysadmin".</h3>
  22
  23     <div class="entry">
  24       <div class="title">
  25         <a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a>
  26       </div>
  27       <div class="date">
  28          1st November 2017
  29       </div>
  30       <div class="body">
  31         <p>If you care about how fault tolerant your storage is, you might
  32 find these articles and papers interesting.  They have formed how I
  33 think of when designing a storage system.</p>
  34
  35 <ul>
  36
  37 <li>USENIX :login; <a
  38 href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
  39 Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
  40 Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
  41 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
  42 H. Arpaci-Dusseau</li>
  43
  44 <li>ZDNet
  45 <a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
  46 RAID 5 stops working in 2009</a> by Robin Harris</li>
  47
  48 <li>ZDNet
  49 <a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
  50 RAID 6 stops working in 2019</a> by Robin Harris</li>
  51
  52 <li>USENIX FAST'07
  53 <a href="http://research.google.com/archive/disk_failures.pdf">Failure
  54 Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
  55 Wolf-Dietrich Weber and Luiz André Barroso</li>
  56
  57 <li>USENIX ;login: <a
  58 href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
  59 Integrity.  Finding Truth in a World of Guesses and Lies</a> by Doug
  60 Hughes</li>
  61
  62 <li>USENIX FAST'08
  63 <a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
  64 Analysis of Data Corruption in the Storage Stack</a> by
  65 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
  66 Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
  67
  68 <li>USENIX FAST'07 <a
  69 href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
  70 failures in the real world: what does an MTTF of 1,000,000 hours mean
  71 to you?</a> by B. Schroeder and G. A. Gibson.</li>
  72
  73 <li>USENIX ;login: <a
  74 href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
  75 Disks the Dominant Contributor for Storage Failures?  A Comprehensive
  76 Study of Storage Subsystem Failure Characteristics</a> by Weihang
  77 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
  78
  79 <li>SIGMETRICS 2007
  80 <a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
  81 analysis of latent sector errors in disk drives</a> by
  82 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
  83
  84 </ul>
  85
  86 <p>Several of these research papers are based on data collected from
  87 hundred thousands or millions of disk, and their findings are eye
  88 opening.  The short story is simply do not implicitly trust RAID or
  89 redundant storage systems.  Details matter.  And unfortunately there
  90 are few options on Linux addressing all the identified issues.  Both
  91 ZFS and Btrfs are doing a fairly good job, but have legal and
  92 practical issues on their own.  I wonder how cluster file systems like
  93 Ceph do in this regard.  After all, there is an old saying, you know
  94 you have a distributed system when the crash of a computer you have
  95 never heard of stops you from getting any work done.  The same holds
  96 true if fault tolerance do not work.</p>
  97
  98 <p>Just remember, in the end, it do not matter how redundant, or how
  99 fault tolerant your storage is, if you do not continuously monitor its
 100 status to detect and replace failed disks.</p>
 101
 102 <p>As usual, if you use Bitcoin and want to show your support of my
 103 activities, please send Bitcoin donations to my address
 104 <b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
 105
 106       </div>
 107       <div class="tags">
 108
 109
 110         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 111
 112
 113       </div>
 114     </div>
 115     <div class="padding"></div>
 116
 117     <div class="entry">
 118       <div class="title">
 119         <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a>
 120       </div>
 121       <div class="date">
 122          9th March 2017
 123       </div>
 124       <div class="body">
 125         <p>Over the years, administrating thousand of NFS mounting linux
 126 computers at the time, I often needed a way to detect if the machine
 127 was experiencing NFS hang.  If you try to use <tt>df</tt> or look at a
 128 file or directory affected by the hang, the process (and possibly the
 129 shell) will hang too.  So you want to be able to detect this without
 130 risking the detection process getting stuck too.  It has not been
 131 obvious how to do this.  When the hang has lasted a while, it is
 132 possible to find messages like these in dmesg:</p>
 133
 134 <p><blockquote>
 135 nfs: server nfsserver not responding, still trying
 136 <br>nfs: server nfsserver OK
 137 </blockquote></p>
 138
 139 <p>It is hard to know if the hang is still going on, and it is hard to
 140 be sure looking in dmesg is going to work.  If there are lots of other
 141 messages in dmesg the lines might have rotated out of site before they
 142 are noticed.</p>
 143
 144 <p>While reading through the nfs client implementation in linux kernel
 145 code, I came across some statistics that seem to give a way to detect
 146 it.  The om_timeouts sunrpc value in the kernel will increase every
 147 time the above log entry is inserted into dmesg.  And after digging a
 148 bit further, I discovered that this value show up in
 149 /proc/self/mountstats on Linux.</p>
 150
 151 <p>The mountstats content seem to be shared between files using the
 152 same file system context, so it is enough to check one of the
 153 mountstats files to get the state of the mount point for the machine.
 154 I assume this will not show lazy umounted NFS points, nor NFS mount
 155 points in a different process context (ie with a different filesystem
 156 view), but that does not worry me.</p>
 157
 158 <p>The content for a NFS mount point look similar to this:</p>
 159
 160 <p><blockquote><pre>
 161 [...]
 162 device /dev/mapper/Debian-var mounted on /var with fstype ext3
 163 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
 164         opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
 165         age:    7863311
 166         caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
 167         sec:    flavor=1,pseudoflavor=1
 168         events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
 169         bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
 170         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
 171         xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
 172         per-op statistics
 173                 NULL: 0 0 0 0 0 0 0 0
 174              GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
 175              SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
 176               LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
 177               ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
 178             READLINK: 125 125 0 20472 18620 0 1112 1118
 179                 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
 180                WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
 181               CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
 182                MKDIR: 3680 3680 0 773980 993920 26 23990 24245
 183              SYMLINK: 903 903 0 233428 245488 6 5865 5917
 184                MKNOD: 80 80 0 20148 21760 0 299 304
 185               REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
 186                RMDIR: 3367 3367 0 645112 484848 22 5782 6002
 187               RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
 188                 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
 189              READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
 190          READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
 191               FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
 192               FSINFO: 2 2 0 232 328 0 1 1
 193             PATHCONF: 1 1 0 116 140 0 0 0
 194               COMMIT: 0 0 0 0 0 0 0 0
 195
 196 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
 197 [...]
 198 </pre></blockquote></p>
 199
 200 <p>The key number to look at is the third number in the per-op list.
 201 It is the number of NFS timeouts experiences per file system
 202 operation.  Here 22 write timeouts and 5 access timeouts.  If these
 203 numbers are increasing, I believe the machine is experiencing NFS
 204 hang.  Unfortunately the timeout value do not start to increase right
 205 away.  The NFS operations need to time out first, and this can take a
 206 while.  The exact timeout value depend on the setup.  For example the
 207 defaults for TCP and UDP mount points are quite different, and the
 208 timeout value is affected by the soft, hard, timeo and retrans NFS
 209 mount options.</p>
 210
 211 <p>The only way I have been able to get working on Debian and RedHat
 212 Enterprise Linux for getting the timeout count is to peek in /proc/.
 213 But according to
 214 <ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
 215 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
 216 command can be used to get these timeout values.  But this do not work
 217 on Linux, as far as I can tell.  I
 218 <ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
 219 but have not seen any replies yet.</p>
 220
 221 <p>Is there a better way to figure out if a Linux NFS client is
 222 experiencing NFS hangs?  Is there a way to detect which processes are
 223 affected?  Is there a way to get the NFS mount going quickly once the
 224 network problem causing the NFS hang has been cleared?  I would very
 225 much welcome some clues, as we regularly run into NFS hangs.</p>
 226
 227       </div>
 228       <div class="tags">
 229
 230
 231         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 232
 233
 234       </div>
 235     </div>
 236     <div class="padding"></div>
 237
 238     <div class="entry">
 239       <div class="title">
 240         <a href="http://people.skolelinux.org/pere/blog/Debian_Jessie__PXE_and_automatic_firmware_installation.html">Debian Jessie, PXE and automatic firmware installation</a>
 241       </div>
 242       <div class="date">
 243         17th October 2014
 244       </div>
 245       <div class="body">
 246         <p>When PXE installing laptops with Debian, I often run into the
 247 problem that the WiFi card require some firmware to work properly.
 248 And it has been a pain to fix this using preseeding in Debian.
 249 Normally something more is needed.  But thanks to
 250 <a href="https://packages.qa.debian.org/i/isenkram.html">my isenkram
 251 package</a> and its recent tasksel extension, it has now become easy
 252 to do this using simple preseeding.</p>
 253
 254 <p>The isenkram-cli package provide tasksel tasks which will install
 255 firmware for the hardware found in the machine (actually, requested by
 256 the kernel modules for the hardware).  (It can also install user space
 257 programs supporting the hardware detected, but that is not the focus
 258 of this story.)</p>
 259
 260 <p>To get this working in the default installation, two preeseding
 261 values are needed.  First, the isenkram-cli package must be installed
 262 into the target chroot (aka the hard drive) before tasksel is executed
 263 in the pkgsel step of the debian-installer system.  This is done by
 264 preseeding the base-installer/includes debconf value to include the
 265 isenkram-cli package.  The package name is next passed to debootstrap
 266 for installation.  With the isenkram-cli package in place, tasksel
 267 will automatically use the isenkram tasks to detect hardware specific
 268 packages for the machine being installed and install them, because
 269 isenkram-cli contain tasksel tasks.</p>
 270
 271 <p>Second, one need to enable the non-free APT repository, because
 272 most firmware unfortunately is non-free.  This is done by preseeding
 273 the apt-mirror-setup step.  This is unfortunate, but for a lot of
 274 hardware it is the only option in Debian.</p>
 275
 276 <p>The end result is two lines needed in your preseeding file to get
 277 firmware installed automatically by the installer:</p>
 278
 279 <p><blockquote><pre>
 280 base-installer base-installer/includes string isenkram-cli
 281 apt-mirror-setup apt-setup/non-free boolean true
 282 </pre></blockquote></p>
 283
 284 <p>The current version of isenkram-cli in testing/jessie will install
 285 both firmware and user space packages when using this method.  It also
 286 do not work well, so use version 0.15 or later.  Installing both
 287 firmware and user space packages might give you a bit more than you
 288 want, so I decided to split the tasksel task in two, one for firmware
 289 and one for user space programs.  The firmware task is enabled by
 290 default, while the one for user space programs is not.  This split is
 291 implemented in the package currently in unstable.</p>
 292
 293 <p>If you decide to give this a go, please let me know (via email) how
 294 this recipe work for you. :)</p>
 295
 296 <p>So, I bet you are wondering, how can this work.  First and
 297 foremost, it work because tasksel is modular, and driven by whatever
 298 files it find in /usr/lib/tasksel/ and /usr/share/tasksel/.  So the
 299 isenkram-cli package place two files for tasksel to find.  First there
 300 is the task description file (/usr/share/tasksel/descs/isenkram.desc):</p>
 301
 302 <p><blockquote><pre>
 303 Task: isenkram-packages
 304 Section: hardware
 305 Description: Hardware specific packages (autodetected by isenkram)
 306  Based on the detected hardware various hardware specific packages are
 307  proposed.
 308 Test-new-install: show show
 309 Relevance: 8
 310 Packages: for-current-hardware
 311
 312 Task: isenkram-firmware
 313 Section: hardware
 314 Description: Hardware specific firmware packages (autodetected by isenkram)
 315  Based on the detected hardware various hardware specific firmware
 316  packages are proposed.
 317 Test-new-install: mark show
 318 Relevance: 8
 319 Packages: for-current-hardware-firmware
 320 </pre></blockquote></p>
 321
 322 <p>The key parts are Test-new-install which indicate how the task
 323 should be handled and the Packages line referencing to a script in
 324 /usr/lib/tasksel/packages/.  The scripts use other scripts to get a
 325 list of packages to install.  The for-current-hardware-firmware script
 326 look like this to list relevant firmware for the machine:
 327
 328 <p><blockquote><pre>
 329 #!/bin/sh
 330 #
 331 PATH=/usr/sbin:$PATH
 332 export PATH
 333 isenkram-autoinstall-firmware -l
 334 </pre></blockquote></p>
 335
 336 <p>With those two pieces in place, the firmware is installed by
 337 tasksel during the normal d-i run. :)</p>
 338
 339 <p>If you want to test what tasksel will install when isenkram-cli is
 340 installed, run <tt>DEBIAN_PRIORITY=critical tasksel --test
 341 --new-install</tt> to get the list of packages that tasksel would
 342 install.</p>
 343
 344 <p><a href="https://wiki.debian.org/DebianEdu/">Debian Edu</a> will be
 345 pilots in testing this feature, as isenkram is used there now to
 346 install firmware, replacing the earlier scripts.</p>
 347
 348       </div>
 349       <div class="tags">
 350
 351
 352         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 353
 354
 355       </div>
 356     </div>
 357     <div class="padding"></div>
 358
 359     <div class="entry">
 360       <div class="title">
 361         <a href="http://people.skolelinux.org/pere/blog/Scripting_the_Cerebrum_bofhd_user_administration_system_using_XML_RPC.html">Scripting the Cerebrum/bofhd user administration system using XML-RPC</a>
 362       </div>
 363       <div class="date">
 364          6th December 2012
 365       </div>
 366       <div class="body">
 367         <p>Where I work at the <a href="http://www.uio.no/">University of
 368 Oslo</a>, we use the
 369 <a href="http://sourceforge.net/projects/cerebrum/">Cerebrum user
 370 administration system</a> to maintain users, groups, DNS, DHCP, etc.
 371 I've known since the system was written that the server is providing
 372 an <a href="http://en.wikipedia.org/wiki/XML-RPC">XML-RPC</a> API, but
 373 I have never spent time to try to figure out how to use it, as we
 374 always use the bofh command line client at work.  Until today.  I want
 375 to script the updating of DNS and DHCP to make it easier to set up
 376 virtual machines.  Here are a few notes on how to use it with
 377 Python.</p>
 378
 379 <p>I started by looking at the source of the Java
 380 <a href="http://cerebrum.svn.sourceforge.net/viewvc/cerebrum/trunk/cerebrum/clients/jbofh/">bofh
 381 client</a>, to figure out how it connected to the API server.  I also
 382 googled for python examples on how to use XML-RPC, and found
 383 <a href="http://tldp.org/HOWTO/XML-RPC-HOWTO/xmlrpc-howto-python.html">a
 384 simple example in</a> the XML-RPC howto.</p>
 385
 386 <p>This simple example code show how to connect, get the list of
 387 commands (as a JSON dump), and how to get the information about the
 388 user currently logged in:</p>
 389
 390 <blockquote><pre>
 391 #!/usr/bin/env python
 392 import getpass
 393 import xmlrpclib
 394 server_url = 'https://cerebrum-uio.uio.no:8000';
 395 username = getpass.getuser()
 396 password = getpass.getpass()
 397 server = xmlrpclib.Server(server_url);
 398 #print server.get_commands(sessionid)
 399 sessionid = server.login(username, password)
 400 print server.run_command(sessionid, "user_info", username)
 401 result = server.logout(sessionid)
 402 print result
 403 </pre></blockquote>
 404
 405 <p>Armed with this knowledge I can now move forward and script the DNS
 406 and DHCP updates I wanted to do.</p>
 407
 408       </div>
 409       <div class="tags">
 410
 411
 412         Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
 413
 414
 415       </div>
 416     </div>
 417     <div class="padding"></div>
 418
 419     <p style="text-align: right;"><a href="sysadmin.rss"><img src="http://people.skolelinux.org/pere/blog/xml.gif" alt="RSS Feed" width="36" height="14" /></a></p>
 420     <div id="sidebar">
 421
 422
 423
 424 <h2>Archive</h2>
 425 <ul>
 426
 427 <li>2019
 428 <ul>
 429
 430 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/01/">January (4)</a></li>
 431
 432 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/02/">February (3)</a></li>
 433
 434 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/03/">March (3)</a></li>
 435
 436 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/05/">May (2)</a></li>
 437
 438 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/06/">June (5)</a></li>
 439
 440 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/07/">July (2)</a></li>
 441
 442 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/08/">August (1)</a></li>
 443
 444 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/09/">September (1)</a></li>
 445
 446 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/11/">November (1)</a></li>
 447
 448 <li><a href="http://people.skolelinux.org/pere/blog/archive/2019/12/">December (3)</a></li>
 449
 450 </ul></li>
 451
 452 <li>2018
 453 <ul>
 454
 455 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/01/">January (1)</a></li>
 456
 457 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/02/">February (5)</a></li>
 458
 459 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/03/">March (5)</a></li>
 460
 461 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/04/">April (3)</a></li>
 462
 463 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/06/">June (2)</a></li>
 464
 465 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/07/">July (5)</a></li>
 466
 467 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/08/">August (3)</a></li>
 468
 469 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/09/">September (3)</a></li>
 470
 471 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/10/">October (5)</a></li>
 472
 473 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/11/">November (2)</a></li>
 474
 475 <li><a href="http://people.skolelinux.org/pere/blog/archive/2018/12/">December (4)</a></li>
 476
 477 </ul></li>
 478
 479 <li>2017
 480 <ul>
 481
 482 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
 483
 484 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
 485
 486 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (5)</a></li>
 487
 488 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
 489
 490 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
 491
 492 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
 493
 494 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
 495
 496 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
 497
 498 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
 499
 500 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (3)</a></li>
 501
 502 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/12/">December (4)</a></li>
 503
 504 </ul></li>
 505
 506 <li>2016
 507 <ul>
 508
 509 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
 510
 511 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
 512
 513 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
 514
 515 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
 516
 517 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
 518
 519 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
 520
 521 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
 522
 523 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
 524
 525 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
 526
 527 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
 528
 529 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
 530
 531 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
 532
 533 </ul></li>
 534
 535 <li>2015
 536 <ul>
 537
 538 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
 539
 540 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
 541
 542 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
 543
 544 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
 545
 546 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
 547
 548 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
 549
 550 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
 551
 552 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
 553
 554 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
 555
 556 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
 557
 558 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
 559
 560 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
 561
 562 </ul></li>
 563
 564 <li>2014
 565 <ul>
 566
 567 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 568
 569 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 570
 571 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 572
 573 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 574
 575 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 576
 577 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
 578
 579 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
 580
 581 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
 582
 583 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
 584
 585 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
 586
 587 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
 588
 589 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
 590
 591 </ul></li>
 592
 593 <li>2013
 594 <ul>
 595
 596 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 597
 598 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 599
 600 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 601
 602 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 603
 604 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 605
 606 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 607
 608 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 609
 610 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 611
 612 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 613
 614 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 615
 616 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 617
 618 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 619
 620 </ul></li>
 621
 622 <li>2012
 623 <ul>
 624
 625 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 626
 627 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 628
 629 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 630
 631 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 632
 633 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 634
 635 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 636
 637 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 638
 639 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 640
 641 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 642
 643 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 644
 645 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 646
 647 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 648
 649 </ul></li>
 650
 651 <li>2011
 652 <ul>
 653
 654 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 655
 656 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 657
 658 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 659
 660 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 661
 662 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 663
 664 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 665
 666 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 667
 668 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 669
 670 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 671
 672 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 673
 674 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 675
 676 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 677
 678 </ul></li>
 679
 680 <li>2010
 681 <ul>
 682
 683 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 684
 685 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 686
 687 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 688
 689 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 690
 691 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 692
 693 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 694
 695 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 696
 697 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 698
 699 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 700
 701 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 702
 703 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 704
 705 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 706
 707 </ul></li>
 708
 709 <li>2009
 710 <ul>
 711
 712 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 713
 714 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 715
 716 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 717
 718 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 719
 720 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 721
 722 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 723
 724 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 725
 726 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 727
 728 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 729
 730 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 731
 732 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 733
 734 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 735
 736 </ul></li>
 737
 738 <li>2008
 739 <ul>
 740
 741 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 742
 743 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 744
 745 </ul></li>
 746
 747 </ul>
 748
 749
 750
 751 <h2>Tags</h2>
 752 <ul>
 753
 754  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (16)</a></li>
 755
 756  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 757
 758  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 759
 760  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 761
 762  <li><a href="http://people.skolelinux.org/pere/blog/tags/betalkontant">betalkontant (8)</a></li>
 763
 764  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (11)</a></li>
 765
 766  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (17)</a></li>
 767
 768  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 769
 770  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 771
 772  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (168)</a></li>
 773
 774  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
 775
 776  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (4)</a></li>
 777
 778  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (11)</a></li>
 779
 780  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (17)</a></li>
 781
 782  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (26)</a></li>
 783
 784  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 785
 786  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (411)</a></li>
 787
 788  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 789
 790  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (14)</a></li>
 791
 792  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (34)</a></li>
 793
 794  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
 795
 796  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (20)</a></li>
 797
 798  <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
 799
 800  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
 801
 802  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (16)</a></li>
 803
 804  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (22)</a></li>
 805
 806  <li><a href="http://people.skolelinux.org/pere/blog/tags/kodi">kodi (4)</a></li>
 807
 808  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 809
 810  <li><a href="http://people.skolelinux.org/pere/blog/tags/lego">lego (4)</a></li>
 811
 812  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
 813
 814  <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
 815
 816  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 817
 818  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 819
 820  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (42)</a></li>
 821
 822  <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (12)</a></li>
 823
 824  <li><a href="http://people.skolelinux.org/pere/blog/tags/noark5">noark5 (18)</a></li>
 825
 826  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (309)</a></li>
 827
 828  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (196)</a></li>
 829
 830  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (37)</a></li>
 831
 832  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 833
 834  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (75)</a></li>
 835
 836  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (111)</a></li>
 837
 838  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (2)</a></li>
 839
 840  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 841
 842  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 843
 844  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
 845
 846  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (12)</a></li>
 847
 848  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 849
 850  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (7)</a></li>
 851
 852  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 853
 854  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (56)</a></li>
 855
 856  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 857
 858  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
 859
 860  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (67)</a></li>
 861
 862  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (6)</a></li>
 863
 864  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
 865
 866  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (57)</a></li>
 867
 868  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (4)</a></li>
 869
 870  <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
 871
 872  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
 873
 874  <li><a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri (17)</a></li>
 875
 876  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (73)</a></li>
 877
 878  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 879
 880  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (42)</a></li>
 881
 882 </ul>
 883
 884
 885     </div>
 886     <p style="text-align: right">
 887  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 888 </p>
 889
 890   </body>
 891 </html>