+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
+ <title>Petter Reinholdtsen: Detecting NFS hangs on Linux without hanging yourself...</title>
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
+
+
+ </head>
+ <body>
+ <div class="title">
+ <h1>
+ <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
+
+ </h1>
+
+</div>
+
+
+ <div class="entry">
+ <div class="title">Detecting NFS hangs on Linux without hanging yourself...</div>
+ <div class="date"> 9th March 2017</div>
+ <div class="body"><p>Over the years, administrating thousand of NFS mounting linux
+computers at the time, I often needed a way to detect if the machine
+was experiencing NFS hang. If you try to use <tt>df</tt> or look at a
+file or directory affected by the hang, the process (and possibly the
+shell) will hang too. So you want to be able to detect this without
+risking the detection process getting stuck too. It has not been
+obvious how to do this. When the hang has lasted a while, it is
+possible to find messages like these in dmesg:</p>
+
+<p><blockquote>
+nfs: server nfsserver not responding, still trying
+<br>nfs: server nfsserver OK
+</blockquote></p>
+
+<p>It is hard to know if the hang is still going on, and it is hard to
+be sure looking in dmesg is going to work. If there are lots of other
+messages in dmesg the lines might have rotated out of site before they
+are noticed.</p>
+
+<p>While reading through the nfs client implementation in linux kernel
+code, I came across some statistics that seem to give a way to detect
+it. The om_timeouts sunrpc value in the kernel will increase every
+time the above log entry is inserted into dmesg. And after digging a
+bit further, I discovered that this value show up in
+/proc/self/mountstats on Linux.</p>
+
+<p>The mountstats content seem to be shared between files using the
+same file system context, so it is enough to check one of the
+mountstats files to get the state of the mount point for the machine.
+I assume this will not show lazy umounted NFS points, nor NFS mount
+points in a different process context (ie with a different filesystem
+view), but that does not worry me.</p>
+
+<p>The content for a NFS mount point look similar to this:</p>
+
+<p><blockquote><pre>
+[...]
+device /dev/mapper/Debian-var mounted on /var with fstype ext3
+device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
+ opts: rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
+ age: 7863311
+ caps: caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
+ sec: flavor=1,pseudoflavor=1
+ events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
+ bytes: 166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
+ RPC iostats version: 1.0 p/v: 100003/3 (nfs)
+ xprt: tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
+ per-op statistics
+ NULL: 0 0 0 0 0 0 0 0
+ GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
+ SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
+ LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
+ ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
+ READLINK: 125 125 0 20472 18620 0 1112 1118
+ READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
+ WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
+ CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
+ MKDIR: 3680 3680 0 773980 993920 26 23990 24245
+ SYMLINK: 903 903 0 233428 245488 6 5865 5917
+ MKNOD: 80 80 0 20148 21760 0 299 304
+ REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
+ RMDIR: 3367 3367 0 645112 484848 22 5782 6002
+ RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
+ LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
+ READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
+ READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
+ FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
+ FSINFO: 2 2 0 232 328 0 1 1
+ PATHCONF: 1 1 0 116 140 0 0 0
+ COMMIT: 0 0 0 0 0 0 0 0
+
+device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
+[...]
+</pre></blockquote></p>
+
+<p>The key number to look at is the third number in the per-op list.
+It is the number of NFS timeouts experiences per file system
+operation. Here 22 write timeouts and 5 access timeouts. If these
+numbers are increasing, I believe the machine is experiencing NFS
+hang. Unfortunately the timeout value do not start to increase right
+away. The NFS operations need to time out first, and this can take a
+while. The exact timeout value depend on the setup. For example the
+defaults for TCP and UDP mount points are quite different, and the
+timeout value is affected by the soft, hard, timeo and retrans NFS
+mount options.</p>
+
+<p>The only way I have been able to get working on Debian and RedHat
+Enterprise Linux for getting the timeout count is to peek in /proc/.
+But according to
+<ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
+10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
+command can be used to get these timeout values. But this do not work
+on Linux, as far as I can tell. I
+<ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
+but have not seen any replies yet.</p>
+
+<p>Is there a better way to figure out if a Linux NFS client is
+experiencing NFS hangs? Is there a way to detect which processes are
+affected? Is there a way to get the NFS mount going quickly once the
+network problem causing the NFS hang has been cleared? I would very
+much welcome some clues, as we regularly run into NFS hangs.</p>
+</div>
+
+ <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.</div>
+
+
+ </div>
+
+
+
+
+ <div id="sidebar">
+
+
+
+<h2>Archive</h2>
+<ul>
+
+<li>2017
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/02/">February (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/03/">March (4)</a></li>
+
+</ul></li>
+
+<li>2016
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
+
+</ul></li>
+
+<li>2015
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2014
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
+
+</ul></li>
+
+<li>2013
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2012
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
+
+</ul></li>
+
+<li>2011
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
+
+</ul></li>
+
+<li>2010
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
+
+</ul></li>
+
+<li>2009
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2008
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
+
+</ul></li>
+
+</ul>
+
+
+
+<h2>Tags</h2>
+<ul>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (148)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (3)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (16)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (23)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (345)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (29)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (20)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (287)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (187)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (28)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (64)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (99)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (52)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (51)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (5)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (48)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (3)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (40)</a></li>
+
+</ul>
+
+
+ </div>
+ <p style="text-align: right">
+ Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
+</p>
+
+ </body>
+</html>
projects / homepage.git / commitdiff