1 <!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Strict//EN"
2 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3 <html xmlns=
"http://www.w3.org/1999/xhtml" dir=
"ltr">
5 <meta http-equiv=
"Content-Type" content=
"text/html;charset=utf-8" />
6 <title>Petter Reinholdtsen: Entries Tagged sysadmin
</title>
7 <link rel=
"stylesheet" type=
"text/css" media=
"screen" href=
"http://people.skolelinux.org/pere/blog/style.css" />
8 <link rel=
"stylesheet" type=
"text/css" media=
"screen" href=
"http://people.skolelinux.org/pere/blog/vim.css" />
9 <link rel=
"alternate" title=
"RSS Feed" href=
"sysadmin.rss" type=
"application/rss+xml" />
14 <a href=
"http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen
</a>
21 <h3>Entries tagged "sysadmin".
</h3>
25 <a href=
"http://people.skolelinux.org/pere/blog/Secure_Socket_API___a_simple_and_powerful_approach_for_TLS_support_in_software.html">Secure Socket API - a simple and powerful approach for TLS support in software
</a>
31 <p>As a member of the
<a href=
"https://www.nuug.no/">Norwegian Unix
32 User Group
</a>, I have the pleasure of receiving the
33 <a href=
"https://www.usenix.org/">USENIX
</a> magazine
34 <a href=
"https://www.usenix.org/publications/login/">;login:
</a>
35 several times a year. I rarely have time to read all the articles,
36 but try to at least skim through them all as there is a lot of nice
37 knowledge passed on there. I even carry the latest issue with me most
38 of the time to try to get through all the articles when I have a few
41 <p>The other day I came across a nice article titled
42 "
<a href=
"https://www.usenix.org/publications/login/winter2018/oneill">The
43 Secure Socket API: TLS as an Operating System Service
</a>" with a
44 marvellous idea I hope can make it all the way into the POSIX standard.
45 The idea is as simple as it is powerful. By introducing a new
46 socket() option IPPROTO_TLS to use TLS, and a system wide service to
47 handle setting up TLS connections, one both make it trivial to add TLS
48 support to any program currently using the POSIX socket API, and gain
49 system wide control over certificates, TLS versions and encryption
50 systems used. Instead of doing this:</p>
53 int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
54 </pre></blockquote></p>
56 <p>the program code would be doing this:<p>
59 int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TLS);
60 </pre></blockquote></p>
62 <p>According to the ;login: article, converting a C program to use TLS
63 would normally modify only 5-10 lines in the code, which is amazing
64 when compared to using for example the OpenSSL API.</p>
66 <p>The project has set up the
67 <a href="https://securesocketapi.org/
">https://securesocketapi.org/</a>
68 web site to spread the idea, and the code for a kernel module and the
69 associated system daemon is available from two github repositories:
70 <a href="https://github.com/markoneill/ssa
">ssa</a> and
71 <a href="https://github.com/markoneill/ssa-daemon
">ssa-daemon</a>.
72 Unfortunately there is no explicit license information with the code,
73 so its copyright status is unclear. A
74 <a href="https://github.com/markoneill/ssa/issues/
2">request to solve
75 this</a> about it has been unsolved since 2018-08-17.</p>
77 <p>I love the idea of extending socket() to gain TLS support, and
78 understand why it is an advantage to implement this as a kernel module
79 and system wide service daemon, but can not help to think that it
80 would be a lot easier to get projects to move to this way of setting
81 up TLS if it was done with a user space approach where programs
82 wanting to use this API approach could just link with a wrapper
85 <p>I recommend you check out this simple and powerful approach to more
86 secure network connections. :)</p>
88 <p>As usual, if you use Bitcoin and want to show your support of my
89 activities, please send Bitcoin donations to my address
90 <b><a href="bitcoin:
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b
">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
96 Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian
">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english
">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet
">sikkerhet</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin
">sysadmin</a>.
101 <div class="padding
"></div>
105 <a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html
">Some notes on fault tolerant storage systems</a>
111 <p>If you care about how fault tolerant your storage is, you might
112 find these articles and papers interesting. They have formed how I
113 think of when designing a storage system.</p>
117 <li>USENIX :login; <a
118 href="https://www.usenix.org/publications/login/summer2017/ganesan
">Redundancy
119 Does Not Imply Fault Tolerance. Analysis of Distributed Storage
120 Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
121 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
122 H. Arpaci-Dusseau</li>
125 <a href="http://www.zdnet.com/article/why-raid-
5-stops-working-in-
2009/
">Why
126 RAID 5 stops working in 2009</a> by Robin Harris</li>
129 <a href="http://www.zdnet.com/article/why-raid-
6-stops-working-in-
2019/
">Why
130 RAID 6 stops working in 2019</a> by Robin Harris</li>
133 <a href="http://research.google.com/archive/disk_failures.pdf
">Failure
134 Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
135 Wolf-Dietrich Weber and Luiz AndreĢ Barroso</li>
137 <li>USENIX ;login: <a
138 href="https://www.usenix.org/system/files/login/articles/hughes12-
04.pdf
">Data
139 Integrity. Finding Truth in a World of Guesses and Lies</a> by Doug
143 <a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/
">An
144 Analysis of Data Corruption in the Storage Stack</a> by
145 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
146 Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
148 <li>USENIX FAST'07 <a
149 href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/
">Disk
150 failures in the real world: what does an MTTF of 1,000,000 hours mean
151 to you?</a> by B. Schroeder and G. A. Gibson.</li>
153 <li>USENIX ;login: <a
154 href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/
">Are
155 Disks the Dominant Contributor for Storage Failures? A Comprehensive
156 Study of Storage Subsystem Failure Characteristics</a> by Weihang
157 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
160 <a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf
">An
161 analysis of latent sector errors in disk drives</a> by
162 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
166 <p>Several of these research papers are based on data collected from
167 hundred thousands or millions of disk, and their findings are eye
168 opening. The short story is simply do not implicitly trust RAID or
169 redundant storage systems. Details matter. And unfortunately there
170 are few options on Linux addressing all the identified issues. Both
171 ZFS and Btrfs are doing a fairly good job, but have legal and
172 practical issues on their own. I wonder how cluster file systems like
173 Ceph do in this regard. After all, there is an old saying, you know
174 you have a distributed system when the crash of a computer you have
175 never heard of stops you from getting any work done. The same holds
176 true if fault tolerance do not work.</p>
178 <p>Just remember, in the end, it do not matter how redundant, or how
179 fault tolerant your storage is, if you do not continuously monitor its
180 status to detect and replace failed disks.</p>
182 <p>As usual, if you use Bitcoin and want to show your support of my
183 activities, please send Bitcoin donations to my address
184 <b><a href="bitcoin:
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b
">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
190 Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english
">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid
">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin
">sysadmin</a>.
195 <div class="padding
"></div>
199 <a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html
">Detecting NFS hangs on Linux without hanging yourself...</a>
205 <p>Over the years, administrating thousand of NFS mounting linux
206 computers at the time, I often needed a way to detect if the machine
207 was experiencing NFS hang. If you try to use <tt>df</tt> or look at a
208 file or directory affected by the hang, the process (and possibly the
209 shell) will hang too. So you want to be able to detect this without
210 risking the detection process getting stuck too. It has not been
211 obvious how to do this. When the hang has lasted a while, it is
212 possible to find messages like these in dmesg:</p>
215 nfs: server nfsserver not responding, still trying
216 <br>nfs: server nfsserver OK
219 <p>It is hard to know if the hang is still going on, and it is hard to
220 be sure looking in dmesg is going to work. If there are lots of other
221 messages in dmesg the lines might have rotated out of site before they
224 <p>While reading through the nfs client implementation in linux kernel
225 code, I came across some statistics that seem to give a way to detect
226 it. The om_timeouts sunrpc value in the kernel will increase every
227 time the above log entry is inserted into dmesg. And after digging a
228 bit further, I discovered that this value show up in
229 /proc/self/mountstats on Linux.</p>
231 <p>The mountstats content seem to be shared between files using the
232 same file system context, so it is enough to check one of the
233 mountstats files to get the state of the mount point for the machine.
234 I assume this will not show lazy umounted NFS points, nor NFS mount
235 points in a different process context (ie with a different filesystem
236 view), but that does not worry me.</p>
238 <p>The content for a NFS mount point look similar to this:</p>
242 device /dev/mapper/Debian-var mounted on /var with fstype ext3
243 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
244 opts: rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
246 caps: caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
247 sec: flavor=1,pseudoflavor=1
248 events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
249 bytes: 166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
250 RPC iostats version: 1.0 p/v: 100003/3 (nfs)
251 xprt: tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
253 NULL: 0 0 0 0 0 0 0 0
254 GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
255 SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
256 LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
257 ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
258 READLINK: 125 125 0 20472 18620 0 1112 1118
259 READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
260 WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
261 CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
262 MKDIR: 3680 3680 0 773980 993920 26 23990 24245
263 SYMLINK: 903 903 0 233428 245488 6 5865 5917
264 MKNOD: 80 80 0 20148 21760 0 299 304
265 REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
266 RMDIR: 3367 3367 0 645112 484848 22 5782 6002
267 RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
268 LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
269 READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
270 READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
271 FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
272 FSINFO: 2 2 0 232 328 0 1 1
273 PATHCONF: 1 1 0 116 140 0 0 0
274 COMMIT: 0 0 0 0 0 0 0 0
276 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
278 </pre></blockquote></p>
280 <p>The key number to look at is the third number in the per-op list.
281 It is the number of NFS timeouts experiences per file system
282 operation. Here 22 write timeouts and 5 access timeouts. If these
283 numbers are increasing, I believe the machine is experiencing NFS
284 hang. Unfortunately the timeout value do not start to increase right
285 away. The NFS operations need to time out first, and this can take a
286 while. The exact timeout value depend on the setup. For example the
287 defaults for TCP and UDP mount points are quite different, and the
288 timeout value is affected by the soft, hard, timeo and retrans NFS
291 <p>The only way I have been able to get working on Debian and RedHat
292 Enterprise Linux for getting the timeout count is to peek in /proc/.
294 <ahref="http://docs.oracle.com/cd/E19253-
01/
816-
4555/netmonitor-
12/index.html
">Solaris
295 10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
296 command can be used to get these timeout values. But this do not work
297 on Linux, as far as I can tell. I
298 <ahref="http://bugs.debian.org/
857043">asked Debian about this</a>,
299 but have not seen any replies yet.</p>
301 <p>Is there a better way to figure out if a Linux NFS client is
302 experiencing NFS hangs? Is there a way to detect which processes are
303 affected? Is there a way to get the NFS mount going quickly once the
304 network problem causing the NFS hang has been cleared? I would very
305 much welcome some clues, as we regularly run into NFS hangs.</p>
311 Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian
">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english
">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin
">sysadmin</a>.
316 <div class="padding
"></div>
320 <a href="http://people.skolelinux.org/pere/blog/Debian_Jessie__PXE_and_automatic_firmware_installation.html
">Debian Jessie, PXE and automatic firmware installation</a>
326 <p>When PXE installing laptops with Debian, I often run into the
327 problem that the WiFi card require some firmware to work properly.
328 And it has been a pain to fix this using preseeding in Debian.
329 Normally something more is needed. But thanks to
330 <a href="https://packages.qa.debian.org/i/isenkram.html
">my isenkram
331 package</a> and its recent tasksel extension, it has now become easy
332 to do this using simple preseeding.</p>
334 <p>The isenkram-cli package provide tasksel tasks which will install
335 firmware for the hardware found in the machine (actually, requested by
336 the kernel modules for the hardware). (It can also install user space
337 programs supporting the hardware detected, but that is not the focus
340 <p>To get this working in the default installation, two preeseding
341 values are needed. First, the isenkram-cli package must be installed
342 into the target chroot (aka the hard drive) before tasksel is executed
343 in the pkgsel step of the debian-installer system. This is done by
344 preseeding the base-installer/includes debconf value to include the
345 isenkram-cli package. The package name is next passed to debootstrap
346 for installation. With the isenkram-cli package in place, tasksel
347 will automatically use the isenkram tasks to detect hardware specific
348 packages for the machine being installed and install them, because
349 isenkram-cli contain tasksel tasks.</p>
351 <p>Second, one need to enable the non-free APT repository, because
352 most firmware unfortunately is non-free. This is done by preseeding
353 the apt-mirror-setup step. This is unfortunate, but for a lot of
354 hardware it is the only option in Debian.</p>
356 <p>The end result is two lines needed in your preseeding file to get
357 firmware installed automatically by the installer:</p>
360 base-installer base-installer/includes string isenkram-cli
361 apt-mirror-setup apt-setup/non-free boolean true
362 </pre></blockquote></p>
364 <p>The current version of isenkram-cli in testing/jessie will install
365 both firmware and user space packages when using this method. It also
366 do not work well, so use version 0.15 or later. Installing both
367 firmware and user space packages might give you a bit more than you
368 want, so I decided to split the tasksel task in two, one for firmware
369 and one for user space programs. The firmware task is enabled by
370 default, while the one for user space programs is not. This split is
371 implemented in the package currently in unstable.</p>
373 <p>If you decide to give this a go, please let me know (via email) how
374 this recipe work for you. :)</p>
376 <p>So, I bet you are wondering, how can this work. First and
377 foremost, it work because tasksel is modular, and driven by whatever
378 files it find in /usr/lib/tasksel/ and /usr/share/tasksel/. So the
379 isenkram-cli package place two files for tasksel to find. First there
380 is the task description file (/usr/share/tasksel/descs/isenkram.desc):</p>
383 Task: isenkram-packages
385 Description: Hardware specific packages (autodetected by isenkram)
386 Based on the detected hardware various hardware specific packages are
388 Test-new-install: show show
390 Packages: for-current-hardware
392 Task: isenkram-firmware
394 Description: Hardware specific firmware packages (autodetected by isenkram)
395 Based on the detected hardware various hardware specific firmware
396 packages are proposed.
397 Test-new-install: mark show
399 Packages: for-current-hardware-firmware
400 </pre></blockquote></p>
402 <p>The key parts are Test-new-install which indicate how the task
403 should be handled and the Packages line referencing to a script in
404 /usr/lib/tasksel/packages/. The scripts use other scripts to get a
405 list of packages to install. The for-current-hardware-firmware script
406 look like this to list relevant firmware for the machine:
413 isenkram-autoinstall-firmware -l
414 </pre></blockquote></p>
416 <p>With those two pieces in place, the firmware is installed by
417 tasksel during the normal d-i run. :)</p>
419 <p>If you want to test what tasksel will install when isenkram-cli is
420 installed, run <tt>DEBIAN_PRIORITY=critical tasksel --test
421 --new-install</tt> to get the list of packages that tasksel would
424 <p><a href="https://wiki.debian.org/DebianEdu/
">Debian Edu</a> will be
425 pilots in testing this feature, as isenkram is used there now to
426 install firmware, replacing the earlier scripts.</p>
432 Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian
">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english
">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/isenkram
">isenkram</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin
">sysadmin</a>.
437 <div class="padding
"></div>
441 <a href="http://people.skolelinux.org/pere/blog/Scripting_the_Cerebrum_bofhd_user_administration_system_using_XML_RPC.html
">Scripting the Cerebrum/bofhd user administration system using XML-RPC</a>
447 <p>Where I work at the <a href="http://www.uio.no/
">University of
449 <a href="http://sourceforge.net/projects/cerebrum/
">Cerebrum user
450 administration system</a> to maintain users, groups, DNS, DHCP, etc.
451 I've known since the system was written that the server is providing
452 an <a href="http://en.wikipedia.org/wiki/XML-RPC
">XML-RPC</a> API, but
453 I have never spent time to try to figure out how to use it, as we
454 always use the bofh command line client at work. Until today. I want
455 to script the updating of DNS and DHCP to make it easier to set up
456 virtual machines. Here are a few notes on how to use it with
459 <p>I started by looking at the source of the Java
460 <a href="http://cerebrum.svn.sourceforge.net/viewvc/cerebrum/trunk/cerebrum/clients/jbofh/
">bofh
461 client</a>, to figure out how it connected to the API server. I also
462 googled for python examples on how to use XML-RPC, and found
463 <a href="http://tldp.org/HOWTO/XML-RPC-HOWTO/xmlrpc-howto-python.html
">a
464 simple example in</a> the XML-RPC howto.</p>
466 <p>This simple example code show how to connect, get the list of
467 commands (as a JSON dump), and how to get the information about the
468 user currently logged in:</p>
471 #!/usr/bin/env python
474 server_url = 'https://cerebrum-uio.uio.no:8000';
475 username = getpass.getuser()
476 password = getpass.getpass()
477 server = xmlrpclib.Server(server_url);
478 #print server.get_commands(sessionid)
479 sessionid = server.login(username, password)
480 print server.run_command(sessionid, "user_info", username)
481 result = server.logout(sessionid)
485 <p>Armed with this knowledge I can now move forward and script the DNS
486 and DHCP updates I wanted to do.
</p>
492 Tags:
<a href=
"http://people.skolelinux.org/pere/blog/tags/english">english
</a>,
<a href=
"http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin
</a>.
497 <div class=
"padding"></div>
499 <p style=
"text-align: right;"><a href=
"sysadmin.rss"><img src=
"http://people.skolelinux.org/pere/blog/xml.gif" alt=
"RSS Feed" width=
"36" height=
"14" /></a></p>
510 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2021/01/">January (
2)
</a></li>
512 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2021/02/">February (
1)
</a></li>
514 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2021/05/">May (
1)
</a></li>
516 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2021/06/">June (
1)
</a></li>
518 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2021/07/">July (
2)
</a></li>
525 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/02/">February (
2)
</a></li>
527 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/03/">March (
2)
</a></li>
529 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/04/">April (
2)
</a></li>
531 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/05/">May (
3)
</a></li>
533 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/06/">June (
2)
</a></li>
535 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/07/">July (
1)
</a></li>
537 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/09/">September (
1)
</a></li>
539 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/10/">October (
1)
</a></li>
541 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2020/11/">November (
1)
</a></li>
548 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/01/">January (
4)
</a></li>
550 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/02/">February (
3)
</a></li>
552 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/03/">March (
3)
</a></li>
554 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/05/">May (
2)
</a></li>
556 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/06/">June (
5)
</a></li>
558 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/07/">July (
2)
</a></li>
560 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/08/">August (
1)
</a></li>
562 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/09/">September (
1)
</a></li>
564 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/11/">November (
1)
</a></li>
566 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2019/12/">December (
4)
</a></li>
573 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/01/">January (
1)
</a></li>
575 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/02/">February (
5)
</a></li>
577 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/03/">March (
5)
</a></li>
579 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/04/">April (
3)
</a></li>
581 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/06/">June (
2)
</a></li>
583 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/07/">July (
5)
</a></li>
585 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/08/">August (
3)
</a></li>
587 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/09/">September (
3)
</a></li>
589 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/10/">October (
5)
</a></li>
591 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/11/">November (
2)
</a></li>
593 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2018/12/">December (
4)
</a></li>
600 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/01/">January (
4)
</a></li>
602 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/02/">February (
3)
</a></li>
604 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/03/">March (
5)
</a></li>
606 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/04/">April (
2)
</a></li>
608 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/06/">June (
5)
</a></li>
610 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/07/">July (
1)
</a></li>
612 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/08/">August (
1)
</a></li>
614 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/09/">September (
3)
</a></li>
616 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/10/">October (
5)
</a></li>
618 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/11/">November (
3)
</a></li>
620 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2017/12/">December (
4)
</a></li>
627 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/01/">January (
3)
</a></li>
629 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/02/">February (
2)
</a></li>
631 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/03/">March (
3)
</a></li>
633 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/04/">April (
8)
</a></li>
635 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/05/">May (
8)
</a></li>
637 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/06/">June (
2)
</a></li>
639 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/07/">July (
2)
</a></li>
641 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/08/">August (
5)
</a></li>
643 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/09/">September (
2)
</a></li>
645 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/10/">October (
3)
</a></li>
647 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/11/">November (
8)
</a></li>
649 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2016/12/">December (
5)
</a></li>
656 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/01/">January (
7)
</a></li>
658 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/02/">February (
6)
</a></li>
660 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/03/">March (
1)
</a></li>
662 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/04/">April (
4)
</a></li>
664 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/05/">May (
3)
</a></li>
666 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/06/">June (
4)
</a></li>
668 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/07/">July (
6)
</a></li>
670 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/08/">August (
2)
</a></li>
672 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/09/">September (
2)
</a></li>
674 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/10/">October (
9)
</a></li>
676 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/11/">November (
6)
</a></li>
678 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2015/12/">December (
3)
</a></li>
685 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/01/">January (
2)
</a></li>
687 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/02/">February (
3)
</a></li>
689 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/03/">March (
8)
</a></li>
691 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/04/">April (
7)
</a></li>
693 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/05/">May (
1)
</a></li>
695 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/06/">June (
2)
</a></li>
697 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/07/">July (
2)
</a></li>
699 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/08/">August (
2)
</a></li>
701 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/09/">September (
5)
</a></li>
703 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/10/">October (
6)
</a></li>
705 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/11/">November (
3)
</a></li>
707 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2014/12/">December (
5)
</a></li>
714 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/01/">January (
11)
</a></li>
716 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/02/">February (
9)
</a></li>
718 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/03/">March (
9)
</a></li>
720 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/04/">April (
6)
</a></li>
722 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/05/">May (
9)
</a></li>
724 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/06/">June (
10)
</a></li>
726 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/07/">July (
7)
</a></li>
728 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/08/">August (
3)
</a></li>
730 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/09/">September (
5)
</a></li>
732 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/10/">October (
7)
</a></li>
734 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/11/">November (
9)
</a></li>
736 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2013/12/">December (
3)
</a></li>
743 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/01/">January (
7)
</a></li>
745 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/02/">February (
10)
</a></li>
747 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/03/">March (
17)
</a></li>
749 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/04/">April (
12)
</a></li>
751 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/05/">May (
12)
</a></li>
753 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/06/">June (
20)
</a></li>
755 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/07/">July (
17)
</a></li>
757 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/08/">August (
6)
</a></li>
759 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/09/">September (
9)
</a></li>
761 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/10/">October (
17)
</a></li>
763 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/11/">November (
10)
</a></li>
765 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2012/12/">December (
7)
</a></li>
772 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/01/">January (
16)
</a></li>
774 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/02/">February (
6)
</a></li>
776 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/03/">March (
6)
</a></li>
778 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/04/">April (
7)
</a></li>
780 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/05/">May (
3)
</a></li>
782 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/06/">June (
2)
</a></li>
784 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/07/">July (
7)
</a></li>
786 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/08/">August (
6)
</a></li>
788 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/09/">September (
4)
</a></li>
790 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/10/">October (
2)
</a></li>
792 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/11/">November (
3)
</a></li>
794 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2011/12/">December (
1)
</a></li>
801 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/01/">January (
2)
</a></li>
803 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/02/">February (
1)
</a></li>
805 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/03/">March (
3)
</a></li>
807 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/04/">April (
3)
</a></li>
809 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/05/">May (
9)
</a></li>
811 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/06/">June (
14)
</a></li>
813 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/07/">July (
12)
</a></li>
815 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/08/">August (
13)
</a></li>
817 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/09/">September (
7)
</a></li>
819 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/10/">October (
9)
</a></li>
821 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/11/">November (
13)
</a></li>
823 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2010/12/">December (
12)
</a></li>
830 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/01/">January (
8)
</a></li>
832 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/02/">February (
8)
</a></li>
834 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/03/">March (
12)
</a></li>
836 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/04/">April (
10)
</a></li>
838 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/05/">May (
9)
</a></li>
840 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/06/">June (
3)
</a></li>
842 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/07/">July (
4)
</a></li>
844 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/08/">August (
3)
</a></li>
846 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/09/">September (
1)
</a></li>
848 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/10/">October (
2)
</a></li>
850 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/11/">November (
3)
</a></li>
852 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2009/12/">December (
3)
</a></li>
859 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2008/11/">November (
5)
</a></li>
861 <li><a href=
"http://people.skolelinux.org/pere/blog/archive/2008/12/">December (
7)
</a></li>
872 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (
16)
</a></li>
874 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/amiga">amiga (
1)
</a></li>
876 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/aros">aros (
1)
</a></li>
878 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/bankid">bankid (
4)
</a></li>
880 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/betalkontant">betalkontant (
9)
</a></li>
882 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (
12)
</a></li>
884 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (
17)
</a></li>
886 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/bsa">bsa (
2)
</a></li>
888 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (
2)
</a></li>
890 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/debian">debian (
176)
</a></li>
892 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (
159)
</a></li>
894 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (
8)
</a></li>
896 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/digistan">digistan (
11)
</a></li>
898 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/dld">dld (
17)
</a></li>
900 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/docbook">docbook (
28)
</a></li>
902 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (
4)
</a></li>
904 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/english">english (
426)
</a></li>
906 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (
23)
</a></li>
908 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (
14)
</a></li>
910 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (
34)
</a></li>
912 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (
9)
</a></li>
914 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (
20)
</a></li>
916 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/h264">h264 (
20)
</a></li>
918 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/intervju">intervju (
43)
</a></li>
920 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (
16)
</a></li>
922 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/kart">kart (
23)
</a></li>
924 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/kodi">kodi (
4)
</a></li>
926 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/ldap">ldap (
9)
</a></li>
928 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/lego">lego (
4)
</a></li>
930 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/lenker">lenker (
8)
</a></li>
932 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (
2)
</a></li>
934 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (
1)
</a></li>
936 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (
8)
</a></li>
938 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (
42)
</a></li>
940 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (
13)
</a></li>
942 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/noark5">noark5 (
23)
</a></li>
944 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/norsk">norsk (
317)
</a></li>
946 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/nuug">nuug (
198)
</a></li>
948 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (
40)
</a></li>
950 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/open311">open311 (
2)
</a></li>
952 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (
75)
</a></li>
954 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/personvern">personvern (
111)
</a></li>
956 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/raid">raid (
2)
</a></li>
958 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/reactos">reactos (
1)
</a></li>
960 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/reprap">reprap (
11)
</a></li>
962 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/rfid">rfid (
3)
</a></li>
964 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/robot">robot (
12)
</a></li>
966 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/rss">rss (
1)
</a></li>
968 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/ruter">ruter (
7)
</a></li>
970 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (
2)
</a></li>
972 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (
59)
</a></li>
974 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (
4)
</a></li>
976 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (
5)
</a></li>
978 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/standard">standard (
71)
</a></li>
980 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (
7)
</a></li>
982 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (
12)
</a></li>
984 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (
61)
</a></li>
986 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (
5)
</a></li>
988 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/usenix">usenix (
2)
</a></li>
990 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/valg">valg (
9)
</a></li>
992 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri (
19)
</a></li>
994 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/video">video (
75)
</a></li>
996 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (
4)
</a></li>
998 <li><a href=
"http://people.skolelinux.org/pere/blog/tags/web">web (
42)
</a></li>
1004 <p style=
"text-align: right">
1005 Created by
<a href=
"http://steve.org.uk/Software/chronicle">Chronicle v4.6
</a>