blog/Some_notes_on_fault_tolerant_storage_systems.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Some notes on fault tolerant storage systems</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://www.hungry.com/~pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://www.hungry.com/~pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="http://www.hungry.com/~pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">Some notes on fault tolerant storage systems</div>
  24       <div class="date"> 1st November 2017</div>
  25       <div class="body"><p>If you care about how fault tolerant your storage is, you might
  26 find these articles and papers interesting.  They have formed how I
  27 think of when designing a storage system.</p>
  28
  29 <ul>
  30
  31 <li>USENIX :login; <a
  32 href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
  33 Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
  34 Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
  35 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
  36 H. Arpaci-Dusseau</li>
  37
  38 <li>ZDNet
  39 <a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
  40 RAID 5 stops working in 2009</a> by Robin Harris</li>
  41
  42 <li>ZDNet
  43 <a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
  44 RAID 6 stops working in 2019</a> by Robin Harris</li>
  45
  46 <li>USENIX FAST'07
  47 <a href="http://research.google.com/archive/disk_failures.pdf">Failure
  48 Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
  49 Wolf-Dietrich Weber and Luiz André Barroso</li>
  50
  51 <li>USENIX ;login: <a
  52 href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
  53 Integrity.  Finding Truth in a World of Guesses and Lies</a> by Doug
  54 Hughes</li>
  55
  56 <li>USENIX FAST'08
  57 <a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
  58 Analysis of Data Corruption in the Storage Stack</a> by
  59 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
  60 Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
  61
  62 <li>USENIX FAST'07 <a
  63 href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
  64 failures in the real world: what does an MTTF of 1,000,000 hours mean
  65 to you?</a> by B. Schroeder and G. A. Gibson.</li>
  66
  67 <li>USENIX ;login: <a
  68 href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
  69 Disks the Dominant Contributor for Storage Failures?  A Comprehensive
  70 Study of Storage Subsystem Failure Characteristics</a> by Weihang
  71 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
  72
  73 <li>SIGMETRICS 2007
  74 <a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
  75 analysis of latent sector errors in disk drives</a> by
  76 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
  77
  78 </ul>
  79
  80 <p>Several of these research papers are based on data collected from
  81 hundred thousands or millions of disk, and their findings are eye
  82 opening.  The short story is simply do not implicitly trust RAID or
  83 redundant storage systems.  Details matter.  And unfortunately there
  84 are few options on Linux addressing all the identified issues.  Both
  85 ZFS and Btrfs are doing a fairly good job, but have legal and
  86 practical issues on their own.  I wonder how cluster file systems like
  87 Ceph do in this regard.  After all, there is an old saying, you know
  88 you have a distributed system when the crash of a computer you have
  89 never heard of stops you from getting any work done.  The same holds
  90 true if fault tolerance do not work.</p>
  91
  92 <p>Just remember, in the end, it do not matter how redundant, or how
  93 fault tolerant your storage is, if you do not continuously monitor its
  94 status to detect and replace failed disks.</p>
  95
  96 <p>As usual, if you use Bitcoin and want to show your support of my
  97 activities, please send Bitcoin donations to my address
  98 <b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
  99 </div>
 100
 101       <div class="tags">Tags: <a href="http://www.hungry.com/~pere/blog/tags/english">english</a>, <a href="http://www.hungry.com/~pere/blog/tags/raid">raid</a>, <a href="http://www.hungry.com/~pere/blog/tags/sysadmin">sysadmin</a>.</div>
 102
 103
 104     </div>
 105
 106
 107
 108
 109     <div id="sidebar">
 110
 111
 112
 113 <h2>Archive</h2>
 114 <ul>
 115
 116 <li>2025
 117 <ul>
 118
 119 <li><a href="http://www.hungry.com/~pere/blog/archive/2025/01/">January (4)</a></li>
 120
 121 <li><a href="http://www.hungry.com/~pere/blog/archive/2025/02/">February (3)</a></li>
 122
 123 </ul></li>
 124
 125 <li>2024
 126 <ul>
 127
 128 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/01/">January (1)</a></li>
 129
 130 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/02/">February (1)</a></li>
 131
 132 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/03/">March (2)</a></li>
 133
 134 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/04/">April (3)</a></li>
 135
 136 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/05/">May (1)</a></li>
 137
 138 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/06/">June (1)</a></li>
 139
 140 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/07/">July (2)</a></li>
 141
 142 <li><a href="http://www.hungry.com/~pere/blog/archive/2024/12/">December (1)</a></li>
 143
 144 </ul></li>
 145
 146 <li>2023
 147 <ul>
 148
 149 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/01/">January (3)</a></li>
 150
 151 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/02/">February (1)</a></li>
 152
 153 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/04/">April (2)</a></li>
 154
 155 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/05/">May (3)</a></li>
 156
 157 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/06/">June (1)</a></li>
 158
 159 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/08/">August (1)</a></li>
 160
 161 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/09/">September (1)</a></li>
 162
 163 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/10/">October (1)</a></li>
 164
 165 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/11/">November (4)</a></li>
 166
 167 <li><a href="http://www.hungry.com/~pere/blog/archive/2023/12/">December (1)</a></li>
 168
 169 </ul></li>
 170
 171 <li>2022
 172 <ul>
 173
 174 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/02/">February (1)</a></li>
 175
 176 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/03/">March (3)</a></li>
 177
 178 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/04/">April (2)</a></li>
 179
 180 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/06/">June (2)</a></li>
 181
 182 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/07/">July (1)</a></li>
 183
 184 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/09/">September (1)</a></li>
 185
 186 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/10/">October (1)</a></li>
 187
 188 <li><a href="http://www.hungry.com/~pere/blog/archive/2022/12/">December (1)</a></li>
 189
 190 </ul></li>
 191
 192 <li>2021
 193 <ul>
 194
 195 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/01/">January (2)</a></li>
 196
 197 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/02/">February (1)</a></li>
 198
 199 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/05/">May (1)</a></li>
 200
 201 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/06/">June (1)</a></li>
 202
 203 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/07/">July (3)</a></li>
 204
 205 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/08/">August (1)</a></li>
 206
 207 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/09/">September (1)</a></li>
 208
 209 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/10/">October (1)</a></li>
 210
 211 <li><a href="http://www.hungry.com/~pere/blog/archive/2021/12/">December (1)</a></li>
 212
 213 </ul></li>
 214
 215 <li>2020
 216 <ul>
 217
 218 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/02/">February (2)</a></li>
 219
 220 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/03/">March (2)</a></li>
 221
 222 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/04/">April (2)</a></li>
 223
 224 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/05/">May (3)</a></li>
 225
 226 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/06/">June (2)</a></li>
 227
 228 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/07/">July (1)</a></li>
 229
 230 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/09/">September (1)</a></li>
 231
 232 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/10/">October (1)</a></li>
 233
 234 <li><a href="http://www.hungry.com/~pere/blog/archive/2020/11/">November (1)</a></li>
 235
 236 </ul></li>
 237
 238 <li>2019
 239 <ul>
 240
 241 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/01/">January (4)</a></li>
 242
 243 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/02/">February (3)</a></li>
 244
 245 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/03/">March (3)</a></li>
 246
 247 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/05/">May (2)</a></li>
 248
 249 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/06/">June (5)</a></li>
 250
 251 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/07/">July (2)</a></li>
 252
 253 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/08/">August (1)</a></li>
 254
 255 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/09/">September (1)</a></li>
 256
 257 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/11/">November (1)</a></li>
 258
 259 <li><a href="http://www.hungry.com/~pere/blog/archive/2019/12/">December (4)</a></li>
 260
 261 </ul></li>
 262
 263 <li>2018
 264 <ul>
 265
 266 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/01/">January (1)</a></li>
 267
 268 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/02/">February (5)</a></li>
 269
 270 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/03/">March (5)</a></li>
 271
 272 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/04/">April (3)</a></li>
 273
 274 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/06/">June (2)</a></li>
 275
 276 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/07/">July (5)</a></li>
 277
 278 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/08/">August (3)</a></li>
 279
 280 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/09/">September (3)</a></li>
 281
 282 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/10/">October (5)</a></li>
 283
 284 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/11/">November (2)</a></li>
 285
 286 <li><a href="http://www.hungry.com/~pere/blog/archive/2018/12/">December (4)</a></li>
 287
 288 </ul></li>
 289
 290 <li>2017
 291 <ul>
 292
 293 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/01/">January (4)</a></li>
 294
 295 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/02/">February (3)</a></li>
 296
 297 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/03/">March (5)</a></li>
 298
 299 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/04/">April (2)</a></li>
 300
 301 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/06/">June (5)</a></li>
 302
 303 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/07/">July (1)</a></li>
 304
 305 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/08/">August (1)</a></li>
 306
 307 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/09/">September (3)</a></li>
 308
 309 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/10/">October (5)</a></li>
 310
 311 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/11/">November (3)</a></li>
 312
 313 <li><a href="http://www.hungry.com/~pere/blog/archive/2017/12/">December (4)</a></li>
 314
 315 </ul></li>
 316
 317 <li>2016
 318 <ul>
 319
 320 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/01/">January (3)</a></li>
 321
 322 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/02/">February (2)</a></li>
 323
 324 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/03/">March (3)</a></li>
 325
 326 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/04/">April (8)</a></li>
 327
 328 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/05/">May (8)</a></li>
 329
 330 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/06/">June (2)</a></li>
 331
 332 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/07/">July (2)</a></li>
 333
 334 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/08/">August (5)</a></li>
 335
 336 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/09/">September (2)</a></li>
 337
 338 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/10/">October (3)</a></li>
 339
 340 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/11/">November (8)</a></li>
 341
 342 <li><a href="http://www.hungry.com/~pere/blog/archive/2016/12/">December (5)</a></li>
 343
 344 </ul></li>
 345
 346 <li>2015
 347 <ul>
 348
 349 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/01/">January (7)</a></li>
 350
 351 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/02/">February (6)</a></li>
 352
 353 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/03/">March (1)</a></li>
 354
 355 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/04/">April (4)</a></li>
 356
 357 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/05/">May (3)</a></li>
 358
 359 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/06/">June (4)</a></li>
 360
 361 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/07/">July (6)</a></li>
 362
 363 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/08/">August (2)</a></li>
 364
 365 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/09/">September (2)</a></li>
 366
 367 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/10/">October (9)</a></li>
 368
 369 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/11/">November (6)</a></li>
 370
 371 <li><a href="http://www.hungry.com/~pere/blog/archive/2015/12/">December (3)</a></li>
 372
 373 </ul></li>
 374
 375 <li>2014
 376 <ul>
 377
 378 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/01/">January (2)</a></li>
 379
 380 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/02/">February (3)</a></li>
 381
 382 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/03/">March (8)</a></li>
 383
 384 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/04/">April (7)</a></li>
 385
 386 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/05/">May (1)</a></li>
 387
 388 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/06/">June (2)</a></li>
 389
 390 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/07/">July (2)</a></li>
 391
 392 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/08/">August (2)</a></li>
 393
 394 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/09/">September (5)</a></li>
 395
 396 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/10/">October (6)</a></li>
 397
 398 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/11/">November (3)</a></li>
 399
 400 <li><a href="http://www.hungry.com/~pere/blog/archive/2014/12/">December (5)</a></li>
 401
 402 </ul></li>
 403
 404 <li>2013
 405 <ul>
 406
 407 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/01/">January (11)</a></li>
 408
 409 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/02/">February (9)</a></li>
 410
 411 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/03/">March (9)</a></li>
 412
 413 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/04/">April (6)</a></li>
 414
 415 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/05/">May (9)</a></li>
 416
 417 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/06/">June (10)</a></li>
 418
 419 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/07/">July (7)</a></li>
 420
 421 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/08/">August (3)</a></li>
 422
 423 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/09/">September (5)</a></li>
 424
 425 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/10/">October (7)</a></li>
 426
 427 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/11/">November (9)</a></li>
 428
 429 <li><a href="http://www.hungry.com/~pere/blog/archive/2013/12/">December (3)</a></li>
 430
 431 </ul></li>
 432
 433 <li>2012
 434 <ul>
 435
 436 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/01/">January (7)</a></li>
 437
 438 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/02/">February (10)</a></li>
 439
 440 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/03/">March (17)</a></li>
 441
 442 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/04/">April (12)</a></li>
 443
 444 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/05/">May (12)</a></li>
 445
 446 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/06/">June (20)</a></li>
 447
 448 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/07/">July (17)</a></li>
 449
 450 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/08/">August (6)</a></li>
 451
 452 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/09/">September (9)</a></li>
 453
 454 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/10/">October (17)</a></li>
 455
 456 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/11/">November (10)</a></li>
 457
 458 <li><a href="http://www.hungry.com/~pere/blog/archive/2012/12/">December (7)</a></li>
 459
 460 </ul></li>
 461
 462 <li>2011
 463 <ul>
 464
 465 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/01/">January (16)</a></li>
 466
 467 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/02/">February (6)</a></li>
 468
 469 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/03/">March (6)</a></li>
 470
 471 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/04/">April (7)</a></li>
 472
 473 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/05/">May (3)</a></li>
 474
 475 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/06/">June (2)</a></li>
 476
 477 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/07/">July (7)</a></li>
 478
 479 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/08/">August (6)</a></li>
 480
 481 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/09/">September (4)</a></li>
 482
 483 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/10/">October (2)</a></li>
 484
 485 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/11/">November (3)</a></li>
 486
 487 <li><a href="http://www.hungry.com/~pere/blog/archive/2011/12/">December (1)</a></li>
 488
 489 </ul></li>
 490
 491 <li>2010
 492 <ul>
 493
 494 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/01/">January (2)</a></li>
 495
 496 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/02/">February (1)</a></li>
 497
 498 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/03/">March (3)</a></li>
 499
 500 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/04/">April (3)</a></li>
 501
 502 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/05/">May (9)</a></li>
 503
 504 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/06/">June (14)</a></li>
 505
 506 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/07/">July (12)</a></li>
 507
 508 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/08/">August (13)</a></li>
 509
 510 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/09/">September (7)</a></li>
 511
 512 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/10/">October (9)</a></li>
 513
 514 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/11/">November (13)</a></li>
 515
 516 <li><a href="http://www.hungry.com/~pere/blog/archive/2010/12/">December (12)</a></li>
 517
 518 </ul></li>
 519
 520 <li>2009
 521 <ul>
 522
 523 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/01/">January (8)</a></li>
 524
 525 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/02/">February (8)</a></li>
 526
 527 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/03/">March (12)</a></li>
 528
 529 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/04/">April (10)</a></li>
 530
 531 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/05/">May (9)</a></li>
 532
 533 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/06/">June (3)</a></li>
 534
 535 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/07/">July (4)</a></li>
 536
 537 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/08/">August (3)</a></li>
 538
 539 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/09/">September (1)</a></li>
 540
 541 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/10/">October (2)</a></li>
 542
 543 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/11/">November (3)</a></li>
 544
 545 <li><a href="http://www.hungry.com/~pere/blog/archive/2009/12/">December (3)</a></li>
 546
 547 </ul></li>
 548
 549 <li>2008
 550 <ul>
 551
 552 <li><a href="http://www.hungry.com/~pere/blog/archive/2008/11/">November (5)</a></li>
 553
 554 <li><a href="http://www.hungry.com/~pere/blog/archive/2008/12/">December (7)</a></li>
 555
 556 </ul></li>
 557
 558 </ul>
 559
 560
 561
 562 <h2>Tags</h2>
 563 <ul>
 564
 565  <li><a href="http://www.hungry.com/~pere/blog/tags/3d-printer">3d-printer (19)</a></li>
 566
 567  <li><a href="http://www.hungry.com/~pere/blog/tags/amiga">amiga (1)</a></li>
 568
 569  <li><a href="http://www.hungry.com/~pere/blog/tags/aros">aros (1)</a></li>
 570
 571  <li><a href="http://www.hungry.com/~pere/blog/tags/bankid">bankid (4)</a></li>
 572
 573  <li><a href="http://www.hungry.com/~pere/blog/tags/betalkontant">betalkontant (9)</a></li>
 574
 575  <li><a href="http://www.hungry.com/~pere/blog/tags/bitcoin">bitcoin (13)</a></li>
 576
 577  <li><a href="http://www.hungry.com/~pere/blog/tags/bootsystem">bootsystem (17)</a></li>
 578
 579  <li><a href="http://www.hungry.com/~pere/blog/tags/bsa">bsa (2)</a></li>
 580
 581  <li><a href="http://www.hungry.com/~pere/blog/tags/chrpath">chrpath (3)</a></li>
 582
 583  <li><a href="http://www.hungry.com/~pere/blog/tags/debian">debian (209)</a></li>
 584
 585  <li><a href="http://www.hungry.com/~pere/blog/tags/debian edu">debian edu (159)</a></li>
 586
 587  <li><a href="http://www.hungry.com/~pere/blog/tags/debian-handbook">debian-handbook (9)</a></li>
 588
 589  <li><a href="http://www.hungry.com/~pere/blog/tags/digistan">digistan (11)</a></li>
 590
 591  <li><a href="http://www.hungry.com/~pere/blog/tags/dld">dld (18)</a></li>
 592
 593  <li><a href="http://www.hungry.com/~pere/blog/tags/docbook">docbook (33)</a></li>
 594
 595  <li><a href="http://www.hungry.com/~pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 596
 597  <li><a href="http://www.hungry.com/~pere/blog/tags/english">english (471)</a></li>
 598
 599  <li><a href="http://www.hungry.com/~pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
 600
 601  <li><a href="http://www.hungry.com/~pere/blog/tags/fildeling">fildeling (14)</a></li>
 602
 603  <li><a href="http://www.hungry.com/~pere/blog/tags/freeculture">freeculture (34)</a></li>
 604
 605  <li><a href="http://www.hungry.com/~pere/blog/tags/freedombox">freedombox (9)</a></li>
 606
 607  <li><a href="http://www.hungry.com/~pere/blog/tags/frikanalen">frikanalen (20)</a></li>
 608
 609  <li><a href="http://www.hungry.com/~pere/blog/tags/h264">h264 (20)</a></li>
 610
 611  <li><a href="http://www.hungry.com/~pere/blog/tags/intervju">intervju (43)</a></li>
 612
 613  <li><a href="http://www.hungry.com/~pere/blog/tags/isenkram">isenkram (21)</a></li>
 614
 615  <li><a href="http://www.hungry.com/~pere/blog/tags/kart">kart (23)</a></li>
 616
 617  <li><a href="http://www.hungry.com/~pere/blog/tags/kodi">kodi (6)</a></li>
 618
 619  <li><a href="http://www.hungry.com/~pere/blog/tags/ldap">ldap (9)</a></li>
 620
 621  <li><a href="http://www.hungry.com/~pere/blog/tags/lego">lego (5)</a></li>
 622
 623  <li><a href="http://www.hungry.com/~pere/blog/tags/lenker">lenker (8)</a></li>
 624
 625  <li><a href="http://www.hungry.com/~pere/blog/tags/linuxcnc">linuxcnc (8)</a></li>
 626
 627  <li><a href="http://www.hungry.com/~pere/blog/tags/lsdvd">lsdvd (3)</a></li>
 628
 629  <li><a href="http://www.hungry.com/~pere/blog/tags/ltsp">ltsp (1)</a></li>
 630
 631  <li><a href="http://www.hungry.com/~pere/blog/tags/madewithcc">madewithcc (3)</a></li>
 632
 633  <li><a href="http://www.hungry.com/~pere/blog/tags/mesh network">mesh network (8)</a></li>
 634
 635  <li><a href="http://www.hungry.com/~pere/blog/tags/multimedia">multimedia (48)</a></li>
 636
 637  <li><a href="http://www.hungry.com/~pere/blog/tags/nice free software">nice free software (15)</a></li>
 638
 639  <li><a href="http://www.hungry.com/~pere/blog/tags/noark5">noark5 (27)</a></li>
 640
 641  <li><a href="http://www.hungry.com/~pere/blog/tags/norsk">norsk (326)</a></li>
 642
 643  <li><a href="http://www.hungry.com/~pere/blog/tags/nuug">nuug (200)</a></li>
 644
 645  <li><a href="http://www.hungry.com/~pere/blog/tags/offentlig innsyn">offentlig innsyn (42)</a></li>
 646
 647  <li><a href="http://www.hungry.com/~pere/blog/tags/open311">open311 (2)</a></li>
 648
 649  <li><a href="http://www.hungry.com/~pere/blog/tags/opensnitch">opensnitch (4)</a></li>
 650
 651  <li><a href="http://www.hungry.com/~pere/blog/tags/opphavsrett">opphavsrett (76)</a></li>
 652
 653  <li><a href="http://www.hungry.com/~pere/blog/tags/personvern">personvern (114)</a></li>
 654
 655  <li><a href="http://www.hungry.com/~pere/blog/tags/raid">raid (4)</a></li>
 656
 657  <li><a href="http://www.hungry.com/~pere/blog/tags/reactos">reactos (1)</a></li>
 658
 659  <li><a href="http://www.hungry.com/~pere/blog/tags/reprap">reprap (11)</a></li>
 660
 661  <li><a href="http://www.hungry.com/~pere/blog/tags/rfid">rfid (3)</a></li>
 662
 663  <li><a href="http://www.hungry.com/~pere/blog/tags/robot">robot (17)</a></li>
 664
 665  <li><a href="http://www.hungry.com/~pere/blog/tags/rss">rss (1)</a></li>
 666
 667  <li><a href="http://www.hungry.com/~pere/blog/tags/ruter">ruter (7)</a></li>
 668
 669  <li><a href="http://www.hungry.com/~pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 670
 671  <li><a href="http://www.hungry.com/~pere/blog/tags/sikkerhet">sikkerhet (60)</a></li>
 672
 673  <li><a href="http://www.hungry.com/~pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 674
 675  <li><a href="http://www.hungry.com/~pere/blog/tags/skepsis">skepsis (5)</a></li>
 676
 677  <li><a href="http://www.hungry.com/~pere/blog/tags/standard">standard (80)</a></li>
 678
 679  <li><a href="http://www.hungry.com/~pere/blog/tags/stavekontroll">stavekontroll (7)</a></li>
 680
 681  <li><a href="http://www.hungry.com/~pere/blog/tags/stortinget">stortinget (14)</a></li>
 682
 683  <li><a href="http://www.hungry.com/~pere/blog/tags/surveillance">surveillance (65)</a></li>
 684
 685  <li><a href="http://www.hungry.com/~pere/blog/tags/sysadmin">sysadmin (6)</a></li>
 686
 687  <li><a href="http://www.hungry.com/~pere/blog/tags/usenix">usenix (2)</a></li>
 688
 689  <li><a href="http://www.hungry.com/~pere/blog/tags/valg">valg (9)</a></li>
 690
 691  <li><a href="http://www.hungry.com/~pere/blog/tags/verkidetfri">verkidetfri (23)</a></li>
 692
 693  <li><a href="http://www.hungry.com/~pere/blog/tags/video">video (82)</a></li>
 694
 695  <li><a href="http://www.hungry.com/~pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 696
 697  <li><a href="http://www.hungry.com/~pere/blog/tags/web">web (42)</a></li>
 698
 699 </ul>
 700
 701
 702     </div>
 703     <p style="text-align: right">
 704  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 705 </p>
 706
 707   </body>
 708 </html>