blog/12_years_of_outages___summarised_by_Stuart_Kendrick.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: 12 years of outages - summarised by Stuart Kendrick</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">12 years of outages - summarised by Stuart Kendrick</div>
  24       <div class="date">26th October 2012</div>
  25       <div class="body"><p>I work at the <a href="http://www.uio.no/">University of Oslo</a>
  26 looking after the computers, mostly on the unix side, but in general
  27 all over the place.  I am also a member (and currently leader) of
  28 <a href="http://www.nuug.no/">the NUUG association</a>, which in turn
  29 make me a member of <a href="http://www.usenix.org/">USENIX</a>.  NUUG
  30 is an member organisation for us in Norway interested in free
  31 software, open standards and unix like operating systems, and USENIX
  32 is a US based member organisation with similar targets.  And thanks to
  33 these memberships, I get all issues of the great USENIX magazine
  34 <a href="https://www.usenix.org/publications/login">;login:</a> in the
  35 mail several times a year.  The magazine is great, and I read most of
  36 it every time.</p>
  37
  38 <p>In the last issue of the USENIX magazine ;login:, there is an
  39 article by <a href="http://www.skendric.com/">Stuart Kendrick</a> from
  40 Fred Hutchinson Cancer Research Center titled
  41 "<a href="https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down">What
  42 Takes Us Down</a>" (longer version also
  43 <a href="http://www.skendric.com/problem/incident-analysis/2012-06-30/What-Takes-Us-Down.pdf">available
  44 from his own site</a>), where he report what he found when he
  45 processed the outage reports (both planned and unplanned) from the
  46 last twelve years and classified them according to cause, time of day,
  47 etc etc.  The article is a good read to get some empirical data on
  48 what kind of problems affect a data centre, but what really inspired
  49 me was the kind of reporting they had put in place since 2000.<p>
  50
  51 <p>The centre set up a mailing list, and started to send fairly
  52 standardised messages to this list when a outage was planned or when
  53 it already occurred, to announce the plan and get feedback on the
  54 assumtions on scope and user impact.  Here is the two example from the
  55 article: First the unplanned outage:
  56
  57 <blockquote><pre>
  58 Subject:     Exchange 2003 Cluster Issues
  59 Severity:    Critical (Unplanned)
  60 Start:       Monday, May 7, 2012, 11:58
  61 End:         Monday, May 7, 2012, 12:38
  62 Duration:    40 minutes
  63 Scope:       Exchange 2003
  64 Description: The HTTPS service on the Exchange cluster crashed, triggering
  65              a cluster failover.
  66
  67 User Impact: During this period, all Exchange users were unable to
  68              access e-mail. Zimbra users were unaffected.
  69 Technician:  [xxx]
  70 </pre></blockquote>
  71
  72 Next the planned outage:
  73
  74 <blockquote><pre>
  75 Subject:     H Building Switch Upgrades
  76 Severity:    Major (Planned)
  77 Start:       Saturday, June 16, 2012, 06:00
  78 End:         Saturday, June 16, 2012, 16:00
  79 Duration:    10 hours
  80 Scope:       H2 Transport
  81 Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
  82              stations. We will replace these with newer Catalyst
  83              4510s.
  84 User Impact: All users on H2 will be isolated from the network during
  85              this work. Afterward, they will have gigabit
  86              connectivity.
  87 Technician:  [xxx]
  88 </pre></blockquote>
  89
  90 <p>He notes in his article that the date formats and other fields have
  91 been a bit too free form to make it easy to automatically process them
  92 into a database for further analysis, and I would have used ISO 8601
  93 dates myself to make it easier to process (in other words I would ask
  94 people to write '2012-06-16 06:00 +0000' instead of the start time
  95 format listed above).  There are also other issues with the format
  96 that could be improved, read the article for the details.</p>
  97
  98 <p>I find the idea of standardising outage messages seem to be such a
  99 good idea that I would like to get it implemented here at the
 100 university too.  We do register
 101 <a href="http://www.uio.no/tjenester/it/aktuelt/planlagte-tjenesteavbrudd/">planned
 102 changes and outages in a calendar</a>, and report the to a mailing
 103 list, but we do not do so in a structured format and there is not a
 104 report to the same location for unplanned outages.  Perhaps something
 105 for other sites to consider too?</p>
 106 </div>
 107
 108       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.</div>
 109
 110
 111     </div>
 112
 113
 114
 115
 116     <div id="sidebar">
 117
 118
 119
 120 <h2>Archive</h2>
 121 <ul>
 122
 123 <li>2014
 124 <ul>
 125
 126 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
 127
 128 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
 129
 130 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
 131
 132 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
 133
 134 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
 135
 136 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (1)</a></li>
 137
 138 </ul></li>
 139
 140 <li>2013
 141 <ul>
 142
 143 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 144
 145 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 146
 147 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 148
 149 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 150
 151 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 152
 153 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 154
 155 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 156
 157 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 158
 159 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
 160
 161 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
 162
 163 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
 164
 165 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
 166
 167 </ul></li>
 168
 169 <li>2012
 170 <ul>
 171
 172 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 173
 174 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 175
 176 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 177
 178 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 179
 180 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 181
 182 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 183
 184 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 185
 186 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 187
 188 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 189
 190 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 191
 192 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 193
 194 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 195
 196 </ul></li>
 197
 198 <li>2011
 199 <ul>
 200
 201 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 202
 203 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 204
 205 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 206
 207 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 208
 209 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 210
 211 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 212
 213 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 214
 215 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 216
 217 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 218
 219 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 220
 221 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 222
 223 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 224
 225 </ul></li>
 226
 227 <li>2010
 228 <ul>
 229
 230 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 231
 232 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 233
 234 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 235
 236 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 237
 238 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 239
 240 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 241
 242 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 243
 244 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 245
 246 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 247
 248 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 249
 250 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 251
 252 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 253
 254 </ul></li>
 255
 256 <li>2009
 257 <ul>
 258
 259 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 260
 261 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 262
 263 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 264
 265 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 266
 267 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 268
 269 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 270
 271 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 272
 273 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 274
 275 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 276
 277 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 278
 279 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 280
 281 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 282
 283 </ul></li>
 284
 285 <li>2008
 286 <ul>
 287
 288 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 289
 290 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 291
 292 </ul></li>
 293
 294 </ul>
 295
 296
 297
 298 <h2>Tags</h2>
 299 <ul>
 300
 301  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 302
 303  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 304
 305  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 306
 307  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 308
 309  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (8)</a></li>
 310
 311  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (14)</a></li>
 312
 313  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 314
 315  <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
 316
 317  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (98)</a></li>
 318
 319  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (146)</a></li>
 320
 321  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 322
 323  <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (15)</a></li>
 324
 325  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (10)</a></li>
 326
 327  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 328
 329  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (247)</a></li>
 330
 331  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (21)</a></li>
 332
 333  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 334
 335  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (12)</a></li>
 336
 337  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (8)</a></li>
 338
 339  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (11)</a></li>
 340
 341  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (40)</a></li>
 342
 343  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (9)</a></li>
 344
 345  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (18)</a></li>
 346
 347  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
 348
 349  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (7)</a></li>
 350
 351  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 352
 353  <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
 354
 355  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (28)</a></li>
 356
 357  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (246)</a></li>
 358
 359  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (162)</a></li>
 360
 361  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (11)</a></li>
 362
 363  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 364
 365  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (46)</a></li>
 366
 367  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (72)</a></li>
 368
 369  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 370
 371  <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
 372
 373  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 374
 375  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
 376
 377  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (9)</a></li>
 378
 379  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 380
 381  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 382
 383  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 384
 385  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (40)</a></li>
 386
 387  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 388
 389  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 390
 391  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (44)</a></li>
 392
 393  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 394
 395  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (9)</a></li>
 396
 397  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (25)</a></li>
 398
 399  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (1)</a></li>
 400
 401  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
 402
 403  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (42)</a></li>
 404
 405  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 406
 407  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (32)</a></li>
 408
 409 </ul>
 410
 411
 412     </div>
 413     <p style="text-align: right">
 414  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 415 </p>
 416
 417   </body>
 418 </html>