blog/12_years_of_outages___summarised_by_Stuart_Kendrick.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: 12 years of outages - summarised by Stuart Kendrick</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">12 years of outages - summarised by Stuart Kendrick</div>
  24       <div class="date">26th October 2012</div>
  25       <div class="body"><p>I work at the <a href="http://www.uio.no/">University of Oslo</a>
  26 looking after the computers, mostly on the unix side, but in general
  27 all over the place.  I am also a member (and currently leader) of
  28 <a href="http://www.nuug.no/">the NUUG association</a>, which in turn
  29 make me a member of <a href="http://www.usenix.org/">USENIX</a>.  NUUG
  30 is an member organisation for us in Norway interested in free
  31 software, open standards and unix like operating systems, and USENIX
  32 is a US based member organisation with similar targets.  And thanks to
  33 these memberships, I get all issues of the great USENIX magazine
  34 <a href="https://www.usenix.org/publications/login">;login:</a> in the
  35 mail several times a year.  The magazine is great, and I read most of
  36 it every time.</p>
  37
  38 <p>In the last issue of the USENIX magazine ;login:, there is an
  39 article by <a href="http://www.skendric.com/">Stuart Kendrick</a> from
  40 Fred Hutchinson Cancer Research Center titled
  41 "<a href="https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down">What
  42 Takes Us Down</a>" (longer version also
  43 <a href="http://www.skendric.com/problem/incident-analysis/2012-06-30/What-Takes-Us-Down.pdf">available
  44 from his own site</a>), where he report what he found when he
  45 processed the outage reports (both planned and unplanned) from the
  46 last twelve years and classified them according to cause, time of day,
  47 etc etc.  The article is a good read to get some empirical data on
  48 what kind of problems affect a data centre, but what really inspired
  49 me was the kind of reporting they had put in place since 2000.<p>
  50
  51 <p>The centre set up a mailing list, and started to send fairly
  52 standardised messages to this list when a outage was planned or when
  53 it already occurred, to announce the plan and get feedback on the
  54 assumtions on scope and user impact.  Here is the two example from the
  55 article: First the unplanned outage:
  56
  57 <blockquote><pre>
  58 Subject:     Exchange 2003 Cluster Issues
  59 Severity:    Critical (Unplanned)
  60 Start:       Monday, May 7, 2012, 11:58
  61 End:         Monday, May 7, 2012, 12:38
  62 Duration:    40 minutes
  63 Scope:       Exchange 2003
  64 Description: The HTTPS service on the Exchange cluster crashed, triggering
  65              a cluster failover.
  66
  67 User Impact: During this period, all Exchange users were unable to
  68              access e-mail. Zimbra users were unaffected.
  69 Technician:  [xxx]
  70 </pre></blockquote>
  71
  72 Next the planned outage:
  73
  74 <blockquote><pre>
  75 Subject:     H Building Switch Upgrades
  76 Severity:    Major (Planned)
  77 Start:       Saturday, June 16, 2012, 06:00
  78 End:         Saturday, June 16, 2012, 16:00
  79 Duration:    10 hours
  80 Scope:       H2 Transport
  81 Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
  82              stations. We will replace these with newer Catalyst
  83              4510s.
  84 User Impact: All users on H2 will be isolated from the network during
  85              this work. Afterward, they will have gigabit
  86              connectivity.
  87 Technician:  [xxx]
  88 </pre></blockquote>
  89
  90 <p>He notes in his article that the date formats and other fields have
  91 been a bit too free form to make it easy to automatically process them
  92 into a database for further analysis, and I would have used ISO 8601
  93 dates myself to make it easier to process (in other words I would ask
  94 people to write '2012-06-16 06:00 +0000' instead of the start time
  95 format listed above).  There are also other issues with the format
  96 that could be improved, read the article for the details.</p>
  97
  98 <p>I find the idea of standardising outage messages seem to be such a
  99 good idea that I would like to get it implemented here at the
 100 university too.  We do register
 101 <a href="http://www.uio.no/tjenester/it/aktuelt/planlagte-tjenesteavbrudd/">planned
 102 changes and outages in a calendar</a>, and report the to a mailing
 103 list, but we do not do so in a structured format and there is not a
 104 report to the same location for unplanned outages.  Perhaps something
 105 for other sites to consider too?</p>
 106 </div>
 107
 108       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.</div>
 109
 110
 111     </div>
 112
 113
 114
 115
 116     <div id="sidebar">
 117
 118
 119
 120 <h2>Archive</h2>
 121 <ul>
 122
 123 <li>2013
 124 <ul>
 125
 126 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 127
 128 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 129
 130 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 131
 132 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 133
 134 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 135
 136 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 137
 138 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 139
 140 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (1)</a></li>
 141
 142 </ul></li>
 143
 144 <li>2012
 145 <ul>
 146
 147 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 148
 149 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 150
 151 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 152
 153 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 154
 155 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 156
 157 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 158
 159 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 160
 161 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 162
 163 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 164
 165 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 166
 167 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 168
 169 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 170
 171 </ul></li>
 172
 173 <li>2011
 174 <ul>
 175
 176 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 177
 178 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 179
 180 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 181
 182 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 183
 184 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 185
 186 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 187
 188 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 189
 190 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 191
 192 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 193
 194 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 195
 196 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 197
 198 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 199
 200 </ul></li>
 201
 202 <li>2010
 203 <ul>
 204
 205 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 206
 207 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 208
 209 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 210
 211 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 212
 213 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 214
 215 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 216
 217 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 218
 219 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 220
 221 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 222
 223 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 224
 225 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 226
 227 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 228
 229 </ul></li>
 230
 231 <li>2009
 232 <ul>
 233
 234 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 235
 236 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 237
 238 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 239
 240 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 241
 242 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 243
 244 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 245
 246 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 247
 248 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 249
 250 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 251
 252 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 253
 254 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 255
 256 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 257
 258 </ul></li>
 259
 260 <li>2008
 261 <ul>
 262
 263 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 264
 265 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 266
 267 </ul></li>
 268
 269 </ul>
 270
 271
 272
 273 <h2>Tags</h2>
 274 <ul>
 275
 276  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 277
 278  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 279
 280  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 281
 282  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 283
 284  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (7)</a></li>
 285
 286  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (12)</a></li>
 287
 288  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 289
 290  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (83)</a></li>
 291
 292  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (138)</a></li>
 293
 294  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 295
 296  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (10)</a></li>
 297
 298  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 299
 300  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (211)</a></li>
 301
 302  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (21)</a></li>
 303
 304  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 305
 306  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (12)</a></li>
 307
 308  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (11)</a></li>
 309
 310  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (37)</a></li>
 311
 312  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (7)</a></li>
 313
 314  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (18)</a></li>
 315
 316  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (8)</a></li>
 317
 318  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (6)</a></li>
 319
 320  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 321
 322  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (25)</a></li>
 323
 324  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (234)</a></li>
 325
 326  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (153)</a></li>
 327
 328  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (8)</a></li>
 329
 330  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 331
 332  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (44)</a></li>
 333
 334  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (65)</a></li>
 335
 336  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 337
 338  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 339
 340  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
 341
 342  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (7)</a></li>
 343
 344  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 345
 346  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 347
 348  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 349
 350  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (29)</a></li>
 351
 352  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 353
 354  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 355
 356  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (43)</a></li>
 357
 358  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 359
 360  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (7)</a></li>
 361
 362  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (15)</a></li>
 363
 364  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (1)</a></li>
 365
 366  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (7)</a></li>
 367
 368  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (38)</a></li>
 369
 370  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 371
 372  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (26)</a></li>
 373
 374 </ul>
 375
 376
 377     </div>
 378     <p style="text-align: right">
 379  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 380 </p>
 381
 382   </body>
 383 </html>