1 <!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Strict//EN"
2 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3 <html xmlns=
"http://www.w3.org/1999/xhtml" dir=
"ltr">
5 <meta http-equiv=
"Content-Type" content=
"text/html;charset=utf-8" />
6 <title>Petter Reinholdtsen:
12 years of outages - summarised by Stuart Kendrick
</title>
7 <link rel=
"stylesheet" type=
"text/css" media=
"screen" href=
"http://people.skolelinux.org/pere/blog/style.css" />
8 <link rel=
"stylesheet" type=
"text/css" media=
"screen" href=
"http://people.skolelinux.org/pere/blog/vim.css" />
15 <a href=
"http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen
</a>
23 <div class=
"title">12 years of outages - summarised by Stuart Kendrick
</div>
24 <div class=
"date">26th October
2012</div>
25 <div class=
"body"><p>I work at the
<a href=
"http://www.uio.no/">University of Oslo
</a>
26 looking after the computers, mostly on the unix side, but in general
27 all over the place. I am also a member (and currently leader) of
28 <a href=
"http://www.nuug.no/">the NUUG association
</a>, which in turn
29 make me a member of
<a href=
"http://www.usenix.org/">USENIX
</a>. NUUG
30 is an member organisation for us in Norway interested in free
31 software, open standards and unix like operating systems, and USENIX
32 is a US based member organisation with similar targets. And thanks to
33 these memberships, I get all issues of the great USENIX magazine
34 <a href=
"https://www.usenix.org/publications/login">;login:
</a> in the
35 mail several times a year. The magazine is great, and I read most of
38 <p>In the last issue of the USENIX magazine ;login:, there is an
39 article by
<a href=
"http://www.skendric.com/">Stuart Kendrick
</a> from
40 Fred Hutchinson Cancer Research Center titled
41 "
<a href=
"https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down">What
42 Takes Us Down
</a>" (longer version also
43 <a href="http://www.skendric.com/problem/incident-analysis/
2012-
06-
30/What-Takes-Us-Down.pdf
">available
44 from his own site</a>), where he report what he found when he
45 processed the outage reports (both planned and unplanned) from the
46 last twelve years and classified them according to cause, time of day,
47 etc etc. The article is a good read to get some empirical data on
48 what kind of problems affect a data centre, but what really inspired
49 me was the kind of reporting they had put in place since 2000.<p>
51 <p>The centre set up a mailing list, and started to send fairly
52 standardised messages to this list when a outage was planned or when
53 it already occurred, to announce the plan and get feedback on the
54 assumtions on scope and user impact. Here is the two example from the
55 article: First the unplanned outage:
58 Subject: Exchange 2003 Cluster Issues
59 Severity: Critical (Unplanned)
60 Start: Monday, May 7, 2012, 11:58
61 End: Monday, May 7, 2012, 12:38
64 Description: The HTTPS service on the Exchange cluster crashed, triggering
67 User Impact: During this period, all Exchange users were unable to
68 access e-mail. Zimbra users were unaffected.
72 Next the planned outage:
75 Subject: H Building Switch Upgrades
76 Severity: Major (Planned)
77 Start: Saturday, June 16, 2012, 06:00
78 End: Saturday, June 16, 2012, 16:00
81 Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
82 stations. We will replace these with newer Catalyst
84 User Impact: All users on H2 will be isolated from the network during
85 this work. Afterward, they will have gigabit
90 <p>He notes in his article that the date formats and other fields have
91 been a bit too free form to make it easy to automatically process them
92 into a database for further analysis, and I would have used ISO 8601
93 dates myself to make it easier to process (in other words I would ask
94 people to write '2012-06-16 06:00 +0000' instead of the start time
95 format listed above). There are also other issues with the format
96 that could be improved, read the article for the details.</p>
98 <p>I find the idea of standardising outage messages seem to be such a
99 good idea that I would like to get it implemented here at the
100 university too. We do register
101 <a href="http://www.uio.no/tjenester/it/aktuelt/planlagte-tjenesteavbrudd/
">planned
102 changes and outages in a calendar</a>, and report the to a mailing
103 list, but we do not do so in a structured format and there is not a
104 report to the same location for unplanned outages. Perhaps something
105 for other sites to consider too?</p>
108 <div class="tags
">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english
">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug
">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard
">standard</a>, <a href="http://people.skolelinux.org/pere/blog/tags/usenix
">usenix</a>.</div>
126 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2017/
01/
">January (1)</a></li>
133 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
01/
">January (3)</a></li>
135 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
02/
">February (2)</a></li>
137 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
03/
">March (3)</a></li>
139 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
04/
">April (8)</a></li>
141 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
05/
">May (8)</a></li>
143 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
06/
">June (2)</a></li>
145 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
07/
">July (2)</a></li>
147 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
08/
">August (5)</a></li>
149 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
09/
">September (2)</a></li>
151 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
10/
">October (3)</a></li>
153 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
11/
">November (8)</a></li>
155 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2016/
12/
">December (5)</a></li>
162 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
01/
">January (7)</a></li>
164 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
02/
">February (6)</a></li>
166 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
03/
">March (1)</a></li>
168 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
04/
">April (4)</a></li>
170 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
05/
">May (3)</a></li>
172 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
06/
">June (4)</a></li>
174 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
07/
">July (6)</a></li>
176 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
08/
">August (2)</a></li>
178 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
09/
">September (2)</a></li>
180 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
10/
">October (9)</a></li>
182 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
11/
">November (6)</a></li>
184 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2015/
12/
">December (3)</a></li>
191 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
01/
">January (2)</a></li>
193 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
02/
">February (3)</a></li>
195 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
03/
">March (8)</a></li>
197 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
04/
">April (7)</a></li>
199 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
05/
">May (1)</a></li>
201 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
06/
">June (2)</a></li>
203 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
07/
">July (2)</a></li>
205 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
08/
">August (2)</a></li>
207 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
09/
">September (5)</a></li>
209 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
10/
">October (6)</a></li>
211 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
11/
">November (3)</a></li>
213 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2014/
12/
">December (5)</a></li>
220 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
01/
">January (11)</a></li>
222 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
02/
">February (9)</a></li>
224 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
03/
">March (9)</a></li>
226 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
04/
">April (6)</a></li>
228 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
05/
">May (9)</a></li>
230 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
06/
">June (10)</a></li>
232 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
07/
">July (7)</a></li>
234 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
08/
">August (3)</a></li>
236 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
09/
">September (5)</a></li>
238 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
10/
">October (7)</a></li>
240 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
11/
">November (9)</a></li>
242 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2013/
12/
">December (3)</a></li>
249 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
01/
">January (7)</a></li>
251 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
02/
">February (10)</a></li>
253 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
03/
">March (17)</a></li>
255 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
04/
">April (12)</a></li>
257 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
05/
">May (12)</a></li>
259 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
06/
">June (20)</a></li>
261 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
07/
">July (17)</a></li>
263 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
08/
">August (6)</a></li>
265 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
09/
">September (9)</a></li>
267 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
10/
">October (17)</a></li>
269 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
11/
">November (10)</a></li>
271 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2012/
12/
">December (7)</a></li>
278 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
01/
">January (16)</a></li>
280 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
02/
">February (6)</a></li>
282 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
03/
">March (6)</a></li>
284 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
04/
">April (7)</a></li>
286 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
05/
">May (3)</a></li>
288 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
06/
">June (2)</a></li>
290 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
07/
">July (7)</a></li>
292 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
08/
">August (6)</a></li>
294 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
09/
">September (4)</a></li>
296 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
10/
">October (2)</a></li>
298 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
11/
">November (3)</a></li>
300 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2011/
12/
">December (1)</a></li>
307 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
01/
">January (2)</a></li>
309 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
02/
">February (1)</a></li>
311 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
03/
">March (3)</a></li>
313 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
04/
">April (3)</a></li>
315 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
05/
">May (9)</a></li>
317 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
06/
">June (14)</a></li>
319 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
07/
">July (12)</a></li>
321 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
08/
">August (13)</a></li>
323 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
09/
">September (7)</a></li>
325 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
10/
">October (9)</a></li>
327 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
11/
">November (13)</a></li>
329 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2010/
12/
">December (12)</a></li>
336 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
01/
">January (8)</a></li>
338 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
02/
">February (8)</a></li>
340 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
03/
">March (12)</a></li>
342 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
04/
">April (10)</a></li>
344 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
05/
">May (9)</a></li>
346 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
06/
">June (3)</a></li>
348 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
07/
">July (4)</a></li>
350 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
08/
">August (3)</a></li>
352 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
09/
">September (1)</a></li>
354 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
10/
">October (2)</a></li>
356 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
11/
">November (3)</a></li>
358 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2009/
12/
">December (3)</a></li>
365 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2008/
11/
">November (5)</a></li>
367 <li><a href="http://people.skolelinux.org/pere/blog/archive/
2008/
12/
">December (7)</a></li>
378 <li><a href="http://people.skolelinux.org/pere/blog/tags/
3d-printer
">3d-printer (13)</a></li>
380 <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga
">amiga (1)</a></li>
382 <li><a href="http://people.skolelinux.org/pere/blog/tags/aros
">aros (1)</a></li>
384 <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid
">bankid (4)</a></li>
386 <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin
">bitcoin (9)</a></li>
388 <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem
">bootsystem (16)</a></li>
390 <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa
">bsa (2)</a></li>
392 <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath
">chrpath (2)</a></li>
394 <li><a href="http://people.skolelinux.org/pere/blog/tags/debian
">debian (144)</a></li>
396 <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu
">debian edu (158)</a></li>
398 <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan
">digistan (10)</a></li>
400 <li><a href="http://people.skolelinux.org/pere/blog/tags/dld
">dld (16)</a></li>
402 <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook
">docbook (23)</a></li>
404 <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser
">drivstoffpriser (4)</a></li>
406 <li><a href="http://people.skolelinux.org/pere/blog/tags/english
">english (337)</a></li>
408 <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami
">fiksgatami (23)</a></li>
410 <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling
">fildeling (12)</a></li>
412 <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture
">freeculture (29)</a></li>
414 <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox
">freedombox (9)</a></li>
416 <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen
">frikanalen (18)</a></li>
418 <li><a href="http://people.skolelinux.org/pere/blog/tags/h264
">h264 (20)</a></li>
420 <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju
">intervju (42)</a></li>
422 <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram
">isenkram (15)</a></li>
424 <li><a href="http://people.skolelinux.org/pere/blog/tags/kart
">kart (19)</a></li>
426 <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap
">ldap (9)</a></li>
428 <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker
">lenker (8)</a></li>
430 <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd
">lsdvd (2)</a></li>
432 <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp
">ltsp (1)</a></li>
434 <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network
">mesh network (8)</a></li>
436 <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia
">multimedia (39)</a></li>
438 <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software
">nice free software (9)</a></li>
440 <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk
">norsk (285)</a></li>
442 <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug
">nuug (182)</a></li>
444 <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn
">offentlig innsyn (26)</a></li>
446 <li><a href="http://people.skolelinux.org/pere/blog/tags/open311
">open311 (2)</a></li>
448 <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett
">opphavsrett (62)</a></li>
450 <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern
">personvern (96)</a></li>
452 <li><a href="http://people.skolelinux.org/pere/blog/tags/raid
">raid (1)</a></li>
454 <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos
">reactos (1)</a></li>
456 <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap
">reprap (11)</a></li>
458 <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid
">rfid (3)</a></li>
460 <li><a href="http://people.skolelinux.org/pere/blog/tags/robot
">robot (10)</a></li>
462 <li><a href="http://people.skolelinux.org/pere/blog/tags/rss
">rss (1)</a></li>
464 <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter
">ruter (5)</a></li>
466 <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki
">scraperwiki (2)</a></li>
468 <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet
">sikkerhet (52)</a></li>
470 <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary
">sitesummary (4)</a></li>
472 <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis
">skepsis (5)</a></li>
474 <li><a href="http://people.skolelinux.org/pere/blog/tags/standard
">standard (50)</a></li>
476 <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll
">stavekontroll (5)</a></li>
478 <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget
">stortinget (10)</a></li>
480 <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance
">surveillance (44)</a></li>
482 <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin
">sysadmin (2)</a></li>
484 <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix
">usenix (2)</a></li>
486 <li><a href="http://people.skolelinux.org/pere/blog/tags/valg
">valg (8)</a></li>
488 <li><a href="http://people.skolelinux.org/pere/blog/tags/video
">video (59)</a></li>
490 <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap
">vitenskap (4)</a></li>
492 <li><a href="http://people.skolelinux.org/pere/blog/tags/web
">web (38)</a></li>
498 <p style="text-align: right
">
499 Created by <a href="http://steve.org.uk/Software/chronicle
">Chronicle v4.6</a>