]> pere.pagekite.me Git - homepage.git/blob - blog/12_years_of_outages___summarised_by_Stuart_Kendrick.html
Ønsker meg radonmåler.
[homepage.git] / blog / 12_years_of_outages___summarised_by_Stuart_Kendrick.html
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
2 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
4 <head>
5 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
6 <title>Petter Reinholdtsen: 12 years of outages - summarised by Stuart Kendrick</title>
7 <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
8 <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
9
10
11 </head>
12 <body>
13 <div class="title">
14 <h1>
15 <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
16
17 </h1>
18
19 </div>
20
21
22 <div class="entry">
23 <div class="title">12 years of outages - summarised by Stuart Kendrick</div>
24 <div class="date">26th October 2012</div>
25 <div class="body"><p>I work at the <a href="http://www.uio.no/">University of Oslo</a>
26 looking after the computers, mostly on the unix side, but in general
27 all over the place. I am also a member (and currently leader) of
28 <a href="http://www.nuug.no/">the NUUG association</a>, which in turn
29 make me a member of <a href="http://www.usenix.org/">USENIX</a>. NUUG
30 is an member organisation for us in Norway interested in free
31 software, open standards and unix like operating systems, and USENIX
32 is a US based member organisation with similar targets. And thanks to
33 these memberships, I get all issues of the great USENIX magazine
34 <a href="https://www.usenix.org/publications/login">;login:</a> in the
35 mail several times a year. The magazine is great, and I read most of
36 it every time.</p>
37
38 <p>In the last issue of the USENIX magazine ;login:, there is an
39 article by <a href="http://www.skendric.com/">Stuart Kendrick</a> from
40 Fred Hutchinson Cancer Research Center titled
41 "<a href="https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down">What
42 Takes Us Down</a>" (longer version also
43 <a href="http://www.skendric.com/problem/incident-analysis/2012-06-30/What-Takes-Us-Down.pdf">available
44 from his own site</a>), where he report what he found when he
45 processed the outage reports (both planned and unplanned) from the
46 last twelve years and classified them according to cause, time of day,
47 etc etc. The article is a good read to get some empirical data on
48 what kind of problems affect a data centre, but what really inspired
49 me was the kind of reporting they had put in place since 2000.<p>
50
51 <p>The centre set up a mailing list, and started to send fairly
52 standardised messages to this list when a outage was planned or when
53 it already occurred, to announce the plan and get feedback on the
54 assumtions on scope and user impact. Here is the two example from the
55 article: First the unplanned outage:
56
57 <blockquote><pre>
58 Subject: Exchange 2003 Cluster Issues
59 Severity: Critical (Unplanned)
60 Start: Monday, May 7, 2012, 11:58
61 End: Monday, May 7, 2012, 12:38
62 Duration: 40 minutes
63 Scope: Exchange 2003
64 Description: The HTTPS service on the Exchange cluster crashed, triggering
65 a cluster failover.
66
67 User Impact: During this period, all Exchange users were unable to
68 access e-mail. Zimbra users were unaffected.
69 Technician: [xxx]
70 </pre></blockquote>
71
72 Next the planned outage:
73
74 <blockquote><pre>
75 Subject: H Building Switch Upgrades
76 Severity: Major (Planned)
77 Start: Saturday, June 16, 2012, 06:00
78 End: Saturday, June 16, 2012, 16:00
79 Duration: 10 hours
80 Scope: H2 Transport
81 Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
82 stations. We will replace these with newer Catalyst
83 4510s.
84 User Impact: All users on H2 will be isolated from the network during
85 this work. Afterward, they will have gigabit
86 connectivity.
87 Technician: [xxx]
88 </pre></blockquote>
89
90 <p>He notes in his article that the date formats and other fields have
91 been a bit too free form to make it easy to automatically process them
92 into a database for further analysis, and I would have used ISO 8601
93 dates myself to make it easier to process (in other words I would ask
94 people to write '2012-06-16 06:00 +0000' instead of the start time
95 format listed above). There are also other issues with the format
96 that could be improved, read the article for the details.</p>
97
98 <p>I find the idea of standardising outage messages seem to be such a
99 good idea that I would like to get it implemented here at the
100 university too. We do register
101 <a href="http://www.uio.no/tjenester/it/aktuelt/planlagte-tjenesteavbrudd/">planned
102 changes and outages in a calendar</a>, and report the to a mailing
103 list, but we do not do so in a structured format and there is not a
104 report to the same location for unplanned outages. Perhaps something
105 for other sites to consider too?</p>
106 </div>
107
108 <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>, <a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix</a>.</div>
109
110
111 </div>
112
113
114
115
116 <div id="sidebar">
117
118
119
120 <h2>Archive</h2>
121 <ul>
122
123 <li>2017
124 <ul>
125
126 <li><a href="http://people.skolelinux.org/pere/blog/archive/2017/01/">January (1)</a></li>
127
128 </ul></li>
129
130 <li>2016
131 <ul>
132
133 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/01/">January (3)</a></li>
134
135 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/02/">February (2)</a></li>
136
137 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/03/">March (3)</a></li>
138
139 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/04/">April (8)</a></li>
140
141 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/05/">May (8)</a></li>
142
143 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/06/">June (2)</a></li>
144
145 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/07/">July (2)</a></li>
146
147 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/08/">August (5)</a></li>
148
149 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/09/">September (2)</a></li>
150
151 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/10/">October (3)</a></li>
152
153 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/11/">November (8)</a></li>
154
155 <li><a href="http://people.skolelinux.org/pere/blog/archive/2016/12/">December (5)</a></li>
156
157 </ul></li>
158
159 <li>2015
160 <ul>
161
162 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/01/">January (7)</a></li>
163
164 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/02/">February (6)</a></li>
165
166 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/03/">March (1)</a></li>
167
168 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/04/">April (4)</a></li>
169
170 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/05/">May (3)</a></li>
171
172 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/06/">June (4)</a></li>
173
174 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/07/">July (6)</a></li>
175
176 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/08/">August (2)</a></li>
177
178 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/09/">September (2)</a></li>
179
180 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/10/">October (9)</a></li>
181
182 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/11/">November (6)</a></li>
183
184 <li><a href="http://people.skolelinux.org/pere/blog/archive/2015/12/">December (3)</a></li>
185
186 </ul></li>
187
188 <li>2014
189 <ul>
190
191 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
192
193 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
194
195 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
196
197 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
198
199 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
200
201 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
202
203 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
204
205 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
206
207 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (5)</a></li>
208
209 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/10/">October (6)</a></li>
210
211 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/11/">November (3)</a></li>
212
213 <li><a href="http://people.skolelinux.org/pere/blog/archive/2014/12/">December (5)</a></li>
214
215 </ul></li>
216
217 <li>2013
218 <ul>
219
220 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
221
222 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
223
224 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
225
226 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
227
228 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
229
230 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
231
232 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
233
234 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
235
236 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
237
238 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
239
240 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
241
242 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
243
244 </ul></li>
245
246 <li>2012
247 <ul>
248
249 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
250
251 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
252
253 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
254
255 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
256
257 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
258
259 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
260
261 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
262
263 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
264
265 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
266
267 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
268
269 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
270
271 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
272
273 </ul></li>
274
275 <li>2011
276 <ul>
277
278 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
279
280 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
281
282 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
283
284 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
285
286 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
287
288 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
289
290 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
291
292 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
293
294 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
295
296 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
297
298 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
299
300 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
301
302 </ul></li>
303
304 <li>2010
305 <ul>
306
307 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
308
309 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
310
311 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
312
313 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
314
315 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
316
317 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
318
319 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
320
321 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
322
323 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
324
325 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
326
327 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
328
329 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
330
331 </ul></li>
332
333 <li>2009
334 <ul>
335
336 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
337
338 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
339
340 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
341
342 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
343
344 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
345
346 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
347
348 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
349
350 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
351
352 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
353
354 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
355
356 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
357
358 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
359
360 </ul></li>
361
362 <li>2008
363 <ul>
364
365 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
366
367 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
368
369 </ul></li>
370
371 </ul>
372
373
374
375 <h2>Tags</h2>
376 <ul>
377
378 <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
379
380 <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
381
382 <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
383
384 <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
385
386 <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (9)</a></li>
387
388 <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (16)</a></li>
389
390 <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
391
392 <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
393
394 <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (144)</a></li>
395
396 <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
397
398 <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
399
400 <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (16)</a></li>
401
402 <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (23)</a></li>
403
404 <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
405
406 <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (337)</a></li>
407
408 <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
409
410 <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
411
412 <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (29)</a></li>
413
414 <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
415
416 <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (18)</a></li>
417
418 <li><a href="http://people.skolelinux.org/pere/blog/tags/h264">h264 (20)</a></li>
419
420 <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (42)</a></li>
421
422 <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (15)</a></li>
423
424 <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (19)</a></li>
425
426 <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
427
428 <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
429
430 <li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
431
432 <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
433
434 <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
435
436 <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (39)</a></li>
437
438 <li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
439
440 <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (285)</a></li>
441
442 <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (182)</a></li>
443
444 <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (26)</a></li>
445
446 <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
447
448 <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (62)</a></li>
449
450 <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (96)</a></li>
451
452 <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
453
454 <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
455
456 <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
457
458 <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (3)</a></li>
459
460 <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (10)</a></li>
461
462 <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
463
464 <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (5)</a></li>
465
466 <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
467
468 <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (52)</a></li>
469
470 <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
471
472 <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (5)</a></li>
473
474 <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (50)</a></li>
475
476 <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (5)</a></li>
477
478 <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (10)</a></li>
479
480 <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (44)</a></li>
481
482 <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (2)</a></li>
483
484 <li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
485
486 <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
487
488 <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
489
490 <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
491
492 <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (38)</a></li>
493
494 </ul>
495
496
497 </div>
498 <p style="text-align: right">
499 Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
500 </p>
501
502 </body>
503 </html>