+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
+ <title>Petter Reinholdtsen: 12 years of outages - summarised by Stuart Kendrick</title>
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
+ </head>
+ <body>
+ <div class="title">
+ <h1>
+ <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
+
+ </h1>
+
+</div>
+
+
+ <div class="entry">
+ <div class="title">12 years of outages - summarised by Stuart Kendrick</div>
+ <div class="date">26th October 2012</div>
+ <div class="body"><p>I work at the <a href="http://www.uio.no/">University of Oslo</a>
+looking after the computers, mostly on the unix side, but in general
+all over the place. I am also a member (and currently leader) of
+<a href="http://www.nuug.no/">the NUUG association</a>, which in turn
+make me a member of <a href="http://www.usenix.org/">USENIX</a>. NUUG
+is an member organisation for us in Norway interested in free
+software, open standards and unix like operating systems, and USENIX
+is a US based member organisation with similar targets. And thanks to
+these memberships, I get all issues of the great USENIX magazine
+<a href="https://www.usenix.org/publications/login">;login:</a> in the
+mail several times a year. The magazine is great, and I read most of
+it every time.</p>
+
+<p>In the last issue of the USENIX magazine ;login:, there is an
+article by <a href="http://www.skendric.com/">Stuart Kendrick</a> from
+Fred Hutchinson Cancer Research Center titled
+"<a href="https://www.usenix.org/publications/login/october-2012-volume-37-number-5/what-takes-us-down">What
+Takes Us Down</a>" (also
+<a href="http://www.skendric.com/problem/incident-analysis/2012-06-30/What-Takes-Us-Down.pdf">available
+from his own site</a>), where he report what he found when he
+processed the outage reports (both planned and unplanned) from the
+last twelve years and classified them according to cause, time of day,
+etc etc. The article is a good read to get some empirical data on
+what kind of problems affect a data centre, but what really inspired
+me was the kind of reporting they had put in place since 2000.<p>
+
+<p>The centre set up a mailing list, and started to send fairly
+standardised messages to this list when a outage was planned or when
+it already occurred, to announce the plan and get feedback on the
+assumtions on scope and user impact. Here is the two example from the
+article: First the unplanned outage:
+
+<blockquote><pre>
+Subject: Exchange 2003 Cluster Issues
+Severity: Critical (Unplanned)
+Start: Monday, May 7, 2012, 11:58
+End: Monday, May 7, 2012, 12:38
+Duration: 40 minutes
+Scope: Exchange 2003
+Description: The HTTPS service on the Exchange cluster crashed, triggering
+ a cluster failover.
+
+User Impact: During this period, all Exchange users were unable to
+ access e-mail. Zimbra users were unaffected.
+Technician: [xxx]
+</pre></blockquote>
+
+Next the planned outage:
+
+<blockquote><pre>
+Subject: H Building Switch Upgrades
+Severity: Major (Planned)
+Start: Saturday, June 16, 2012, 06:00
+End: Saturday, June 16, 2012, 16:00
+Duration: 10 hours
+Scope: H2 Transport
+Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
+ stations. We will replace these with newer Catalyst
+ 4510s.
+User Impact: All users on H2 will be isolated from the network during
+ this work. Afterward, they will have gigabit
+ connectivity.
+Technician: [xxx]
+</pre></blockquote>
+
+<p>He notes in his article that the date formats and other fields have
+been a bit too free form to make it easy to automatically process them
+into a database for further analysis, and I would have used ISO 8601
+dates myself to make it easier to process (in other words I would ask
+people to write '2012-06-16 06:00 +0000' instead of the start time
+format listed above). There are also other issues with the format
+that could be improved, read the article for the details.</p>
+
+<p>I find the idea of standardising outage messages seem to be such a
+good idea that I would like to get it implemented here at the
+university too. We do register
+<a href="http://www.uio.no/tjenester/it/aktuelt/planlagte-tjenesteavbrudd/">planned
+changes and outages in a calendar</a>, and report the to a mailing
+list, but we do not do so in a structured format and there is not a
+report to the same location for unplanned outages. Perhaps something
+for other sites to consider too?</p>
+</div>
+
+ <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.</div>
+
+
+ </div>
+
+
+
+
+ <div id="sidebar">
+
+
+
+<h2>Archive</h2>
+<ul>
+
+<li>2012
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
+
+</ul></li>
+
+<li>2011
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
+
+</ul></li>
+
+<li>2010
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
+
+</ul></li>
+
+<li>2009
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2008
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
+
+</ul></li>
+
+</ul>
+
+
+
+<h2>Tags</h2>
+<ul>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (12)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (57)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (115)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (7)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (157)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (19)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (31)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (17)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (25)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (209)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (146)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (6)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (37)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (54)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (24)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (39)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (5)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (7)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (35)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (26)</a></li>
+
+</ul>
+
+
+ </div>
+ <p style="text-align: right">
+ Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.4</a>
+</p>
+
+ </body>
+</html>
projects / homepage.git / commitdiff