blog/Automatically_upgrading_server_firmware_on_Dell_PowerEdge.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Automatically upgrading server firmware on Dell PowerEdge</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9   </head>
  10   <body>
  11     <div class="title">
  12  <h1>
  13      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  14
  15  </h1>
  16
  17 </div>
  18
  19
  20     <div class="entry">
  21       <div class="title">Automatically upgrading server firmware on Dell PowerEdge</div>
  22       <div class="date">21st November 2011</div>
  23       <div class="body"><p>At work we have heaps of servers.  I believe the total count is
  24 around 1000 at the moment.  To be able to get help from the vendors
  25 when something go wrong, we want to keep the firmware on the servers
  26 up to date.  If the firmware isn't the latest and greatest, the
  27 vendors typically refuse to start debugging any problems until the
  28 firmware is upgraded.  So before every reboot, we want to upgrade the
  29 firmware, and we would really like everyone handling servers at the
  30 university to do this themselves when they plan to reboot a machine.
  31 For that to happen we at the unix server admin group need to provide
  32 the tools to do so.</p>
  33
  34 <p>To make firmware upgrading easier, I am working on a script to
  35 fetch and install the latest firmware for the servers we got.  Most of
  36 our hardware are from Dell and HP, so I have focused on these servers
  37 so far.  This blog post is about the Dell part.</P>
  38
  39 <p>On the Dell FTP site I was lucky enough to find
  40 <a href="ftp://ftp.us.dell.com/catalog/Catalog.xml.gz">an XML file</a>
  41 with firmware information for all 11th generation servers, listing
  42 which firmware should be used on a given model and where on the FTP
  43 site I can find it.  Using a simple perl XML parser I can then
  44 download the shell scripts Dell provides to do firmware upgrades from
  45 within Linux and reboot when all the firmware is primed and ready to
  46 be activated on the first reboot.</p>
  47
  48 <p>This is the Dell related fragment of the perl code I am working on.
  49 Are there anyone working on similar tools for firmware upgrading all
  50 servers at a site?  Please get in touch and lets share resources.</p>
  51
  52 <p><pre>
  53 #!/usr/bin/perl
  54 use strict;
  55 use warnings;
  56 use File::Temp qw(tempdir);
  57 BEGIN {
  58     # Install needed RHEL packages if missing
  59     my %rhelmodules = (
  60         'XML::Simple' => 'perl-XML-Simple',
  61         );
  62     for my $module (keys %rhelmodules) {
  63         eval "use $module;";
  64         if ($@) {
  65             my $pkg = $rhelmodules{$module};
  66             system("yum install -y $pkg");
  67             eval "use $module;";
  68         }
  69     }
  70 }
  71 my $errorsto = 'pere@hungry.com';
  72
  73 upgrade_dell();
  74
  75 exit 0;
  76
  77 sub run_firmware_script {
  78     my ($opts, $script) = @_;
  79     unless ($script) {
  80         print STDERR "fail: missing script name\n";
  81         exit 1
  82     }
  83     print STDERR "Running $script\n\n";
  84
  85     if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
  86         print STDERR "success: firmware script ran succcessfully\n";
  87     } else {
  88         print STDERR "fail: firmware script returned error\n";
  89     }
  90 }
  91
  92 sub run_firmware_scripts {
  93     my ($opts, @dirs) = @_;
  94     # Run firmware packages
  95     for my $dir (@dirs) {
  96         print STDERR "info: Running scripts in $dir\n";
  97         opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
  98         while (my $s = readdir $dh) {
  99             next if $s =~ m/^\.\.?/;
 100             run_firmware_script($opts, "$dir/$s");
 101         }
 102         closedir $dh;
 103     }
 104 }
 105
 106 sub download {
 107     my $url = shift;
 108     print STDERR "info: Downloading $url\n";
 109     system("wget --quiet \"$url\"");
 110 }
 111
 112 sub upgrade_dell {
 113     my @dirs;
 114     my $product = `dmidecode -s system-product-name`;
 115     chomp $product;
 116
 117     if ($product =~ m/PowerEdge/) {
 118
 119         # on RHEL, these pacakges are needed by the firwmare upgrade scripts
 120         system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');
 121
 122         my $tmpdir = tempdir(
 123             CLEANUP => 1
 124             );
 125         chdir($tmpdir);
 126         fetch_dell_fw('catalog/Catalog.xml.gz');
 127         system('gunzip Catalog.xml.gz');
 128         my @paths = fetch_dell_fw_list('Catalog.xml');
 129         # -q is quiet, disabling interactivity and reducing console output
 130         my $fwopts = "-q";
 131         if (@paths) {
 132             for my $url (@paths) {
 133                 fetch_dell_fw($url);
 134             }
 135             run_firmware_scripts($fwopts, $tmpdir);
 136         } else {
 137             print STDERR "error: Unsupported Dell model '$product'.\n";
 138             print STDERR "error: Please report to $errorsto.\n";
 139         }
 140         chdir('/');
 141     } else {
 142         print STDERR "error: Unsupported Dell model '$product'.\n";
 143         print STDERR "error: Please report to $errorsto.\n";
 144     }
 145 }
 146
 147 sub fetch_dell_fw {
 148     my $path = shift;
 149     my $url = "ftp://ftp.us.dell.com/$path";
 150     download($url);
 151 }
 152
 153 # Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
 154 # firmware packages to download from Dell.  Only work for Linux
 155 # machines and 11th generation Dell servers.
 156 sub fetch_dell_fw_list {
 157     my $filename = shift;
 158
 159     my $product = `dmidecode -s system-product-name`;
 160     chomp $product;
 161     my ($mybrand, $mymodel) = split(/\s+/, $product);
 162
 163     print STDERR "Finding firmware bundles for $mybrand $mymodel\n";
 164
 165     my $xml = XMLin($filename);
 166     my @paths;
 167     for my $bundle (@{$xml->{SoftwareBundle}}) {
 168         my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
 169         my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
 170         my $oscode;
 171         if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
 172             $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
 173         } else {
 174             $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
 175         }
 176         if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
 177         {
 178             @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
 179         }
 180     }
 181     for my $component (@{$xml->{SoftwareComponent}}) {
 182         my $componenttype = $component->{ComponentType}->{value};
 183
 184         # Drop application packages, only firmware and BIOS
 185         next if 'APAC' eq $componenttype;
 186
 187         my $cpath = $component->{path};
 188         for my $path (@paths) {
 189             if ($cpath =~ m%/$path$%) {
 190                 push(@paths, $cpath);
 191             }
 192         }
 193     }
 194     return @paths;
 195 }
 196 </pre>
 197
 198 <p>The code is only tested on RedHat Enterprise Linux, but I suspect
 199 it could work on other platforms with some tweaking.  Anyone know a
 200 index like Catalog.xml is available from HP for HP servers?  At the
 201 moment I maintain a similar list manually and it is quickly getting
 202 outdated.</p>
 203 </div>
 204
 205       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.</div>
 206
 207
 208     </div>
 209
 210
 211
 212
 213     <div id="sidebar">
 214
 215
 216
 217 <h2>Archive</h2>
 218 <ul>
 219
 220 <li>2012
 221 <ul>
 222
 223 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 224
 225 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 226
 227 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 228
 229 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 230
 231 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 232
 233 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 234
 235 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 236
 237 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 238
 239 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 240
 241 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 242
 243 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (9)</a></li>
 244
 245 </ul></li>
 246
 247 <li>2011
 248 <ul>
 249
 250 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 251
 252 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 253
 254 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 255
 256 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 257
 258 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 259
 260 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 261
 262 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 263
 264 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 265
 266 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 267
 268 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 269
 270 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 271
 272 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 273
 274 </ul></li>
 275
 276 <li>2010
 277 <ul>
 278
 279 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 280
 281 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 282
 283 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 284
 285 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 286
 287 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 288
 289 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 290
 291 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 292
 293 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 294
 295 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 296
 297 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 298
 299 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 300
 301 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 302
 303 </ul></li>
 304
 305 <li>2009
 306 <ul>
 307
 308 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 309
 310 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 311
 312 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 313
 314 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 315
 316 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 317
 318 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 319
 320 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 321
 322 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 323
 324 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 325
 326 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 327
 328 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 329
 330 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 331
 332 </ul></li>
 333
 334 <li>2008
 335 <ul>
 336
 337 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 338
 339 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 340
 341 </ul></li>
 342
 343 </ul>
 344
 345
 346
 347 <h2>Tags</h2>
 348 <ul>
 349
 350  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 351
 352  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 353
 354  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 355
 356  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (3)</a></li>
 357
 358  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (12)</a></li>
 359
 360  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 361
 362  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (57)</a></li>
 363
 364  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (116)</a></li>
 365
 366  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (9)</a></li>
 367
 368  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (7)</a></li>
 369
 370  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 371
 372  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (160)</a></li>
 373
 374  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (20)</a></li>
 375
 376  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 377
 378  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (9)</a></li>
 379
 380  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (9)</a></li>
 381
 382  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (32)</a></li>
 383
 384  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (17)</a></li>
 385
 386  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (8)</a></li>
 387
 388  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (4)</a></li>
 389
 390  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 391
 392  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (25)</a></li>
 393
 394  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (215)</a></li>
 395
 396  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (147)</a></li>
 397
 398  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (6)</a></li>
 399
 400  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 401
 402  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (38)</a></li>
 403
 404  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (59)</a></li>
 405
 406  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 407
 408  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 409
 410  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
 411
 412  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (4)</a></li>
 413
 414  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 415
 416  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 417
 418  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 419
 420  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (28)</a></li>
 421
 422  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 423
 424  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 425
 426  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (39)</a></li>
 427
 428  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 429
 430  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (5)</a></li>
 431
 432  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (12)</a></li>
 433
 434  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (7)</a></li>
 435
 436  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (35)</a></li>
 437
 438  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 439
 440  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (26)</a></li>
 441
 442 </ul>
 443
 444
 445     </div>
 446     <p style="text-align: right">
 447  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.4</a>
 448 </p>
 449
 450   </body>
 451 </html>