blog/Automatically_upgrading_server_firmware_on_Dell_PowerEdge.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Automatically upgrading server firmware on Dell PowerEdge</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9   </head>
  10   <body>
  11     <div class="title">
  12  <h1>
  13      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  14
  15  </h1>
  16
  17 </div>
  18
  19
  20     <div class="entry">
  21       <div class="title">Automatically upgrading server firmware on Dell PowerEdge</div>
  22       <div class="date">21st November 2011</div>
  23       <div class="body"><p>At work we have heaps of servers.  I believe the total count is
  24 around 1000 at the moment.  To be able to get help from the vendors
  25 when something go wrong, we want to keep the firmware on the servers
  26 up to date.  If the firmware isn't the latest and greatest, the
  27 vendors typically refuse to start debugging any problems until the
  28 firmware is upgraded.  So before every reboot, we want to upgrade the
  29 firmware, and we would really like everyone handling servers at the
  30 university to do this themselves when they plan to reboot a machine.
  31 For that to happen we at the unix server admin group need to provide
  32 the tools to do so.</p>
  33
  34 <p>To make firmware upgrading easier, I am working on a script to
  35 fetch and install the latest firmware for the servers we got.  Most of
  36 our hardware are from Dell and HP, so I have focused on these servers
  37 so far.  This blog post is about the Dell part.</P>
  38
  39 <p>On the Dell FTP site I was lucky enough to find
  40 <a href="ftp://ftp.us.dell.com/catalog/Catalog.xml.gz">an XML file</a>
  41 with firmware information for all 11th generation servers, listing
  42 which firmware should be used on a given model and where on the FTP
  43 site I can find it.  Using a simple perl XML parser I can then
  44 download the shell scripts Dell provides to do firmware upgrades from
  45 within Linux and reboot when all the firmware is primed and ready to
  46 be activated on the first reboot.</p>
  47
  48 <p>This is the Dell related fragment of the perl code I am working on.
  49 Are there anyone working on similar tools for firmware upgrading all
  50 servers at a site?  Please get in touch and lets share resources.</p>
  51
  52 <p><pre>
  53 #!/usr/bin/perl
  54 use strict;
  55 use warnings;
  56 use File::Temp qw(tempdir);
  57 BEGIN {
  58     # Install needed RHEL packages if missing
  59     my %rhelmodules = (
  60         'XML::Simple' => 'perl-XML-Simple',
  61         );
  62     for my $module (keys %rhelmodules) {
  63         eval "use $module;";
  64         if ($@) {
  65             my $pkg = $rhelmodules{$module};
  66             system("yum install -y $pkg");
  67             eval "use $module;";
  68         }
  69     }
  70 }
  71 my $errorsto = 'pere@hungry.com';
  72
  73 upgrade_dell();
  74
  75 exit 0;
  76
  77 sub run_firmware_script {
  78     my ($opts, $script) = @_;
  79     unless ($script) {
  80         print STDERR "fail: missing script name\n";
  81         exit 1
  82     }
  83     print STDERR "Running $script\n\n";
  84
  85     if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
  86         print STDERR "success: firmware script ran succcessfully\n";
  87     } else {
  88         print STDERR "fail: firmware script returned error\n";
  89     }
  90 }
  91
  92 sub run_firmware_scripts {
  93     my ($opts, @dirs) = @_;
  94     # Run firmware packages
  95     for my $dir (@dirs) {
  96         print STDERR "info: Running scripts in $dir\n";
  97         opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
  98         while (my $s = readdir $dh) {
  99             next if $s =~ m/^\.\.?/;
 100             run_firmware_script($opts, "$dir/$s");
 101         }
 102         closedir $dh;
 103     }
 104 }
 105
 106 sub download {
 107     my $url = shift;
 108     print STDERR "info: Downloading $url\n";
 109     system("wget --quiet \"$url\"");
 110 }
 111
 112 sub upgrade_dell {
 113     my @dirs;
 114     my $product = `dmidecode -s system-product-name`;
 115     chomp $product;
 116
 117     if ($product =~ m/PowerEdge/) {
 118
 119         # on RHEL, these pacakges are needed by the firwmare upgrade scripts
 120         system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');
 121
 122         my $tmpdir = tempdir(
 123             CLEANUP => 1
 124             );
 125         chdir($tmpdir);
 126         fetch_dell_fw('catalog/Catalog.xml.gz');
 127         system('gunzip Catalog.xml.gz');
 128         my @paths = fetch_dell_fw_list('Catalog.xml');
 129         # -q is quiet, disabling interactivity and reducing console output
 130         my $fwopts = "-q";
 131         if (@paths) {
 132             for my $url (@paths) {
 133                 fetch_dell_fw($url);
 134             }
 135             run_firmware_scripts($fwopts, $tmpdir);
 136         } else {
 137             print STDERR "error: Unsupported Dell model '$product'.\n";
 138             print STDERR "error: Please report to $errorsto.\n";
 139         }
 140         chdir('/');
 141     } else {
 142         print STDERR "error: Unsupported Dell model '$product'.\n";
 143         print STDERR "error: Please report to $errorsto.\n";
 144     }
 145 }
 146
 147 sub fetch_dell_fw {
 148     my $path = shift;
 149     my $url = "ftp://ftp.us.dell.com/$path";
 150     download($url);
 151 }
 152
 153 # Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
 154 # firmware packages to download from Dell.  Only work for Linux
 155 # machines and 11th generation Dell servers.
 156 sub fetch_dell_fw_list {
 157     my $filename = shift;
 158
 159     my $product = `dmidecode -s system-product-name`;
 160     chomp $product;
 161     my ($mybrand, $mymodel) = split(/\s+/, $product);
 162
 163     print STDERR "Finding firmware bundles for $mybrand $mymodel\n";
 164
 165     my $xml = XMLin($filename);
 166     my @paths;
 167     for my $bundle (@{$xml->{SoftwareBundle}}) {
 168         my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
 169         my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
 170         my $oscode;
 171         if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
 172             $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
 173         } else {
 174             $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
 175         }
 176         if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
 177         {
 178             @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
 179         }
 180     }
 181     for my $component (@{$xml->{SoftwareComponent}}) {
 182         my $componenttype = $component->{ComponentType}->{value};
 183
 184         # Drop application packages, only firmware and BIOS
 185         next if 'APAC' eq $componenttype;
 186
 187         my $cpath = $component->{path};
 188         for my $path (@paths) {
 189             if ($cpath =~ m%/$path$%) {
 190                 push(@paths, $cpath);
 191             }
 192         }
 193     }
 194     return @paths;
 195 }
 196 </pre>
 197
 198 <p>The code is only tested on RedHat Enterprise Linux, but I suspect
 199 it could work on other platforms with some tweaking.  Anyone know a
 200 index like Catalog.xml is available from HP for HP servers?  At the
 201 moment I maintain a similar list manually and it is quickly getting
 202 outdated.</p>
 203 </div>
 204
 205       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.</div>
 206
 207
 208     </div>
 209
 210
 211
 212
 213     <div id="sidebar">
 214
 215
 216
 217 <h2>Archive</h2>
 218 <ul>
 219
 220 <li>2013
 221 <ul>
 222
 223 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (3)</a></li>
 224
 225 </ul></li>
 226
 227 <li>2012
 228 <ul>
 229
 230 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 231
 232 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 233
 234 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 235
 236 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 237
 238 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 239
 240 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 241
 242 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 243
 244 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 245
 246 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 247
 248 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 249
 250 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 251
 252 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 253
 254 </ul></li>
 255
 256 <li>2011
 257 <ul>
 258
 259 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 260
 261 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 262
 263 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 264
 265 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 266
 267 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 268
 269 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 270
 271 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 272
 273 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 274
 275 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 276
 277 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 278
 279 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 280
 281 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 282
 283 </ul></li>
 284
 285 <li>2010
 286 <ul>
 287
 288 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 289
 290 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 291
 292 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 293
 294 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 295
 296 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 297
 298 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 299
 300 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 301
 302 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 303
 304 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 305
 306 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 307
 308 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 309
 310 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 311
 312 </ul></li>
 313
 314 <li>2009
 315 <ul>
 316
 317 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 318
 319 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 320
 321 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 322
 323 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 324
 325 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 326
 327 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 328
 329 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 330
 331 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 332
 333 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 334
 335 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 336
 337 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 338
 339 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 340
 341 </ul></li>
 342
 343 <li>2008
 344 <ul>
 345
 346 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 347
 348 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 349
 350 </ul></li>
 351
 352 </ul>
 353
 354
 355
 356 <h2>Tags</h2>
 357 <ul>
 358
 359  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 360
 361  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 362
 363  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 364
 365  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 366
 367  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (5)</a></li>
 368
 369  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (12)</a></li>
 370
 371  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 372
 373  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (60)</a></li>
 374
 375  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (118)</a></li>
 376
 377  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (9)</a></li>
 378
 379  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (7)</a></li>
 380
 381  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 382
 383  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (167)</a></li>
 384
 385  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (21)</a></li>
 386
 387  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 388
 389  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (10)</a></li>
 390
 391  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (9)</a></li>
 392
 393  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (32)</a></li>
 394
 395  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (17)</a></li>
 396
 397  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (8)</a></li>
 398
 399  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (6)</a></li>
 400
 401  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 402
 403  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (25)</a></li>
 404
 405  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (219)</a></li>
 406
 407  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (148)</a></li>
 408
 409  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (6)</a></li>
 410
 411  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 412
 413  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (41)</a></li>
 414
 415  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (61)</a></li>
 416
 417  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 418
 419  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 420
 421  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
 422
 423  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (5)</a></li>
 424
 425  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 426
 427  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 428
 429  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 430
 431  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (28)</a></li>
 432
 433  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 434
 435  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 436
 437  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (39)</a></li>
 438
 439  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 440
 441  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (5)</a></li>
 442
 443  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (12)</a></li>
 444
 445  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (1)</a></li>
 446
 447  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (7)</a></li>
 448
 449  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (35)</a></li>
 450
 451  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 452
 453  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (26)</a></li>
 454
 455 </ul>
 456
 457
 458     </div>
 459     <p style="text-align: right">
 460  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.4</a>
 461 </p>
 462
 463   </body>
 464 </html>