blog/Automatically_upgrading_server_firmware_on_Dell_PowerEdge.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   2           "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   3 <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
   4   <head>
   5     <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
   6     <title>Petter Reinholdtsen: Automatically upgrading server firmware on Dell PowerEdge</title>
   7     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
   8     <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
   9
  10
  11   </head>
  12   <body>
  13     <div class="title">
  14  <h1>
  15      <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
  16
  17  </h1>
  18
  19 </div>
  20
  21
  22     <div class="entry">
  23       <div class="title">Automatically upgrading server firmware on Dell PowerEdge</div>
  24       <div class="date">21st November 2011</div>
  25       <div class="body"><p>At work we have heaps of servers.  I believe the total count is
  26 around 1000 at the moment.  To be able to get help from the vendors
  27 when something go wrong, we want to keep the firmware on the servers
  28 up to date.  If the firmware isn't the latest and greatest, the
  29 vendors typically refuse to start debugging any problems until the
  30 firmware is upgraded.  So before every reboot, we want to upgrade the
  31 firmware, and we would really like everyone handling servers at the
  32 university to do this themselves when they plan to reboot a machine.
  33 For that to happen we at the unix server admin group need to provide
  34 the tools to do so.</p>
  35
  36 <p>To make firmware upgrading easier, I am working on a script to
  37 fetch and install the latest firmware for the servers we got.  Most of
  38 our hardware are from Dell and HP, so I have focused on these servers
  39 so far.  This blog post is about the Dell part.</P>
  40
  41 <p>On the Dell FTP site I was lucky enough to find
  42 <a href="ftp://ftp.us.dell.com/catalog/Catalog.xml.gz">an XML file</a>
  43 with firmware information for all 11th generation servers, listing
  44 which firmware should be used on a given model and where on the FTP
  45 site I can find it.  Using a simple perl XML parser I can then
  46 download the shell scripts Dell provides to do firmware upgrades from
  47 within Linux and reboot when all the firmware is primed and ready to
  48 be activated on the first reboot.</p>
  49
  50 <p>This is the Dell related fragment of the perl code I am working on.
  51 Are there anyone working on similar tools for firmware upgrading all
  52 servers at a site?  Please get in touch and lets share resources.</p>
  53
  54 <p><pre>
  55 #!/usr/bin/perl
  56 use strict;
  57 use warnings;
  58 use File::Temp qw(tempdir);
  59 BEGIN {
  60     # Install needed RHEL packages if missing
  61     my %rhelmodules = (
  62         'XML::Simple' => 'perl-XML-Simple',
  63         );
  64     for my $module (keys %rhelmodules) {
  65         eval "use $module;";
  66         if ($@) {
  67             my $pkg = $rhelmodules{$module};
  68             system("yum install -y $pkg");
  69             eval "use $module;";
  70         }
  71     }
  72 }
  73 my $errorsto = 'pere@hungry.com';
  74
  75 upgrade_dell();
  76
  77 exit 0;
  78
  79 sub run_firmware_script {
  80     my ($opts, $script) = @_;
  81     unless ($script) {
  82         print STDERR "fail: missing script name\n";
  83         exit 1
  84     }
  85     print STDERR "Running $script\n\n";
  86
  87     if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
  88         print STDERR "success: firmware script ran succcessfully\n";
  89     } else {
  90         print STDERR "fail: firmware script returned error\n";
  91     }
  92 }
  93
  94 sub run_firmware_scripts {
  95     my ($opts, @dirs) = @_;
  96     # Run firmware packages
  97     for my $dir (@dirs) {
  98         print STDERR "info: Running scripts in $dir\n";
  99         opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
 100         while (my $s = readdir $dh) {
 101             next if $s =~ m/^\.\.?/;
 102             run_firmware_script($opts, "$dir/$s");
 103         }
 104         closedir $dh;
 105     }
 106 }
 107
 108 sub download {
 109     my $url = shift;
 110     print STDERR "info: Downloading $url\n";
 111     system("wget --quiet \"$url\"");
 112 }
 113
 114 sub upgrade_dell {
 115     my @dirs;
 116     my $product = `dmidecode -s system-product-name`;
 117     chomp $product;
 118
 119     if ($product =~ m/PowerEdge/) {
 120
 121         # on RHEL, these pacakges are needed by the firwmare upgrade scripts
 122         system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');
 123
 124         my $tmpdir = tempdir(
 125             CLEANUP => 1
 126             );
 127         chdir($tmpdir);
 128         fetch_dell_fw('catalog/Catalog.xml.gz');
 129         system('gunzip Catalog.xml.gz');
 130         my @paths = fetch_dell_fw_list('Catalog.xml');
 131         # -q is quiet, disabling interactivity and reducing console output
 132         my $fwopts = "-q";
 133         if (@paths) {
 134             for my $url (@paths) {
 135                 fetch_dell_fw($url);
 136             }
 137             run_firmware_scripts($fwopts, $tmpdir);
 138         } else {
 139             print STDERR "error: Unsupported Dell model '$product'.\n";
 140             print STDERR "error: Please report to $errorsto.\n";
 141         }
 142         chdir('/');
 143     } else {
 144         print STDERR "error: Unsupported Dell model '$product'.\n";
 145         print STDERR "error: Please report to $errorsto.\n";
 146     }
 147 }
 148
 149 sub fetch_dell_fw {
 150     my $path = shift;
 151     my $url = "ftp://ftp.us.dell.com/$path";
 152     download($url);
 153 }
 154
 155 # Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
 156 # firmware packages to download from Dell.  Only work for Linux
 157 # machines and 11th generation Dell servers.
 158 sub fetch_dell_fw_list {
 159     my $filename = shift;
 160
 161     my $product = `dmidecode -s system-product-name`;
 162     chomp $product;
 163     my ($mybrand, $mymodel) = split(/\s+/, $product);
 164
 165     print STDERR "Finding firmware bundles for $mybrand $mymodel\n";
 166
 167     my $xml = XMLin($filename);
 168     my @paths;
 169     for my $bundle (@{$xml->{SoftwareBundle}}) {
 170         my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
 171         my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
 172         my $oscode;
 173         if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
 174             $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
 175         } else {
 176             $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
 177         }
 178         if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
 179         {
 180             @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
 181         }
 182     }
 183     for my $component (@{$xml->{SoftwareComponent}}) {
 184         my $componenttype = $component->{ComponentType}->{value};
 185
 186         # Drop application packages, only firmware and BIOS
 187         next if 'APAC' eq $componenttype;
 188
 189         my $cpath = $component->{path};
 190         for my $path (@paths) {
 191             if ($cpath =~ m%/$path$%) {
 192                 push(@paths, $cpath);
 193             }
 194         }
 195     }
 196     return @paths;
 197 }
 198 </pre>
 199
 200 <p>The code is only tested on RedHat Enterprise Linux, but I suspect
 201 it could work on other platforms with some tweaking.  Anyone know a
 202 index like Catalog.xml is available from HP for HP servers?  At the
 203 moment I maintain a similar list manually and it is quickly getting
 204 outdated.</p>
 205 </div>
 206
 207       <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.</div>
 208
 209
 210     </div>
 211
 212
 213
 214
 215     <div id="sidebar">
 216
 217
 218
 219 <h2>Archive</h2>
 220 <ul>
 221
 222 <li>2013
 223 <ul>
 224
 225 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
 226
 227 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
 228
 229 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
 230
 231 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
 232
 233 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
 234
 235 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
 236
 237 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
 238
 239 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
 240
 241 <li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (2)</a></li>
 242
 243 </ul></li>
 244
 245 <li>2012
 246 <ul>
 247
 248 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
 249
 250 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
 251
 252 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
 253
 254 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
 255
 256 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
 257
 258 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
 259
 260 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
 261
 262 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
 263
 264 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
 265
 266 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
 267
 268 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
 269
 270 <li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
 271
 272 </ul></li>
 273
 274 <li>2011
 275 <ul>
 276
 277 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
 278
 279 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
 280
 281 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
 282
 283 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
 284
 285 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
 286
 287 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
 288
 289 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
 290
 291 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
 292
 293 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
 294
 295 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
 296
 297 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
 298
 299 <li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
 300
 301 </ul></li>
 302
 303 <li>2010
 304 <ul>
 305
 306 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
 307
 308 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
 309
 310 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
 311
 312 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
 313
 314 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
 315
 316 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
 317
 318 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
 319
 320 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
 321
 322 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
 323
 324 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
 325
 326 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
 327
 328 <li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
 329
 330 </ul></li>
 331
 332 <li>2009
 333 <ul>
 334
 335 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
 336
 337 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
 338
 339 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
 340
 341 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
 342
 343 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
 344
 345 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
 346
 347 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
 348
 349 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
 350
 351 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
 352
 353 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
 354
 355 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
 356
 357 <li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
 358
 359 </ul></li>
 360
 361 <li>2008
 362 <ul>
 363
 364 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
 365
 366 <li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
 367
 368 </ul></li>
 369
 370 </ul>
 371
 372
 373
 374 <h2>Tags</h2>
 375 <ul>
 376
 377  <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
 378
 379  <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
 380
 381  <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
 382
 383  <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
 384
 385  <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (7)</a></li>
 386
 387  <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (12)</a></li>
 388
 389  <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
 390
 391  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (85)</a></li>
 392
 393  <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (139)</a></li>
 394
 395  <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
 396
 397  <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (10)</a></li>
 398
 399  <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
 400
 401  <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (214)</a></li>
 402
 403  <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (21)</a></li>
 404
 405  <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
 406
 407  <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (12)</a></li>
 408
 409  <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (1)</a></li>
 410
 411  <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (11)</a></li>
 412
 413  <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (37)</a></li>
 414
 415  <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (7)</a></li>
 416
 417  <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (18)</a></li>
 418
 419  <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (8)</a></li>
 420
 421  <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (6)</a></li>
 422
 423  <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
 424
 425  <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (25)</a></li>
 426
 427  <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (235)</a></li>
 428
 429  <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (153)</a></li>
 430
 431  <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (8)</a></li>
 432
 433  <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
 434
 435  <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (44)</a></li>
 436
 437  <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (66)</a></li>
 438
 439  <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
 440
 441  <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
 442
 443  <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
 444
 445  <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (7)</a></li>
 446
 447  <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
 448
 449  <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
 450
 451  <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
 452
 453  <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (30)</a></li>
 454
 455  <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
 456
 457  <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
 458
 459  <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (43)</a></li>
 460
 461  <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
 462
 463  <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (8)</a></li>
 464
 465  <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (17)</a></li>
 466
 467  <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (1)</a></li>
 468
 469  <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
 470
 471  <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (38)</a></li>
 472
 473  <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
 474
 475  <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (27)</a></li>
 476
 477 </ul>
 478
 479
 480     </div>
 481     <p style="text-align: right">
 482  Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
 483 </p>
 484
 485   </body>
 486 </html>