blog/data/2011-11-21-fw-upgrade.txt

   1 Title: Automatically upgrading server firmware on Dell PowerEdge
   2 Tags: english, debian
   3 Date: 2011-11-21 12:00
   4
   5 <p>At work we have heaps of servers.  I believe the total count is
   6 around 1000 at the moment.  To be able to get help from the vendors
   7 when something go wrong, we want to keep the firmware on the servers
   8 up to date.  If the firmware isn't the latest and greatest, the
   9 vendors typically refuse to start debugging any problems until the
  10 firmware is upgraded.  So before every reboot, we want to upgrade the
  11 firmware, and we would really like everyone handling servers at the
  12 university to do this themselves when they plan to reboot a machine.
  13 For that to happen we at the unix server admin group need to provide
  14 the tools to do so.</p>
  15
  16 <p>To make firmware upgrading easier, I am working on a script to
  17 fetch and install the latest firmware for the servers we got.  Most of
  18 our hardware are from Dell and HP, so I have focused on these servers
  19 so far.  This blog post is about the Dell part.</P>
  20
  21 <p>On the Dell FTP site I was lucky enough to find
  22 <a href="ftp://ftp.us.dell.com/catalog/Catalog.xml.gz">an XML file</a>
  23 with firmware information for all 11th generation servers, listing
  24 which firmware should be used on a given model and where on the FTP
  25 site I can find it.  Using a simple perl XML parser I can then
  26 download the shell scripts Dell provides to do firmware upgrades from
  27 within Linux and reboot when all the firmware is primed and ready to
  28 be activated on the first reboot.</p>
  29
  30 <p>This is the Dell related fragment of the perl code I am working on.
  31 Are there anyone working on similar tools for firmware upgrading all
  32 servers at a site?  Please get in touch and lets share resources.</p>
  33
  34 <p><pre>
  35 #!/usr/bin/perl
  36 use strict;
  37 use warnings;
  38 use File::Temp qw(tempdir);
  39 BEGIN {
  40     # Install needed RHEL packages if missing
  41     my %rhelmodules = (
  42         'XML::Simple' => 'perl-XML-Simple',
  43         );
  44     for my $module (keys %rhelmodules) {
  45         eval "use $module;";
  46         if ($@) {
  47             my $pkg = $rhelmodules{$module};
  48             system("yum install -y $pkg");
  49             eval "use $module;";
  50         }
  51     }
  52 }
  53 my $errorsto = 'pere@hungry.com';
  54
  55 upgrade_dell();
  56
  57 exit 0;
  58
  59 sub run_firmware_script {
  60     my ($opts, $script) = @_;
  61     unless ($script) {
  62         print STDERR "fail: missing script name\n";
  63         exit 1
  64     }
  65     print STDERR "Running $script\n\n";
  66
  67     if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
  68         print STDERR "success: firmware script ran succcessfully\n";
  69     } else {
  70         print STDERR "fail: firmware script returned error\n";
  71     }
  72 }
  73
  74 sub run_firmware_scripts {
  75     my ($opts, @dirs) = @_;
  76     # Run firmware packages
  77     for my $dir (@dirs) {
  78         print STDERR "info: Running scripts in $dir\n";
  79         opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
  80         while (my $s = readdir $dh) {
  81             next if $s =~ m/^\.\.?/;
  82             run_firmware_script($opts, "$dir/$s");
  83         }
  84         closedir $dh;
  85     }
  86 }
  87
  88 sub download {
  89     my $url = shift;
  90     print STDERR "info: Downloading $url\n";
  91     system("wget --quiet \"$url\"");
  92 }
  93
  94 sub upgrade_dell {
  95     my @dirs;
  96     my $product = `dmidecode -s system-product-name`;
  97     chomp $product;
  98
  99     if ($product =~ m/PowerEdge/) {
 100
 101         # on RHEL, these pacakges are needed by the firwmare upgrade scripts
 102         system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');
 103
 104         my $tmpdir = tempdir(
 105             CLEANUP => 1
 106             );
 107         chdir($tmpdir);
 108         fetch_dell_fw('catalog/Catalog.xml.gz');
 109         system('gunzip Catalog.xml.gz');
 110         my @paths = fetch_dell_fw_list('Catalog.xml');
 111         # -q is quiet, disabling interactivity and reducing console output
 112         my $fwopts = "-q";
 113         if (@paths) {
 114             for my $url (@paths) {
 115                 fetch_dell_fw($url);
 116             }
 117             run_firmware_scripts($fwopts, $tmpdir);
 118         } else {
 119             print STDERR "error: Unsupported Dell model '$product'.\n";
 120             print STDERR "error: Please report to $errorsto.\n";
 121         }
 122         chdir('/');
 123     } else {
 124         print STDERR "error: Unsupported Dell model '$product'.\n";
 125         print STDERR "error: Please report to $errorsto.\n";
 126     }
 127 }
 128
 129 sub fetch_dell_fw {
 130     my $path = shift;
 131     my $url = "ftp://ftp.us.dell.com/$path";
 132     download($url);
 133 }
 134
 135 # Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
 136 # firmware packages to download from Dell.  Only work for Linux
 137 # machines and 11th generation Dell servers.
 138 sub fetch_dell_fw_list {
 139     my $filename = shift;
 140
 141     my $product = `dmidecode -s system-product-name`;
 142     chomp $product;
 143     my ($mybrand, $mymodel) = split(/\s+/, $product);
 144
 145     print STDERR "Finding firmware bundles for $mybrand $mymodel\n";
 146
 147     my $xml = XMLin($filename);
 148     my @paths;
 149     for my $bundle (@{$xml->{SoftwareBundle}}) {
 150         my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
 151         my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
 152         my $oscode;
 153         if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
 154             $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
 155         } else {
 156             $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
 157         }
 158         if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
 159         {
 160             @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
 161         }
 162     }
 163     for my $component (@{$xml->{SoftwareComponent}}) {
 164         my $componenttype = $component->{ComponentType}->{value};
 165
 166         # Drop application packages, only firmware and BIOS
 167         next if 'APAC' eq $componenttype;
 168
 169         my $cpath = $component->{path};
 170         for my $path (@paths) {
 171             if ($cpath =~ m%/$path$%) {
 172                 push(@paths, $cpath);
 173             }
 174         }
 175     }
 176     return @paths;
 177 }
 178 </pre>
 179
 180 <p>The code is only tested on RedHat Enterprise Linux, but I suspect
 181 it could work on other platforms with some tweaking.  Anyone know a
 182 index like Catalog.xml is available from HP for HP servers?  At the
 183 moment I maintain a similar list manually and it is quickly getting
 184 outdated.</p>