At work we have heaps of servers. I believe the total count is +around 1000 at the moment. To be able to get help from the vendors +when something go wrong, we want to keep the firmware on the servers +up to date. If the firmware isn't the latest and greatest, the +vendors typically refuse to start debugging any problems until the +firmware is upgraded. So before every reboot, we want to upgrade the +firmware, and we would really like everyone handling servers at the +university to do this themselves when they plan to reboot a machine. +For that to happen we at the unix server admin group need to provide +the tools to do so.
+ +To make firmware upgrading easier, I am working on a script to +fetch and install the latest firmware for the servers we got. Most of +our hardware are from Dell and HP, so I have focused on these servers +so far. This blog post is about the Dell part.
+ +On the Dell FTP site I was lucky enough to find +an XML file +with firmware information for all 11th generation servers, listing +which firmware should be used on a given model and where on the FTP +site I can find it. Using a simple perl XML parser I can then +download the shell scripts Dell provides to do firmware upgrades from +within Linux and reboot when all the firmware is primed and ready to +be activated on the first reboot.
+ +This is the Dell related fragment of the perl code I am working on. +Are there anyone working on similar tools for firmware upgrading all +servers at a site? Please get in touch and lets share resources.
+ ++#!/usr/bin/perl +use strict; +use warnings; +use File::Temp qw(tempdir); +BEGIN { + # Install needed RHEL packages if missing + my %rhelmodules = ( + 'XML::Simple' => 'perl-XML-Simple', + ); + for my $module (keys %rhelmodules) { + eval "use $module;"; + if ($@) { + my $pkg = $rhelmodules{$module}; + system("yum install -y $pkg"); + eval "use $module;"; + } + } +} +my $errorsto = 'pere@hungry.com'; + +upgrade_dell(); + +exit 0; + +sub run_firmware_script { + my ($opts, $script) = @_; + unless ($script) { + print STDERR "fail: missing script name\n"; + exit 1 + } + print STDERR "Running $script\n\n"; + + if (0 == system("sh $script $opts")) { # FIXME correct exit code handling + print STDERR "success: firmware script ran succcessfully\n"; + } else { + print STDERR "fail: firmware script returned error\n"; + } +} + +sub run_firmware_scripts { + my ($opts, @dirs) = @_; + # Run firmware packages + for my $dir (@dirs) { + print STDERR "info: Running scripts in $dir\n"; + opendir(my $dh, $dir) or die "Unable to open directory $dir: $!"; + while (my $s = readdir $dh) { + next if $s =~ m/^\.\.?/; + run_firmware_script($opts, "$dir/$s"); + } + closedir $dh; + } +} + +sub download { + my $url = shift; + print STDERR "info: Downloading $url\n"; + system("wget --quiet \"$url\""); +} + +sub upgrade_dell { + my @dirs; + my $product = `dmidecode -s system-product-name`; + chomp $product; + + if ($product =~ m/PowerEdge/) { + + # on RHEL, these pacakges are needed by the firwmare upgrade scripts + system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail'); + + my $tmpdir = tempdir( + CLEANUP => 1 + ); + chdir($tmpdir); + fetch_dell_fw('catalog/Catalog.xml.gz'); + system('gunzip Catalog.xml.gz'); + my @paths = fetch_dell_fw_list('Catalog.xml'); + # -q is quiet, disabling interactivity and reducing console output + my $fwopts = "-q"; + if (@paths) { + for my $url (@paths) { + fetch_dell_fw($url); + } + run_firmware_scripts($fwopts, $tmpdir); + } else { + print STDERR "error: Unsupported Dell model '$product'.\n"; + print STDERR "error: Please report to $errorsto.\n"; + } + chdir('/'); + } else { + print STDERR "error: Unsupported Dell model '$product'.\n"; + print STDERR "error: Please report to $errorsto.\n"; + } +} + +sub fetch_dell_fw { + my $path = shift; + my $url = "ftp://ftp.us.dell.com/$path"; + download($url); +} + +# Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which +# firmware packages to download from Dell. Only work for Linux +# machines and 11th generation Dell servers. +sub fetch_dell_fw_list { + my $filename = shift; + + my $product = `dmidecode -s system-product-name`; + chomp $product; + my ($mybrand, $mymodel) = split(/\s+/, $product); + + print STDERR "Finding firmware bundles for $mybrand $mymodel\n"; + + my $xml = XMLin($filename); + my @paths; + for my $bundle (@{$xml->{SoftwareBundle}}) { + my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content}; + my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content}; + my $oscode; + if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) { + $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode}; + } else { + $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode}; + } + if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode) + { + @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}}; + } + } + for my $component (@{$xml->{SoftwareComponent}}) { + my $componenttype = $component->{ComponentType}->{value}; + + # Drop application packages, only firmware and BIOS + next if 'APAC' eq $componenttype; + + my $cpath = $component->{path}; + for my $path (@paths) { + if ($cpath =~ m%/$path$%) { + push(@paths, $cpath); + } + } + } + return @paths; +} ++ +
The code is only tested on RedHat Enterprise Linux, but I suspect +it could work on other platforms with some tweaking. Anyone know a +index like Catalog.xml is available from HP for HP servers? At the +moment I maintain a similar list manually and it is quickly getting +outdated.
+