At work we have heaps of servers. I believe the total count is
around 1000 at the moment. To be able to get help from the vendors
when something go wrong, we want to keep the firmware on the servers
up to date. If the firmware isn't the latest and greatest, the
vendors typically refuse to start debugging any problems until the
firmware is upgraded. So before every reboot, we want to upgrade the
firmware, and we would really like everyone handling servers at the
university to do this themselves when they plan to reboot a machine.
For that to happen we at the unix server admin group need to provide
the tools to do so.
To make firmware upgrading easier, I am working on a script to
fetch and install the latest firmware for the servers we got. Most of
our hardware are from Dell and HP, so I have focused on these servers
so far. This blog post is about the Dell part.
On the Dell FTP site I was lucky enough to find
an XML file
with firmware information for all 11th generation servers, listing
which firmware should be used on a given model and where on the FTP
site I can find it. Using a simple perl XML parser I can then
download the shell scripts Dell provides to do firmware upgrades from
within Linux and reboot when all the firmware is primed and ready to
be activated on the first reboot.
This is the Dell related fragment of the perl code I am working on.
Are there anyone working on similar tools for firmware upgrading all
servers at a site? Please get in touch and lets share resources.
#!/usr/bin/perl
use strict;
use warnings;
use File::Temp qw(tempdir);
BEGIN {
# Install needed RHEL packages if missing
my %rhelmodules = (
'XML::Simple' => 'perl-XML-Simple',
);
for my $module (keys %rhelmodules) {
eval "use $module;";
if ($@) {
my $pkg = $rhelmodules{$module};
system("yum install -y $pkg");
eval "use $module;";
}
}
}
my $errorsto = 'pere@hungry.com';
upgrade_dell();
exit 0;
sub run_firmware_script {
my ($opts, $script) = @_;
unless ($script) {
print STDERR "fail: missing script name\n";
exit 1
}
print STDERR "Running $script\n\n";
if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
print STDERR "success: firmware script ran succcessfully\n";
} else {
print STDERR "fail: firmware script returned error\n";
}
}
sub run_firmware_scripts {
my ($opts, @dirs) = @_;
# Run firmware packages
for my $dir (@dirs) {
print STDERR "info: Running scripts in $dir\n";
opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
while (my $s = readdir $dh) {
next if $s =~ m/^\.\.?/;
run_firmware_script($opts, "$dir/$s");
}
closedir $dh;
}
}
sub download {
my $url = shift;
print STDERR "info: Downloading $url\n";
system("wget --quiet \"$url\"");
}
sub upgrade_dell {
my @dirs;
my $product = `dmidecode -s system-product-name`;
chomp $product;
if ($product =~ m/PowerEdge/) {
# on RHEL, these pacakges are needed by the firwmare upgrade scripts
system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');
my $tmpdir = tempdir(
CLEANUP => 1
);
chdir($tmpdir);
fetch_dell_fw('catalog/Catalog.xml.gz');
system('gunzip Catalog.xml.gz');
my @paths = fetch_dell_fw_list('Catalog.xml');
# -q is quiet, disabling interactivity and reducing console output
my $fwopts = "-q";
if (@paths) {
for my $url (@paths) {
fetch_dell_fw($url);
}
run_firmware_scripts($fwopts, $tmpdir);
} else {
print STDERR "error: Unsupported Dell model '$product'.\n";
print STDERR "error: Please report to $errorsto.\n";
}
chdir('/');
} else {
print STDERR "error: Unsupported Dell model '$product'.\n";
print STDERR "error: Please report to $errorsto.\n";
}
}
sub fetch_dell_fw {
my $path = shift;
my $url = "ftp://ftp.us.dell.com/$path";
download($url);
}
# Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
# firmware packages to download from Dell. Only work for Linux
# machines and 11th generation Dell servers.
sub fetch_dell_fw_list {
my $filename = shift;
my $product = `dmidecode -s system-product-name`;
chomp $product;
my ($mybrand, $mymodel) = split(/\s+/, $product);
print STDERR "Finding firmware bundles for $mybrand $mymodel\n";
my $xml = XMLin($filename);
my @paths;
for my $bundle (@{$xml->{SoftwareBundle}}) {
my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
my $oscode;
if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
$oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
} else {
$oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
}
if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
{
@paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
}
}
for my $component (@{$xml->{SoftwareComponent}}) {
my $componenttype = $component->{ComponentType}->{value};
# Drop application packages, only firmware and BIOS
next if 'APAC' eq $componenttype;
my $cpath = $component->{path};
for my $path (@paths) {
if ($cpath =~ m%/$path$%) {
push(@paths, $cpath);
}
}
}
return @paths;
}
The code is only tested on RedHat Enterprise Linux, but I suspect
it could work on other platforms with some tweaking. Anyone know a
index like Catalog.xml is available from HP for HP servers? At the
moment I maintain a similar list manually and it is quickly getting
outdated.
