[homepage.git] / blog / tags / raid / raid.rss

<?xml version="1.0" encoding="utf-8"?>
<rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/'>
        <channel>
                <title>Petter Reinholdtsen - Entries tagged raid</title>
                <description>Entries tagged raid</description>
                <link>https://people.skolelinux.org/pere/blog/</link>

        
        <item>
                <title>RAID status from LSI Megaraid controllers in Debian</title>
                <link>https://people.skolelinux.org/pere/blog/RAID_status_from_LSI_Megaraid_controllers_in_Debian.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/RAID_status_from_LSI_Megaraid_controllers_in_Debian.html</guid>
                <pubDate>Wed, 17 Apr 2024 17:00:00 +0200</pubDate>
                <description>&lt;p&gt;I am happy to report that
&lt;ahref=&quot;https://github.com/namiltd/megactl&quot;&gt;the megactl package&lt;/a&gt;,
useful to fetch RAID status when using the LSI Megaraid controller,
now is available in Debian.  It passed NEW a few days ago, and is now
&lt;ahref=&quot;https://tracker.debian.org/pkg/megactl&quot;&gt;available in
unstable&lt;/a&gt;, and probably showing up in testing in a weeks time.  The
new version should provide Appstream hardware mapping and should
integrate nicely with isenkram.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;

</description>
        </item>
        
        <item>
                <title>RAID status from LSI Megaraid controllers using free software</title>
                <link>https://people.skolelinux.org/pere/blog/RAID_status_from_LSI_Megaraid_controllers_using_free_software.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/RAID_status_from_LSI_Megaraid_controllers_using_free_software.html</guid>
                <pubDate>Sun, 3 Mar 2024 22:40:00 +0100</pubDate>
                <description>&lt;p&gt;The last few days I have revisited RAID setup using the LSI
Megaraid controller.  These are a family of controllers called PERC by
Dell, and is present in several old PowerEdge servers, and I recently
got my hands on one of these.  I had forgotten how to handle this RAID
controller in Debian, so I had to take a peek in the
&lt;a href=&quot;https://wiki.debian.org/LinuxRaidForAdmins&quot;&gt;Debian wiki page
&quot;Linux and Hardware RAID: an administrator&#39;s summary&quot;&lt;/a&gt; to remember
what kind of software is available to configure and monitor the disks
and controller.  I prefer Free Software alternatives to proprietary
tools, as the later tend to fall into disarray once the manufacturer
loose interest, and often do not work with newer Linux Distributions.
Sadly there is no free software tool to configure the RAID setup, only
to monitor it. RAID can provide improved reliability and resilience in
a storage solution, but only if it is being regularly checked and any
broken disks are being replaced in time.  I thus want to ensure some
automatic monitoring is available.&lt;/p&gt;

&lt;p&gt;In the discovery process, I came across a old free software tool to
monitor PERC2, PERC3, PERC4 and PERC5 controllers, which to my
surprise is not present in debian.  To help change that I created a
&lt;a href=&quot;https://bugs.debian.org/1065322&quot;&gt;request for packaging of the
megactl package&lt;/a&gt;, and tried to track down a usable version.
&lt;a href=&quot;https://sourceforge.net/p/megactl/&quot;&gt;The original project
site&lt;/a&gt; is on Sourceforge, but as far as I can tell that project has
been dead for more than 15 years.  I managed to find a
&lt;a href=&quot;https://github.com/hmage/megactl&quot;&gt;more recent fork on
github&lt;/a&gt; from user hmage, but it is unclear to me if this is still
being maintained.  It has not seen much improvements since 2016.  A
&lt;a href=&quot;https://github.com/namiltd/megactl&quot;&gt;more up to date
edition&lt;/a&gt; is a git fork from the original github fork by user
namiltd, and this newer fork seem a lot more promising.  The owner of
this github repository has replied to change proposals within hours,
and had already added some improvements and support for more hardware.
Sadly he is reluctant to commit to maintaining the tool and stated in
&lt;a href=&quot;https://github.com/namiltd/megactl/pull/1&quot;&gt;my first pull
request&lt;/A&gt; that he think a new release should be made based on the
git repository owned by hmage.  I perfectly understand this
reluctance, as I feel the same about maintaining yet another package
in Debian when I barely have time to take care of the ones I already
maintain, but do not really have high hopes that hmage will have time
to spend on it and hope namiltd will change his mind.&lt;/p&gt;

&lt;p&gt;In any case, I created
&lt;a href=&quot;https://salsa.debian.org/debian/megactl&quot;&gt;a draft package&lt;/a&gt;
based on the namiltd edition and put it under the debian group on
salsa.debian.org.  If you own a Dell PowerEdge server with one of the
PERC controllers, or any other RAID controller using the megaraid or
megaraid_sas Linux kernel modules, you might want to check it out.  If
enough people are interested, perhaps the package will make it into
the Debian archive.&lt;/p&gt;

&lt;p&gt;There are two tools provided, megactl for the megaraid Linux kernel
module, and megasasctl for the megaraid_sas Linux kernel module. The
simple output from the command on one of my machines look like this
(yes, I know some of the disks have problems. :).&lt;/p&gt;

&lt;pre&gt;
# megasasctl 
a0       PERC H730 Mini           encl:1 ldrv:2  batt:good
a0d0       558GiB RAID 1   1x2  optimal
a0d1      3067GiB RAID 0   1x11 optimal
a0e32s0     558GiB  a0d0  online   errs: media:0  other:19
a0e32s1     279GiB  a0d1  online  
a0e32s2     279GiB  a0d1  online  
a0e32s3     279GiB  a0d1  online  
a0e32s4     279GiB  a0d1  online  
a0e32s5     279GiB  a0d1  online  
a0e32s6     279GiB  a0d1  online  
a0e32s8     558GiB  a0d0  online   errs: media:0  other:17
a0e32s9     279GiB  a0d1  online  
a0e32s10    279GiB  a0d1  online  
a0e32s11    279GiB  a0d1  online  
a0e32s12    279GiB  a0d1  online  
a0e32s13    279GiB  a0d1  online  

#
&lt;/pre&gt;

&lt;p&gt;In addition to displaying a simple status report, it can also test
individual drives and print the various event logs.  Perhaps you too
find it useful?&lt;/p&gt;

&lt;p&gt;In the packaging process I provided some patches upstream to
improve installation and ensure
&lt;ahref=&quot;https://github.com/namiltd/megactl/pull/2&quot;&gt;a Appstream
metainfo file is provided&lt;/a&gt; to list all supported HW, to allow
&lt;a href=&quot;https://tracker.debian.org/isenkram&quot;&gt;isenkram&lt;/a&gt; to propose
the package on all servers with a relevant PCI card.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;

</description>
        </item>
        
        <item>
                <title>Some notes on fault tolerant storage systems</title>
                <link>https://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</guid>
                <pubDate>Wed, 1 Nov 2017 15:35:00 +0100</pubDate>
                <description>&lt;p&gt;If you care about how fault tolerant your storage is, you might
find these articles and papers interesting.  They have formed how I
think of when designing a storage system.&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;USENIX :login; &lt;a
href=&quot;https://www.usenix.org/publications/login/summer2017/ganesan&quot;&gt;Redundancy
Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
Reactions to Single Errors and Corruptions&lt;/a&gt; by Aishwarya Ganesan,
Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
H. Arpaci-Dusseau&lt;/li&gt;

&lt;li&gt;ZDNet
&lt;a href=&quot;http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/&quot;&gt;Why
RAID 5 stops working in 2009&lt;/a&gt; by Robin Harris&lt;/li&gt;

&lt;li&gt;ZDNet
&lt;a href=&quot;http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/&quot;&gt;Why
RAID 6 stops working in 2019&lt;/a&gt; by Robin Harris&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;07
&lt;a href=&quot;http://research.google.com/archive/disk_failures.pdf&quot;&gt;Failure
Trends in a Large Disk Drive Population&lt;/a&gt; by Eduardo Pinheiro,
Wolf-Dietrich Weber and Luiz André Barroso&lt;/li&gt;

&lt;li&gt;USENIX ;login: &lt;a
href=&quot;https://www.usenix.org/system/files/login/articles/hughes12-04.pdf&quot;&gt;Data
Integrity.  Finding Truth in a World of Guesses and Lies&lt;/a&gt; by Doug
Hughes&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;08
&lt;a href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/&quot;&gt;An
Analysis of Data Corruption in the Storage Stack&lt;/a&gt; by
L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
Arpaci-Dusseau, and R. H. Arpaci-Dusseau&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;07 &lt;a
href=&quot;https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/&quot;&gt;Disk
failures in the real world: what does an MTTF of 1,000,000 hours mean
to you?&lt;/a&gt; by B. Schroeder and G. A. Gibson.&lt;/li&gt;

&lt;li&gt;USENIX ;login: &lt;a
href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/&quot;&gt;Are
Disks the Dominant Contributor for Storage Failures?  A Comprehensive
Study of Storage Subsystem Failure Characteristics&lt;/a&gt; by Weihang
Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky&lt;/li&gt;

&lt;li&gt;SIGMETRICS 2007
&lt;a href=&quot;http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf&quot;&gt;An
analysis of latent sector errors in disk drives&lt;/a&gt; by
L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;Several of these research papers are based on data collected from
hundred thousands or millions of disk, and their findings are eye
opening.  The short story is simply do not implicitly trust RAID or
redundant storage systems.  Details matter.  And unfortunately there
are few options on Linux addressing all the identified issues.  Both
ZFS and Btrfs are doing a fairly good job, but have legal and
practical issues on their own.  I wonder how cluster file systems like
Ceph do in this regard.  After all, there is an old saying, you know
you have a distributed system when the crash of a computer you have
never heard of stops you from getting any work done.  The same holds
true if fault tolerance do not work.&lt;/p&gt;

&lt;p&gt;Just remember, in the end, it do not matter how redundant, or how
fault tolerant your storage is, if you do not continuously monitor its
status to detect and replace failed disks.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>How to figure out which RAID disk to replace when it fail</title>
                <link>https://people.skolelinux.org/pere/blog/How_to_figure_out_which_RAID_disk_to_replace_when_it_fail.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/How_to_figure_out_which_RAID_disk_to_replace_when_it_fail.html</guid>
                <pubDate>Tue, 14 Feb 2012 21:25:00 +0100</pubDate>
                <description>&lt;p&gt;Once in a while my home server have disk problems.  Thanks to Linux
Software RAID, I have not lost data yet (but
&lt;a href=&quot;http://comments.gmane.org/gmane.linux.raid/34532&quot;&gt;I was
close&lt;/a&gt; this summer :).  But once a disk is starting to behave
funny, a practical problem present itself.  How to get from the Linux
device name (like /dev/sdd) to something that can be used to identify
the disk when the computer is turned off?  In my case I have SATA
disks with a unique ID printed on the label.  All I need is a way to
figure out how to query the disk to get the ID out.&lt;/p&gt;

&lt;p&gt;After fumbling a bit, I
&lt;a href=&quot;http://www.cyberciti.biz/faq/linux-getting-scsi-ide-harddisk-information/&quot;&gt;found
that hdparm -I&lt;/a&gt; will report the disk serial number, which is
printed on the disk label.  The following (almost) one-liner can be
used to look up the ID of all the failed disks:&lt;/p&gt;

&lt;blockquote&gt;&lt;pre&gt;
for d in $(cat /proc/mdstat |grep &#39;(F)&#39;|tr &#39; &#39; &quot;\n&quot;|grep &#39;(F)&#39;|cut -d\[ -f1|sort -u);
do
    printf &quot;Failed disk $d: &quot;
    hdparm -I /dev/$d |grep &#39;Serial Num&#39;
done
&lt;/blockquote&gt;&lt;/pre&gt;

&lt;p&gt;Putting it here to make sure I do not have to search for it the
next time, and in case other find it useful.&lt;/p&gt;

&lt;p&gt;At the moment I have two failing disk. :(&lt;/p&gt;

&lt;blockquote&gt;&lt;pre&gt;
Failed disk sdd1:       Serial Number:      WD-WCASJ1860823
Failed disk sdd2:       Serial Number:      WD-WCASJ1860823
Failed disk sde2:       Serial Number:      WD-WCASJ1840589
&lt;/blockquote&gt;&lt;/pre&gt;

&lt;p&gt;The last time I had failing disks, I added the serial number on
labels I printed and stuck on the short sides of each disk, to be able
to figure out which disk to take out of the box without having to
remove each disk to look at the physical vendor label.  The vendor
label is at the top of the disk, which is hidden when the disks are
mounted inside my box.&lt;/p&gt;

&lt;p&gt;I really wish the check_linux_raid Nagios plugin for checking Linux
Software RAID in the
&lt;a href=&quot;http://packages.qa.debian.org/n/nagios-plugins.html&quot;&gt;nagios-plugins-standard&lt;/a&gt;
debian package would look up this value automatically, as it would
make the plugin a lot more useful when my disks fail.  At the moment
it only report a failure when there are no more spares left (it really
should warn as soon as a disk is failing), and it do not tell me which
disk(s) is failing when the RAID is running short on disks.&lt;/p&gt;
</description>
        </item>
        
        </channel>
</rss>