[RndTbl] RAID5 rebuild performance

Fri May 20 15:10:25 CDT 2011

When the SCSI controller BIOS is initialized, it lists all of the drives on
both channels, and most of them (as I recall) are described as being set to
80 MB/s with a note that the OS will probably set them to 320 MB/s. Am I
recalling this correctly? Does this make any sense? Does Linux do this? Is
there anything I have to do to enable this? I didn't see any messages in
dmesg to this effect; I don't know if it's relevant.

Kevin

On Fri, May 20, 2011 at 2:05 PM, Adam Thompson <athompso at athompso.net>wrote:

> I can tell you that those numbers are pretty much bang on.  Writes would be
> faster if you enabled the drives' write caches, the HP controller will leave
> them at default, which is disabled.
> The RAID rebuild speed should have been about 90% of that 122Mb number,
> which it obviously wasn't.  I don't have any good theories why at the
> moment.
> -Adam
>
>
> Kevin McGregor <kevin.a.mcgregor at gmail.com> wrote:
>
> >So far I made this
> >md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9]
> >sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
> >      1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
> >and got this
> ># dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768
> >32768+0 records in
> >32768+0 records out
> >34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s
> ># dd of=/dev/null if=/srv/d0/bigzerofile bs=1M
> >32768+0 records in
> >32768+0 records out
> >34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s
> >
> >I'm wondering if 12 drives would over-saturate one Ultra-320 channel.
> >Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of
> 320
> >MB/s? I could try setting up a stripe set/RAID0 of varying numbers of
> drives
> >and compare. What do you think?
> >
> >I don't think the external enclosure (HP MSA30?) allows for splitting the
> >drives into two groups. Only one cable can be connected to it, although
> >there may have been an option for a second at purchase.
> >
> >On Thu, May 19, 2011 at 9:27 PM, Adam Thompson <athompso at athompso.net
> >wrote:
> >
> >> Well, I did say there was more than one variable involved!  25MB/sec is
> a
> >> bit slow, but I don’t know how efficiently that LSI chip is at managing
> bus
> >> contention, or how efficient the kernel driver for that chip is.  I do
> know
> >> from experience that 8-disk arrays are well into the land of diminishing
> >> returns from a speed perspective: RAID-1 on two disks or RAID-10 on four
> >> seems to be the sweet spot for speed.
> >>
> >>
> >>
> >> Once your rebuild has finished, I would recommend doing some throughput
> >> tests (both reading and writing) on the array; something perhaps like:
> >>
> >> # sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M
> >> count=1024;sync’
> >>
> >> followed by
> >>
> >> # time dd if=/dev/md0 of=/dev/null bs=1M count=1024
> >>
> >> Those are both very naïve approaches, but should give you a feel for the
> >> maximum read and write speeds of your array.  I strongly suspect those
> >> numbers will be much higher than the raid re-sync rate, again, mostly
> due to
> >> write-flush barriers in the md code.
> >>
> >>
> >>
> >> I’m interested in knowing how this ends, personally… please let us know.
> >>
> >>
> >>
> >> -Adam
> >>
> >>
> >>
> >> (P.S. does anyone know how to avoid top-posting in Outlook 2010?)
> >>
> >>
> >>
> >>
> >>
> >> *From:* roundtable-bounces at muug.mb.ca [mailto:
> >> roundtable-bounces at muug.mb.ca] *On Behalf Of *Kevin McGregor
> >> *Sent:* Thursday, May 19, 2011 14:52
> >> *To:* Continuation of Round Table discussion
> >> *Subject:* Re: [RndTbl] RAID5 rebuild performance
> >>
> >>
> >>
> >> raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable?
> >> :-) Oh well, I guess I can wait until tomorrow for the rebuild to
> finish.
> >>
> >> On Thu, May 19, 2011 at 2:12 PM, Adam Thompson <athompso at athompso.net>
> >> wrote:
> >>
> >> Look back in dmesg output for the RAID module speed tests, notice which
> one
> >> was selected, and divide that number by 2.  There's your theoretical
> >> bottleneck on the CPU.
> >> Take the minimum sustained disk/channel/controller throughput, factor in
> >> interrupt latency, device driver efficiency, etc. and make a rough guess
> as
> >> to the overall throughput.
> >> Consider that md code seems to have a lot of write barriers for safety -
> >>  so even a rebuild will aspens much of its time waiting for the disk to
> >> sync().
> >> All in all, I think your numbers are probably reasonable.
> >> -Adam
> >>
> >>
> >>
> >> Kevin McGregor <kevin.a.mcgregor at gmail.com> wrote:
> >>
> >> >I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3
> (4 x
> >> >dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the
> on-board
> >> >SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software
> >> RAID
> >> >5 set up as a 7-drive array with 1 hot-spare drive. All drives are the
> >> exact
> >> >same model with the same firmware version.
> >> >
> >> >It's currently rebuilding the array (because I just created the array)
> and
> >> >/proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that
> >> >sound "right" in the sense that it's the right order of magnitude? I
> >> though
> >> >it should be higher, but I haven't set up such an array before, so I
> don't
> >> >have anything to compare it to.
> >> >
> >> >If it's slow, does anyone have a suggestion for speeding it up?
> >> >
> >> >Kevin
> >> >
> >> >_______________________________________________
> >> >Roundtable mailing list
> >> >Roundtable at muug.mb.ca
> >> >http://www.muug.mb.ca/mailman/listinfo/roundtable
> >>
> >> _______________________________________________
> >> Roundtable mailing list
> >> Roundtable at muug.mb.ca
> >> http://www.muug.mb.ca/mailman/listinfo/roundtable
> >>
> >>
> >>
> >> _______________________________________________
> >> Roundtable mailing list
> >> Roundtable at muug.mb.ca
> >> http://www.muug.mb.ca/mailman/listinfo/roundtable
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.muug.mb.ca/pipermail/roundtable/attachments/20110520/9d3a5b51/attachment.html 