I can tell you that those numbers are pretty much bang on. Writes would be faster if you enabled the drives' write caches, the HP controller will leave them at default, which is disabled. The RAID rebuild speed should have been about 90% of that 122Mb number, which it obviously wasn't. I don't have any good theories why at the moment. -Adam
Kevin McGregor kevin.a.mcgregor@gmail.com wrote:
So far I made this md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0] 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] and got this # dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s # dd of=/dev/null if=/srv/d0/bigzerofile bs=1M 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s
I'm wondering if 12 drives would over-saturate one Ultra-320 channel. Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of 320 MB/s? I could try setting up a stripe set/RAID0 of varying numbers of drives and compare. What do you think?
I don't think the external enclosure (HP MSA30?) allows for splitting the drives into two groups. Only one cable can be connected to it, although there may have been an option for a second at purchase.
On Thu, May 19, 2011 at 9:27 PM, Adam Thompson athompso@athompso.netwrote:
Well, I did say there was more than one variable involved! 25MB/sec is a bit slow, but I don’t know how efficiently that LSI chip is at managing bus contention, or how efficient the kernel driver for that chip is. I do know from experience that 8-disk arrays are well into the land of diminishing returns from a speed perspective: RAID-1 on two disks or RAID-10 on four seems to be the sweet spot for speed.
Once your rebuild has finished, I would recommend doing some throughput tests (both reading and writing) on the array; something perhaps like:
# sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M count=1024;sync’
followed by
# time dd if=/dev/md0 of=/dev/null bs=1M count=1024
Those are both very naïve approaches, but should give you a feel for the maximum read and write speeds of your array. I strongly suspect those numbers will be much higher than the raid re-sync rate, again, mostly due to write-flush barriers in the md code.
I’m interested in knowing how this ends, personally… please let us know.
-Adam
(P.S. does anyone know how to avoid top-posting in Outlook 2010?)
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor *Sent:* Thursday, May 19, 2011 14:52 *To:* Continuation of Round Table discussion *Subject:* Re: [RndTbl] RAID5 rebuild performance
raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable? :-) Oh well, I guess I can wait until tomorrow for the rebuild to finish.
On Thu, May 19, 2011 at 2:12 PM, Adam Thompson athompso@athompso.net wrote:
Look back in dmesg output for the RAID module speed tests, notice which one was selected, and divide that number by 2. There's your theoretical bottleneck on the CPU. Take the minimum sustained disk/channel/controller throughput, factor in interrupt latency, device driver efficiency, etc. and make a rough guess as to the overall throughput. Consider that md code seems to have a lot of write barriers for safety - so even a rebuild will aspens much of its time waiting for the disk to sync(). All in all, I think your numbers are probably reasonable. -Adam
Kevin McGregor kevin.a.mcgregor@gmail.com wrote:
I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3 (4 x dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the on-board SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software
RAID
5 set up as a 7-drive array with 1 hot-spare drive. All drives are the
exact
same model with the same firmware version.
It's currently rebuilding the array (because I just created the array) and /proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that sound "right" in the sense that it's the right order of magnitude? I
though
it should be higher, but I haven't set up such an array before, so I don't have anything to compare it to.
If it's slow, does anyone have a suggestion for speeding it up?
Kevin
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
When the SCSI controller BIOS is initialized, it lists all of the drives on both channels, and most of them (as I recall) are described as being set to 80 MB/s with a note that the OS will probably set them to 320 MB/s. Am I recalling this correctly? Does this make any sense? Does Linux do this? Is there anything I have to do to enable this? I didn't see any messages in dmesg to this effect; I don't know if it's relevant.
Kevin
On Fri, May 20, 2011 at 2:05 PM, Adam Thompson athompso@athompso.netwrote:
I can tell you that those numbers are pretty much bang on. Writes would be faster if you enabled the drives' write caches, the HP controller will leave them at default, which is disabled. The RAID rebuild speed should have been about 90% of that 122Mb number, which it obviously wasn't. I don't have any good theories why at the moment. -Adam
Kevin McGregor kevin.a.mcgregor@gmail.com wrote:
So far I made this md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0] 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] and got this # dd if=/dev/zero of=/srv/d0/bigzerofile bs=1M count=32768 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 281.034 s, 122 MB/s # dd of=/dev/null if=/srv/d0/bigzerofile bs=1M 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 126.21 s, 272 MB/s
I'm wondering if 12 drives would over-saturate one Ultra-320 channel. Doesn't Ultra-320 suggest a maximum usable (or theoretical) capacity of
320
MB/s? I could try setting up a stripe set/RAID0 of varying numbers of
drives
and compare. What do you think?
I don't think the external enclosure (HP MSA30?) allows for splitting the drives into two groups. Only one cable can be connected to it, although there may have been an option for a second at purchase.
On Thu, May 19, 2011 at 9:27 PM, Adam Thompson <athompso@athompso.net wrote:
Well, I did say there was more than one variable involved! 25MB/sec is
a
bit slow, but I don’t know how efficiently that LSI chip is at managing
bus
contention, or how efficient the kernel driver for that chip is. I do
know
from experience that 8-disk arrays are well into the land of diminishing returns from a speed perspective: RAID-1 on two disks or RAID-10 on four seems to be the sweet spot for speed.
Once your rebuild has finished, I would recommend doing some throughput tests (both reading and writing) on the array; something perhaps like:
# sync; time sh –c ‘dd if=/dev/zero of=/mnt/raidarray/BIGFILE.ZERO bs=1M count=1024;sync’
followed by
# time dd if=/dev/md0 of=/dev/null bs=1M count=1024
Those are both very naïve approaches, but should give you a feel for the maximum read and write speeds of your array. I strongly suspect those numbers will be much higher than the raid re-sync rate, again, mostly
due to
write-flush barriers in the md code.
I’m interested in knowing how this ends, personally… please let us know.
-Adam
(P.S. does anyone know how to avoid top-posting in Outlook 2010?)
*From:* roundtable-bounces@muug.mb.ca [mailto: roundtable-bounces@muug.mb.ca] *On Behalf Of *Kevin McGregor *Sent:* Thursday, May 19, 2011 14:52 *To:* Continuation of Round Table discussion *Subject:* Re: [RndTbl] RAID5 rebuild performance
raid6: using algorithm sse2x2 (2883 MB/s). So, 1% of that is reasonable? :-) Oh well, I guess I can wait until tomorrow for the rebuild to
finish.
On Thu, May 19, 2011 at 2:12 PM, Adam Thompson athompso@athompso.net wrote:
Look back in dmesg output for the RAID module speed tests, notice which
one
was selected, and divide that number by 2. There's your theoretical bottleneck on the CPU. Take the minimum sustained disk/channel/controller throughput, factor in interrupt latency, device driver efficiency, etc. and make a rough guess
as
to the overall throughput. Consider that md code seems to have a lot of write barriers for safety - so even a rebuild will aspens much of its time waiting for the disk to sync(). All in all, I think your numbers are probably reasonable. -Adam
Kevin McGregor kevin.a.mcgregor@gmail.com wrote:
I installed Ubuntu Server 10.04.2 LTS AMD64 on a HP ProLiant ML370 G3
(4 x
dual-core/hyperthreaded Xeon 2.66 GHz, 8 GB RAM) and I used the
on-board
SCSI controller to manage 8 x 300 GB 15K RPM SCSI drives in a software
RAID
5 set up as a 7-drive array with 1 hot-spare drive. All drives are the
exact
same model with the same firmware version.
It's currently rebuilding the array (because I just created the array)
and
/proc/mdstat is reporting "finish=165.7min speed=25856K/sec". Does that sound "right" in the sense that it's the right order of magnitude? I
though
it should be higher, but I haven't set up such an array before, so I
don't
have anything to compare it to.
If it's slow, does anyone have a suggestion for speeding it up?
Kevin
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 2011-05-20 Kevin McGregor wrote:
When the SCSI controller BIOS is initialized, it lists all of the drives on both channels, and most of them (as I recall) are described as being set to 80 MB/s with a note that the OS will probably set
Weird, see if the SCSI BIOS setup screens let you manually select the SCSI speed. You might have to drill down to the per-drive area. I know my U320 card gives me choices.
80MB/s per drive isn't shabby, though your drives can probably push 100-150MB/s.
-
Adam Thompson -
Kevin McGregor -
Trevor Cordes