My RAID5 array is 8x 300 GB 15K RPM U320 drives, 7 active with 1 hot spare
md_d1 : active raid5 sdu1[6] sdv1[7](S) sdt1[5] sds1[4] sdr1[3] sdq1[2] sdp1[1] sdo1[0]
1757804544 blocks level 5, 256k chunk, algorithm 2 [7/7] [UUUUUUU]
Just one drive from this array shows:
/dev/sdr:
Timing cached reads: 1438 MB in 2.00 seconds = 718.54 MB/sec
Timing buffered disk reads: 384 MB in 3.00 seconds = 127.97 MB/sec
/srv/d1# dd if=/dev/zero of=bigtestfile bs=2M count=16384
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 414.181 s, 83.0 MB/s [writing]
/srv/d1# dd of=/dev/null if=bigtestfile bs=2M
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 126.176 s, 272 MB/s [reading]
My RAID10 array is 14x 300 GB 10K RPM U320 drives, 12 active with two hot spares
md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
Just one drive from this array shows:
/dev/sdl:
Timing cached reads: 1440 MB in 2.00 seconds = 719.94 MB/sec
Timing buffered disk reads: 254 MB in 3.00 seconds = 84.63 MB/sec
/srv/d0# dd if=/dev/zero of=bigtestfile bs=2M count=16384
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 280.52 s, 122 MB/s [writing]
/srv/d0# dd of=/dev/null if=bigtestfile bs=2M
16384+0 records in
16384+0 records out
34359738368 bytes (34 GB) copied, 126.134 s, 272 MB/s [reading]
Or one could say that reading is the same speed from either array, and pretty much at the maximum practical for an Ultra-320 bus; writing is faster (for my configuration) by 50% on the RAID10 array as compared to the RAID5 array, despite the RAID5 array sporting drives which are ~50% faster (15K RPM vs. 10K RPM).
Curiously, copying the 'bigtestfile' from d0 to d1 or d1 to d0 results in ~80 MB/s either way. I can't think of an explanation off the top of my head.
There is a lot more testing which could be done, and I'm not saying one configuration is better than the other. I think I may leave it as is for now.
Kevin
On Mon, May 23, 2011 at 5:33 PM, Kevin McGregor
<kevin.a.mcgregor@gmail.com> wrote:
For yet another comparison, my RAID10 6x 750 GB SATA XFS gives:# dd if=/dev/zero of=bigfile bs=1M count=16384
16384+0 records in
16384+0 records out
# dd of=/dev/null if=bigfile bs=1M
16384+0 records in
16384+0 records out
Maybe I should switch to RAID6. ;-)
Kevin
My RAID6 8x 2TB-drive SATA XFS gives:
#dd if=/dev/zero of=/new/test bs=1M count=32768
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 152.032 s, 226 MB/s
#dd of=/dev/null if=/new/test bs=1M
32768+0 records in
32768+0 records out
34359738368 bytes (34 GB) copied, 57.7027 s, 595 MB/s
(wow!!)
During the whole write time both CPU cores were 90-100%, mostly 95-99%!
Glad to see RAID/XFS code is multi-core aware. For reading it was 1
core at 100% and the other around 20%. The write limiting factor
appears to be my piddly Pentium D on my file server. Still, this is
3-4X the speed my old (1TB drives, crappy PCI SATA cards) array was
giving me. The read is quite interesting in that the 100% CPU
indicates it is probably doing parity checks on every read.
I think a big part of the good speed is my new 8-port SATA card, an
Intel PCI-Express x 8 in a x8 slot. If your SCSI card is just PCI,
then the PCI MB/s speed limit is what's killing you. Even PCI-X may be
limiting. And the Intel card was pretty cheap, under $200.
BTW, I got stuck with two spare SATA card expander cables (1 card
port to 4 SATA drives) if anyone wants some cheap. I can get in the
Intel cards too, if anyone wants a complete package.