Okay, here's (maybe) the last word on my RAID issues at work, if anyone's still reading these. :)
My RAID5 array is 8x 300 GB 15K RPM U320 drives, 7 active with 1 hot spare md_d1 : active raid5 sdu1[6] sdv1[7](S) sdt1[5] sds1[4] sdr1[3] sdq1[2] sdp1[1] sdo1[0] 1757804544 blocks level 5, 256k chunk, algorithm 2 [7/7] [UUUUUUU]
Just one drive from this array shows: /dev/sdr: Timing cached reads: 1438 MB in 2.00 seconds = 718.54 MB/sec Timing buffered disk reads: 384 MB in 3.00 seconds = 127.97 MB/sec
/srv/d1# dd if=/dev/zero of=bigtestfile bs=2M count=16384 16384+0 records in 16384+0 records out 34359738368 bytes (34 GB) copied, 414.181 s, 83.0 MB/s [writing]
/srv/d1# dd of=/dev/null if=bigtestfile bs=2M 16384+0 records in 16384+0 records out 34359738368 bytes (34 GB) copied, 126.176 s, 272 MB/s [reading]
My RAID10 array is 14x 300 GB 10K RPM U320 drives, 12 active with two hot spares md_d0 : active raid10 sdn1[12](S) sdm1[13](S) sdl1[11] sdk1[10] sdj1[9] sdi1[8] sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0] 1757804544 blocks 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
Just one drive from this array shows: /dev/sdl: Timing cached reads: 1440 MB in 2.00 seconds = 719.94 MB/sec Timing buffered disk reads: 254 MB in 3.00 seconds = 84.63 MB/sec
/srv/d0# dd if=/dev/zero of=bigtestfile bs=2M count=16384 16384+0 records in 16384+0 records out 34359738368 bytes (34 GB) copied, 280.52 s, 122 MB/s [writing]
/srv/d0# dd of=/dev/null if=bigtestfile bs=2M 16384+0 records in 16384+0 records out 34359738368 bytes (34 GB) copied, 126.134 s, 272 MB/s [reading]
Or one could say that reading is the same speed from either array, and pretty much at the maximum practical for an Ultra-320 bus; writing is faster (for my configuration) by 50% on the RAID10 array as compared to the RAID5 array, despite the RAID5 array sporting drives which are ~50% faster (15K RPM vs. 10K RPM).
Curiously, copying the 'bigtestfile' from d0 to d1 or d1 to d0 results in ~80 MB/s either way. I can't think of an explanation off the top of my head.
There is a lot more testing which could be done, and I'm not saying one configuration is better than the other. I think I may leave it as is for now.
Kevin
On Mon, May 23, 2011 at 5:33 PM, Kevin McGregor kevin.a.mcgregor@gmail.comwrote:
For yet another comparison, my RAID10 6x 750 GB SATA XFS gives: # dd if=/dev/zero of=bigfile bs=1M count=16384 16384+0 records in 16384+0 records out 17179869184 bytes (17 GB) copied, 79.5334 s, 216 MB/s # dd of=/dev/null if=bigfile bs=1M 16384+0 records in 16384+0 records out 17179869184 bytes (17 GB) copied, 31.4869 s, 546 MB/s
Maybe I should switch to RAID6. ;-)
Kevin
On Fri, May 20, 2011 at 4:54 PM, Trevor Cordes trevor@tecnopolis.cawrote:
My RAID6 8x 2TB-drive SATA XFS gives:
#dd if=/dev/zero of=/new/test bs=1M count=32768 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 152.032 s, 226 MB/s
#dd of=/dev/null if=/new/test bs=1M 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 57.7027 s, 595 MB/s (wow!!)
During the whole write time both CPU cores were 90-100%, mostly 95-99%! Glad to see RAID/XFS code is multi-core aware. For reading it was 1 core at 100% and the other around 20%. The write limiting factor appears to be my piddly Pentium D on my file server. Still, this is 3-4X the speed my old (1TB drives, crappy PCI SATA cards) array was giving me. The read is quite interesting in that the 100% CPU indicates it is probably doing parity checks on every read.
I think a big part of the good speed is my new 8-port SATA card, an Intel PCI-Express x 8 in a x8 slot. If your SCSI card is just PCI, then the PCI MB/s speed limit is what's killing you. Even PCI-X may be limiting. And the Intel card was pretty cheap, under $200.
BTW, I got stuck with two spare SATA card expander cables (1 card port to 4 SATA drives) if anyone wants some cheap. I can get in the Intel cards too, if anyone wants a complete package. _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable