Thanks a ton for all the great suggestions at the meeting the other night about my slow disk I/O. (For those not at the meeting, this is the slow I/O issue I've been battling with for months and have asked about here on the mailing list numerous times.)
I think Adam(?) suggested testing dd with an option to avoid the cache completely, after others thought it sounded like a cache issue. The option for that (after a quick RTFM) is oflag=direct.
Here's the results of that on the system after being up a day (without my hourly test running):
spinning rust: 5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync 209715200 bytes (210 MB) copied, 9.07724 s, 23.1 MB/s 5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync oflag=direct 209715200 bytes (210 MB) copied, 2.04618 s, 102 MB/s
ssd: 5:18am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync 209715200 bytes (210 MB) copied, 3.38177 s, 62.0 MB/s 5:16am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync oflag=direct 209715200 bytes (210 MB) copied, 0.508799 s, 412 MB/s
Whoa! Indeed we have a cache issue. Disks are near full speed when writing direct to disk, but slow when going through the cache.
FYI, here's what top looks like after I ran the tests (for mem info):
top - 05:21:49 up 1 day, 4:43, 1 user, load average: 0.00, 0.03, 0.07 Tasks: 212 total, 1 running, 211 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16522952 total, 4668792 free, 706636 used, 11147524 buff/cache KiB Swap: 0 total, 0 free, 0 used. 15709432 avail Mem
So it looks like buff/cache is getting quite large after all (11GB). So we'll try the next suggestion:
5:23am ~#sync ; echo 1 >> /proc/sys/vm/drop_caches
top KiB Mem : 16522952 total, 15508640 free, 706136 used, 308176 buff/cache
5:24am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync 209715200 bytes (210 MB) copied, 4.39049 s, 47.8 MB/s
So that doesn't help at all (that's the ssd and should be 400-500MB/s). Very strange indeed!
At least I have some better search terms to google with in the next few days... maybe I'll get lucky.