Thanks a ton for all the great suggestions at the meeting the other night
about my slow disk I/O. (For those not at the meeting, this is the slow
I/O issue I've been battling with for months and have asked about here on
the mailing list numerous times.)
I think Adam(?) suggested testing dd with an option to avoid the cache
completely, after others thought it sounded like a cache issue. The
option for that (after a quick RTFM) is oflag=direct.
Here's the results of that on the system after being up a day (without my
hourly test running):
spinning rust:
5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync
209715200 bytes (210 MB) copied, 9.07724 s, 23.1 MB/s
5:15am ~#dd if=/dev/zero of=/dev/sda2 bs=1M count=200 conv=fdatasync oflag=direct
209715200 bytes (210 MB) copied, 2.04618 s, 102 MB/s
ssd:
5:18am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync
209715200 bytes (210 MB) copied, 3.38177 s, 62.0 MB/s
5:16am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync oflag=direct
209715200 bytes (210 MB) copied, 0.508799 s, 412 MB/s
Whoa! Indeed we have a cache issue. Disks are near full speed when
writing direct to disk, but slow when going through the cache.
FYI, here's what top looks like after I ran the tests (for mem info):
top - 05:21:49 up 1 day, 4:43, 1 user, load average: 0.00, 0.03, 0.07
Tasks: 212 total, 1 running, 211 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16522952 total, 4668792 free, 706636 used, 11147524 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15709432 avail Mem
So it looks like buff/cache is getting quite large after all (11GB). So
we'll try the next suggestion:
5:23am ~#sync ; echo 1 >> /proc/sys/vm/drop_caches
top
KiB Mem : 16522952 total, 15508640 free, 706136 used, 308176 buff/cache
5:24am ~#dd if=/dev/zero of=/dev/sdb2 bs=1M count=200 conv=fdatasync
209715200 bytes (210 MB) copied, 4.39049 s, 47.8 MB/s
So that doesn't help at all (that's the ssd and should be 400-500MB/s).
Very strange indeed!
At least I have some better search terms to google with in the next few
days... maybe I'll get lucky.