On 2012-11-20 08:48, Trevor Cordes wrote:
My main box has 8GB RAM. There's usually 3GB of programs running. The rest usually is shown as free or going to buffers/cache.
I have a set of files I often linearly scan through and they are around 1GB. They used to read in and stay cached for hours (I can tell because my scan takes 3s when cached and 90s when not!). Now they seem to get dropped and require re-reading all the time, like if I don't use them for a few minutes.
AFAICT my ps load has not changed and I'm not doing anything that is requiring more cache/buffers that is pushing them out.
I wouldn't be so sure about that, unless you've really checked that closely, and/or disabled most system services that might affect memory usage in the background. I've found that newer Fedora releases tend to stay busier than earlier ones, so you might just end up with more swapping activity and cache flushing than before.
This seems to have started when I upgraded my kernel from 3.4 to 3.6 (Fedora 16) though I can't be sure at this point.
I'm not too familiar with the 3.x kernel changes that might affect memory usage, particularly with caching. A quick look at the changes in 3.6 doesn't show any really obvious changes to that, though.
Does anyone know of some tunables I can try to tell the kernel to be more likely to cache for longer? I'd rather cache at the expense of pushing more programs to swap (I run a zillion programs at all times and never close them).
This is rather out of date for the 3.x kernels, but here's a pretty good tuning guide for RHEL 6 systems (2.6 kernel)...
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/ht...
I have played with /proc/sys/vm/swappiness (80 has been great for me in the past) but a) it doesn't seem to affect my problem now, and b) the docs leave me scratching my head about how it really affects file caching.
Have you tried setting swappiness all the way to 100? Did that have no effect?
At this point I'm not interested in solutions that are "rethink your scan!" as I do plan to one day database-itize and index the 1GB data set... but not today!
Adding more RAM might be another option. :)