My main box has 8GB RAM. There's usually 3GB of programs running. The rest usually is shown as free or going to buffers/cache.
I have a set of files I often linearly scan through and they are around 1GB. They used to read in and stay cached for hours (I can tell because my scan takes 3s when cached and 90s when not!). Now they seem to get dropped and require re-reading all the time, like if I don't use them for a few minutes.
AFAICT my ps load has not changed and I'm not doing anything that is requiring more cache/buffers that is pushing them out.
This seems to have started when I upgraded my kernel from 3.4 to 3.6 (Fedora 16) though I can't be sure at this point.
Does anyone know of some tunables I can try to tell the kernel to be more likely to cache for longer? I'd rather cache at the expense of pushing more programs to swap (I run a zillion programs at all times and never close them).
I have played with /proc/sys/vm/swappiness (80 has been great for me in the past) but a) it doesn't seem to affect my problem now, and b) the docs leave me scratching my head about how it really affects file caching.
At this point I'm not interested in solutions that are "rethink your scan!" as I do plan to one day database-itize and index the 1GB data set... but not today!
On 2012-11-20 08:48, Trevor Cordes wrote:
My main box has 8GB RAM. There's usually 3GB of programs running. The rest usually is shown as free or going to buffers/cache.
I have a set of files I often linearly scan through and they are around 1GB. They used to read in and stay cached for hours (I can tell because my scan takes 3s when cached and 90s when not!). Now they seem to get dropped and require re-reading all the time, like if I don't use them for a few minutes.
AFAICT my ps load has not changed and I'm not doing anything that is requiring more cache/buffers that is pushing them out.
I wouldn't be so sure about that, unless you've really checked that closely, and/or disabled most system services that might affect memory usage in the background. I've found that newer Fedora releases tend to stay busier than earlier ones, so you might just end up with more swapping activity and cache flushing than before.
This seems to have started when I upgraded my kernel from 3.4 to 3.6 (Fedora 16) though I can't be sure at this point.
I'm not too familiar with the 3.x kernel changes that might affect memory usage, particularly with caching. A quick look at the changes in 3.6 doesn't show any really obvious changes to that, though.
Does anyone know of some tunables I can try to tell the kernel to be more likely to cache for longer? I'd rather cache at the expense of pushing more programs to swap (I run a zillion programs at all times and never close them).
This is rather out of date for the 3.x kernels, but here's a pretty good tuning guide for RHEL 6 systems (2.6 kernel)...
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/ht...
I have played with /proc/sys/vm/swappiness (80 has been great for me in the past) but a) it doesn't seem to affect my problem now, and b) the docs leave me scratching my head about how it really affects file caching.
Have you tried setting swappiness all the way to 100? Did that have no effect?
At this point I'm not interested in solutions that are "rethink your scan!" as I do plan to one day database-itize and index the 1GB data set... but not today!
Adding more RAM might be another option. :)
On 2012-11-20 Gilbert E. Detillieux wrote:
I wouldn't be so sure about that, unless you've really checked that closely, and/or disabled most system services that might affect
I have tried to monitor things closely with top, especially sorted by memory usage. I'm doubtful it's a change in usage that is causing this because I have such a big (5GB) usual buffer/cached/free area, and I really am doing nothing during these tests that would chew through 5GB of files or data. For instance, my cached line in top stays roughly the same. That would mean that it would have to be 4GB+ of other files being read to flush out the stuff I want cached.
I run a lot of services but I've disabled all "automatic" and "desktop" stuff, for instance I run no desktop index program, or anything like that.
It used to be that the only thing that would "uncache" my files is watching some videos that obviously can chew through a few GB cache quickly.
This is rather out of date for the 3.x kernels, but here's a pretty good tuning guide for RHEL 6 systems (2.6 kernel)...
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/ht...
I'll check that out. Most of the tunables haven't changed since 2.6
Have you tried setting swappiness all the way to 100? Did that have no effect?
I'll try that next. The more I read about swappiness, the more I get confused whether in my case I want it set closer to 0 or closer to 100. I thought 100, but it really doesn't frame things in terms of cache/buffers.
Adding more RAM might be another option. :)
Older box with ECC, and 8 is the max (I have 4 sticks of 2), I think, but I will double-check as RAM has gotten so cheap. However, if I'm looking at a $200 RAM bill (I think I paid $500 for the existing 8GB?) then I'd be more likely to use it as a chance to get a whole new ECC box :-)
On 2012-11-20 13:06, Trevor Cordes wrote:
On 2012-11-20 Gilbert E. Detillieux wrote:
I wouldn't be so sure about that, unless you've really checked that closely, and/or disabled most system services that might affect
I have tried to monitor things closely with top, especially sorted by memory usage. I'm doubtful it's a change in usage that is causing this because I have such a big (5GB) usual buffer/cached/free area, and I really am doing nothing during these tests that would chew through 5GB of files or data. For instance, my cached line in top stays roughly the same. That would mean that it would have to be 4GB+ of other files being read to flush out the stuff I want cached.
I run a lot of services but I've disabled all "automatic" and "desktop" stuff, for instance I run no desktop index program, or anything like that.
It used to be that the only thing that would "uncache" my files is watching some videos that obviously can chew through a few GB cache quickly.
You might also be interested in this article...
http://www.admin-magazine.com/Articles/Tuning-Your-Filesystem-s-Cache/
Specifically, the part about the vmtouch command and the "-dl" (daemonize and lock) option. It would allow you to lock your large file(s) into memory while you work on them, I think. That way, other system activity won't evict your cached pages.
Have you tried setting swappiness all the way to 100? Did that have no effect?
I'll try that next. The more I read about swappiness, the more I get confused whether in my case I want it set closer to 0 or closer to 100. I thought 100, but it really doesn't frame things in terms of cache/buffers.
I think you'd want to go to 100, to encourage more swapping, which would have the side-effect of (hopefully) leaving more memory available for disk caching.
On 2012-11-21 Gilbert E. Detillieux wrote:
You might also be interested in this article...
http://www.admin-magazine.com/Articles/Tuning-Your-Filesystem-s-Cache/
Specifically, the part about the vmtouch command and the "-dl" (daemonize and lock) option. It would allow you to lock your large file(s) into memory while you work on them, I think. That way, other system activity won't evict your cached pages.
That's good to know for sure, but in this case my files are in thousands of little files (maildir emails) so locking certain files isn't going to help.
I finally had a chance to poke around more and I think I found the solution!!
/proc/sys/vm/vfs_cache_pressure
Defaults to 100 (what that value means appears arbitrary). I set mine to 60 and all of a sudden my files are staying cached nearly forever like they used to. There *must* have been a change to either the default setting or what the number means to the kernel around 3.3 or 3.4. It's like night and day.
Probably the best way I found to illustrate and explain this problem is that back in 2.6 and 3.0-3.1-ish days my 8GB RAM would show almost nothing free, and massive (like 5GB) cached files (using top as the investigative method). I noticed with 3.4 that I'd always have around 1.5-2.0 GB free, and cached files would sit between 1.0-2.0. My working set (all normal apps) must be closer to 3 or 4GB of files.
With vfs_cache_pressure set to 100 in 3.4 it must be dropping cached files to try to leave a lot of open mem for apps. But it's nutso about it, as why would I ever need 2GB free mem at all times?
vfs_cache_pressure to 60 and now my free mem sits around 400-500MB, which is much better, with around 4GB file cache.
So anyone who works with tons of GB of files they want cached and upgrades to 3.4+ and notices a huge slowdown, set your vfs_cache_pressure to 60(ish) and your swappiness to 80(ish). (I haven't had to go to swappiness 100 yet.)
-
Gilbert E. Detillieux -
Trevor Cordes