On 2012-11-21 Gilbert E. Detillieux wrote:
You might also be interested in this article...
http://www.admin-magazine.com/Articles/Tuning-Your-Filesystem-s-Cache/
Specifically, the part about the vmtouch command and the "-dl" (daemonize and lock) option. It would allow you to lock your large file(s) into memory while you work on them, I think. That way, other system activity won't evict your cached pages.
That's good to know for sure, but in this case my files are in thousands of little files (maildir emails) so locking certain files isn't going to help.
I finally had a chance to poke around more and I think I found the solution!!
/proc/sys/vm/vfs_cache_pressure
Defaults to 100 (what that value means appears arbitrary). I set mine to 60 and all of a sudden my files are staying cached nearly forever like they used to. There *must* have been a change to either the default setting or what the number means to the kernel around 3.3 or 3.4. It's like night and day.
Probably the best way I found to illustrate and explain this problem is that back in 2.6 and 3.0-3.1-ish days my 8GB RAM would show almost nothing free, and massive (like 5GB) cached files (using top as the investigative method). I noticed with 3.4 that I'd always have around 1.5-2.0 GB free, and cached files would sit between 1.0-2.0. My working set (all normal apps) must be closer to 3 or 4GB of files.
With vfs_cache_pressure set to 100 in 3.4 it must be dropping cached files to try to leave a lot of open mem for apps. But it's nutso about it, as why would I ever need 2GB free mem at all times?
vfs_cache_pressure to 60 and now my free mem sits around 400-500MB, which is much better, with around 4GB file cache.
So anyone who works with tons of GB of files they want cached and upgrades to 3.4+ and notices a huge slowdown, set your vfs_cache_pressure to 60(ish) and your swappiness to 80(ish). (I haven't had to go to swappiness 100 yet.)