(A real question this time!)
I have a web (+ other tasks) server that sometimes does a lot of tasks in a short time that spike the load up to large values (like 30). The tasks are all long jobs of lots (tens of thousands) of little (10/s) jobs. I've written code to, between each little job, check the /proc/loadavg load (1 and 5 min) and insert usleeps (.5+ secs) based on the loadavg (exponential relationship). My goal is to keep the load < 2.0 on a 2 core system.
Anyhow, it seems to be working fine. The weird thing is, at 2.0 loadavg, looking at top, I see the CPUs mostly at idle. Not even very much wait going on. Why would the load be 2 when both CPUs are mostly at 5-10% including wait?
I basically want to make sure the system doesn't get overloaded and can respond to non-batch request (like a web hit) promptly. What loadavg should I target if not (1.0 * numcores)? I guess I can arbitrarily pick numbers and see what happens, but I like things like this to make sense, and to me 1.0 * numcores makes sense, unless I'm not understanding loadavg properly.
Thanks!
On 2012-01-12 15:28, Trevor Cordes wrote:
(A real question this time!)
I have a web (+ other tasks) server that sometimes does a lot of tasks in a short time that spike the load up to large values (like 30). The tasks are all long jobs of lots (tens of thousands) of little (10/s) jobs. I've written code to, between each little job, check the /proc/loadavg load (1 and 5 min) and insert usleeps (.5+ secs) based on the loadavg (exponential relationship). My goal is to keep the load< 2.0 on a 2 core system.
Anyhow, it seems to be working fine. The weird thing is, at 2.0 loadavg, looking at top, I see the CPUs mostly at idle. Not even very much wait going on. Why would the load be 2 when both CPUs are mostly at 5-10% including wait?
My guess is that your overall processing is disk-bound. Load average includes not only processes in in the run queue, but those waiting for disk I/O (normally, fairly short waits). The MUUG server sees similar loads when a lot of people are hammering at the mirror file system (over http, ftp and/or rsync), with relatively low CPU percentages.
I basically want to make sure the system doesn't get overloaded and can respond to non-batch request (like a web hit) promptly. What loadavg should I target if not (1.0 * numcores)? I guess I can arbitrarily pick numbers and see what happens, but I like things like this to make sense, and to me 1.0 * numcores makes sense, unless I'm not understanding loadavg properly.
Yeah, I think you'll have to experiment, since it depends on how much of your load is disk vs CPU. I would certainly make sure to set the target threshold higher than 2, and probably lower than 10. Maybe start with 4, and if your CPU usage is still low, bring it up toward 8.
YMMV!
Gilbert
On 2012-01-12 Gilbert E. Detillieux wrote:
My guess is that your overall processing is disk-bound. Load average includes not only processes in in the run queue, but those waiting for disk I/O (normally, fairly short waits). The MUUG server sees similar loads when a lot of people are hammering at the mirror file system (over http, ftp and/or rsync), with relatively low CPU percentages.
Could be. For each mini-job there's a select on a big db, some little indexed selects and a few tiny db writes.
Yeah, I think you'll have to experiment, since it depends on how much of your load is disk vs CPU. I would certainly make sure to set the target threshold higher than 2, and probably lower than 10. Maybe start with 4, and if your CPU usage is still low, bring it up toward 8.
Ya, I think a 3 or 4 load threshold setting might work better. I'll play around a bit with it.
On 2012-01-12 John Lange wrote:
Perhaps setting a "nice" value on the processes might be a easier solution?
My long jobs are [io]niced fully. But, they are heavily mysql dependent and I can't [io]nice mysqld. So I can't fully rely on nice here. And I find that in practice, even maximum [io]nice doesn't nice things as much as you'd want. Nice'd ps's still can grab a huge chunk of resources. At least ionice gives you some cool "never run unless idle" options, though I don't usually use them :-)
On 2012-01-12 Sean Walberg wrote:
see high load average and low CPU it means you've got high I/O. Watching "vmstat 2" usually confirms this with blocked processes on the left.
At loadavg 2, vmstat 2 shows about 80% 0 b's, 15% 1 b's and 5% 2 b's.
From what I can gather, that seems ok.
I'm wondering if usleep works differently from sleep? Not necessarily a busy loop but it doesn't go to sleep to wait for a signal?
The usleep definitely is working, and I'm pretty sure linux turned it into a non-busyloop ages ago.
The problem isn't that I'm not throttling enough, my current setup has the load well under control. It's that I seem to be throttling too much and not utilizing the system to (near) capacity.
That's why I'm trying to understand why what (top/vmstat) appears to be a lightly loaded system is giving me a loadavg of 2.
On 2012-01-12 Sean Walberg wrote:
aware. There's a Ruby gem called Stalker that makes it really easy to run a worker daemon that runs the commands you want (bindings in just about every language out there, too) Then it's just a matter of running the number of worker instances, say one per CPU.
Could look at that later, but I've already done all the work and have a successful native-code throttler :-) I'm just trying to tweak now.
Of course, after this is all done, I'll play with optimizing the rest of my code. The big goal here (which I think I've been successful at) is stopping the 40 loadavgs that start dropping connections.
On 2012-01-12 Adam Thompson wrote:
A) only run as many jobs in parallel as you have cores, and
The box is running a lot of different jobs / types of jobs. Luckily most are "batchy" and now have my throttling in them. However, users outside our control can start big jobs at will and they need to start running pretty much immediately ("look, we're doing something!"). So throttling based on load seems to be the only (best?) solution.
Thanks to all for input!!
That seems like a good idea but it's probably tricky to implement in practice.
Since the load average is a 1, 5, and 15 minute average and CPU is a snapshot in realtime, CPU could easily be idle but it would take a while for the load average to drop back down.
Perhaps setting a "nice" value on the processes might be a easier solution?
Load average is an exponentially weighted average of the number of processes in the run queue. Strange that you're not seeing much iowait, though maybe not everything shows up there. Usually if you see high load average and low CPU it means you've got high I/O. Watching "vmstat 2" usually confirms this with blocked processes on the left.
I'm wondering if usleep works differently from sleep? Not necessarily a busy loop but it doesn't go to sleep to wait for a signal?
BTW if you're trying to manage jobs, something like beanstalkd might be more efficient than managing your own load, plus it's network aware. There's a Ruby gem called Stalker that makes it really easy to run a worker daemon that runs the commands you want (bindings in just about every language out there, too) Then it's just a matter of running the number of worker instances, say one per CPU.
Sean
On Thu, Jan 12, 2012 at 3:28 PM, Trevor Cordes trevor@tecnopolis.ca wrote:
(A real question this time!)
I have a web (+ other tasks) server that sometimes does a lot of tasks in a short time that spike the load up to large values (like 30). The tasks are all long jobs of lots (tens of thousands) of little (10/s) jobs. I've written code to, between each little job, check the /proc/loadavg load (1 and 5 min) and insert usleeps (.5+ secs) based on the loadavg (exponential relationship). My goal is to keep the load < 2.0 on a 2 core system.
Anyhow, it seems to be working fine. The weird thing is, at 2.0 loadavg, looking at top, I see the CPUs mostly at idle. Not even very much wait going on. Why would the load be 2 when both CPUs are mostly at 5-10% including wait?
I basically want to make sure the system doesn't get overloaded and can respond to non-batch request (like a web hit) promptly. What loadavg should I target if not (1.0 * numcores)? I guess I can arbitrarily pick numbers and see what happens, but I like things like this to make sense, and to me 1.0 * numcores makes sense, unless I'm not understanding loadavg properly.
Thanks! _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-
Gilbert E. Detillieux -
John Lange -
Sean Walberg -
Trevor Cordes