I have some code (written in PHP) that checks on the status of a single "server" but I need this code to scale up to hundreds of parallel checks.
For example, you call the script like this:
http://mytest.com/checkserver.php?ip=10.11.12.13
where the ip=X.X.X.X is different every time depending on which server you are checking.
And the PHP looks something like this:
<?php
$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "http://%22.$ip.%22/status.html"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $output = curl_exec($ch); curl_close($ch);
?>
To my way of thinking, this should scale easily. Lets say 200 people hit the page at the same time all passing different "ip=" values, this should check all 200 different servers in parallel and return results.
It doesn't. I wrote this little bash script to test it (yes I'm using command line curl to test php-curl, please don't be confused by that): --- #!/bin/bash
IPS="10.18.136.20 10.18.136.21 10.18.136.22 ... ( a hundred more IPs)"
for IP in $IPS ; do curl -s http://mytest.com/checkserver.php?ip=$IP & done ---
All the curl commands launch into the background as you expect and I see all the apache child threads startup, but the results return one by one.
I can't for the life of me figure out how this can be possible. Each apache thread should run it's PHP in a separate thread and return in parallel. The only thing I can think of is that php is tracking all the requests as being part of the same session and imposing some limit on the outbound curl requests... Or maybe it's apache blocking it?
Just thought someone might have had to do something similar in the past and run into this.
By the way, if I go direct like this:
for IP in $IPS ; do curl -s http://%22.$IP%22/status.html & done
It works just exactly as you'd expect, all results return in parallel so it's got to be a problem with either PHP or apache.