I have a mystery hit on my apache server that is giving weird results that by config should never happen... and I can't figure it out. What's happening is very rarely a hit will "break out" of the dirs that I think all hits should be limited to. And I can't reproduce it with telnet tests.
Somehow a hit is getting access to /var/www/html/ when that should be impossible. Yes, it's my document root in the global area, but immediately in my virtualhost for that IP and port I redefine docroot to a sub dir of that. There should be no way someone can hit the site and get to the original docroot.
Yes, I am probably going to change the default global docroot to just be the same as my subdir one in the virthost, but I really want to figure out why the request can break out the way it is.
The only thing weird about the hit, based on my customized logging details, is that they are hitting a ServerAlias of the virthost, which is a bit odd because that particular alias isn't public knowledge (though it's not really hidden either). Oh ya, and the IP is in HK and is trying fuzzing attacks again me.
Here's the hit (IPs/hostnames changed): 1.2.3.4 - - [17/Feb/2020:02:40:07 -0600] "GET / HTTP/1.1" 403 199 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 80 9-w1.foo.com /var/www/html/
Note port 80, note the serveralias they hit us with, note the resulting directory /var/www/html
Here's the error it generates: [Mon Feb 17 14:43:13.444291 2020] [autoindex:error] [pid 16365:tid 140334311606016] [client 2.3.4.5:54603] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.phtml,index.html,index.phtml,index.php) found, and server-generated directory index forbidden by Options directive
It's ok it's forbidden (in fact, might have saved my bacon) as I have dirindex forbidden globally. The thing is no hit should ever be trying to find a index.html in /var/www/html!
When I try to recreate the hit to match the above manually with telnet, no matter what I do I can't get the serveralias to match their hit:
telnet foo.com GET / HTTP/1.1 Host: 9-w1.foo.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
Gives me a log result which matches what I expect the correct behaviour to be:
50.71.247.87 - - [17/Feb/2020:15:48:21 -0600] "GET / HTTP/1.1" 302 136 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 80 www.foo.com proxy:fcgi://localhost/var/www/html/Foo/Live/index.phtml
What on earth are the other parameters or headers they are passing in that results in a different result than my test? I'm truly stumped. I could try to capture packets of a hit, but the "attacks" only happen 2-3 times a day and always from different IPs in Asia.
Also, you've got the IP and you say they're persistent, tcpdump/tshark some packets to a file and see the contents of the request in more detail?
Theo
On Mon, Feb 17, 2020 at 5:33 PM athompso@athompso.net wrote:
First thought: what other hits come from that IP address previously? Could it be Redirect or rewrite? -Adam
On Feb. 17, 2020 16:29, Trevor Cordes trevor@tecnopolis.ca wrote:
I have a mystery hit on my apache server that is giving weird results that by config should never happen... and I can't figure it out. What's happening is very rarely a hit will "break out" of the dirs that I think all hits should be limited to. And I can't reproduce it with telnet tests.
Somehow a hit is getting access to /var/www/html/ when that should be impossible. Yes, it's my document root in the global area, but immediately in my virtualhost for that IP and port I redefine docroot to a sub dir of that. There should be no way someone can hit the site and get to the original docroot.
Yes, I am probably going to change the default global docroot to just be the same as my subdir one in the virthost, but I really want to figure out why the request can break out the way it is.
The only thing weird about the hit, based on my customized logging details, is that they are hitting a ServerAlias of the virthost, which is a bit odd because that particular alias isn't public knowledge (though it's not really hidden either). Oh ya, and the IP is in HK and is trying fuzzing attacks again me.
Here's the hit (IPs/hostnames changed): 1.2.3.4 - - [17/Feb/2020:02:40:07 -0600] "GET / HTTP/1.1" 403 199 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 80 9-w1.foo.com /var/www/html/
Note port 80, note the serveralias they hit us with, note the resulting directory /var/www/html
Here's the error it generates: [Mon Feb 17 14:43:13.444291 2020] [autoindex:error] [pid 16365:tid 140334311606016] [client 2.3.4.5:54603] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.phtml,index.html,index.phtml,index.php) found, and server-generated directory index forbidden by Options directive
It's ok it's forbidden (in fact, might have saved my bacon) as I have dirindex forbidden globally. The thing is no hit should ever be trying to find a index.html in /var/www/html!
When I try to recreate the hit to match the above manually with telnet, no matter what I do I can't get the serveralias to match their hit:
telnet foo.com GET / HTTP/1.1 Host: 9-w1.foo.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
Gives me a log result which matches what I expect the correct behaviour to be:
50.71.247.87 - - [17/Feb/2020:15:48:21 -0600] "GET / HTTP/1.1" 302 136 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 80 www.foo.com proxy:fcgi://localhost/var/www/html/Foo/Live/index.phtml
What on earth are the other parameters or headers they are passing in that results in a different result than my test? I'm truly stumped. I could try to capture packets of a hit, but the "attacks" only happen 2-3 times a day and always from different IPs in Asia. _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
Deeper into the rabbit hole...
I started logging a ton of extra stuff on each apache hit to get a better idea of what is going on with these weird hits, and finally after a 30 hour absence one arrived:
1.2.3.4 - - [20/Feb/2020:17:28:35 -0600] "GET / HTTP/1.1" 403 199 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" 80 foo.com /var/www/html/ HTTP/1.1 Xk8WIxeqhm968vI6R9xOIwAAAIY GET 443 1.2.3.4 1.2.3.4 1.1.1.2
My log setup: LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %p %v %f %H %L %m %{local}p %a %{c}a %A" foo
No, you don't need to wade through all of that. There's 2 interesting bits:
%A says they are hitting our 1.1.1.2 (ip changed to protect the innocent) when our main IP is 1.1.1.1! We don't even have any dns pointing to 1.1.1.2 any more. Yet apache is set to listen on * (all nics). Ok, not the end of the world, but interesting.
Doubly interesting is we don't have a 1.1.1.2:443 virthost in apache! Apache seems to listen to 1.1.1.2:443 anyhow.
Also note that %p ("canonical port") is saying the hit is on port 80 but %{local}p ("local port") says it's port 443! What what what? That's insane!
The clincher was my tcpdump on all interfaces on port 80 was *not* showing any matching traffic for these hits! So they must have been coming in on port 80.
So finally after all of that I've reproduced the problem with simple telnet of all things, after hammering away with openssl and gnutls-cli to no avail (and you'll see why)...
telnet 1.1.1.2 443
GET / HTTP/1.1 Host: foo.com
Voila, I get a log entry on can-port 80 (!!!), local port 443 and it hits my "forbidden" dir of /var/www/html.
So apache will happily talk http (port 80) on port 443, with no ssl/tls handshake/support, and instead of using my sole virthost on 443 (1.1.1.1:443) *or* the *:80 virthost (!!!) it uses no virthost at all and defaults to the global docroot! Ok, it sort of makes sense.
So my next task (ideas appreciated!) is to tell apache to not speak non-ssl on any 443 port. It already seems to do this on the 1.1.1.1:443 vhost, but not on the non-specified ips (!!). I could also be strict with my listens and interfaces to make sure I'm explicitly listing every interface and have matching virthosts for all of them. This may be complicated by the fact the extra ips are on the same physical interface. Lastly, I will still change my global docroot to point to some honeypot dir where I can easily grep for anomalous hits in the future. (Any other ideas?)
In theory, fudging with SNI should allow similar behavior, but I couldn't get our server to work trying to fudge with SNI options in gnutls-cli. Maybe our apache isn't working with SNI because only 1 :443 virthost exists, I'm not sure. Or maybe I'm not using it correct in the client.
The "\x16\x03\x01\x01L\x01" request string hits I reproduced by using gnutls-cli on the 1.1.1.2 IP which immediately barfs every time, probably because apache doesn't know to talk ssl on that ip on any port.
I find much of the above somewhat disturbing given that some of the behavior choices are quite odd and/or dangerous. I'm sure I'm not the only one with a config like this.
On 2020-02-21 athompso@athompso.net wrote:
Pretty sure this wouldn't have happened with nginx :-D.
Hahaha. Maybe!
Seriously, why not just make the TLS Virthost *:443 to both cover this scenario _and_ enable SNI simultaneously? Is there any harm in people using the other IP address? -Adam
Well, we were and will in the future use the 2nd/3rd IP addresses in apache again. It's just at the moment we are not. If I fix it that way now, then I enable the 2nd IP again in the future, I'll still have the same bug problem on the 3rd IP.
As for SNI... what is everyone's opinion of that? Is it "here" now in the sense that 99.999% of end-user browsers will support it? Put another way, does FB and google require SNI support to hit their sites? On our production server we don't want to lock out any user just because they don't happen to support SNI. (Yes, TLS limitations will probably bite people before SNI limitations...)
On 2020-02-21 12:42, Trevor Cordes wrote:
On 2020-02-21 athompso@athompso.net wrote:
Pretty sure this wouldn't have happened with nginx :-D.
Hahaha. Maybe!
Seriously, why not just make the TLS Virthost *:443 to both cover this scenario _and_ enable SNI simultaneously? Is there any harm in people using the other IP address? -Adam
Well, we were and will in the future use the 2nd/3rd IP addresses in apache again. It's just at the moment we are not. If I fix it that way now, then I enable the 2nd IP again in the future, I'll still have the same bug problem on the 3rd IP.
As for SNI... what is everyone's opinion of that? Is it "here" now in the sense that 99.999% of end-user browsers will support it? Put another way, does FB and google require SNI support to hit their sites? On our production server we don't want to lock out any user just because they don't happen to support SNI. (Yes, TLS limitations will probably bite people before SNI limitations...)
SNI went mainstream (i.e. >90% client support) several years ago, and yeah, I guess we're probably at the >>99% mark by now? Literally XP SP3 w/IE7 is the last thing I know of that doesn't support SNI. Or Android 2.1, and I don't think any of those devices are still alive.
-Adam
I'm hosting hundreds of websites using SNI and nobody ever complains. It has definitely been mainstream for years now.
On 2020-02-21 13:21, Adam Thompson wrote:
On 2020-02-21 12:42, Trevor Cordes wrote:
On 2020-02-21 athompso@athompso.net wrote:
Pretty sure this wouldn't have happened with nginx :-D.
Hahaha. Maybe!
Seriously, why not just make the TLS Virthost *:443 to both cover this scenario _and_ enable SNI simultaneously? Is there any harm in people using the other IP address? -Adam
Well, we were and will in the future use the 2nd/3rd IP addresses in apache again. It's just at the moment we are not. If I fix it that way now, then I enable the 2nd IP again in the future, I'll still have the same bug problem on the 3rd IP.
As for SNI... what is everyone's opinion of that? Is it "here" now in the sense that 99.999% of end-user browsers will support it? Put another way, does FB and google require SNI support to hit their sites? On our production server we don't want to lock out any user just because they don't happen to support SNI. (Yes, TLS limitations will probably bite people before SNI limitations...)
SNI went mainstream (i.e. >90% client support) several years ago, and yeah, I guess we're probably at the >>99% mark by now? Literally XP SP3 w/IE7 is the last thing I know of that doesn't support SNI. Or Android 2.1, and I don't think any of those devices are still alive.
-Adam _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
Doh. I can also confirm that you can exploit this "flaw" to read any file in /var/www/html and its subdirs even if other virthost <Location> and <Directory> rules forbid it. Further, php files get spit out verbatim (as source) without execution. However, you have to guess the exact file paths/names. Luckily I had dirindexes turned off globally!
I guess the moral of the story is global docroot should never point to anywhere that has real files when you use virthosts for everything. However, once I change global docroot, I'll have to make sure every global setting that applies to docroot and below will be duplicated in the virthosts, as they may no longer apply to the subdirs... I'll have to look into that.
Also, having all dir definitions outside of virthosts would have helped. I like to keep things nested though as it makes more sense to me to have dirs inside the only virthosts they can be accessed by.
All this plus the explicit listens on only certain IPs has solved it. Plus, I realized that newer apaches added support for adding "https" to the end of a Listen to force that Listen line (port) to only talk https and not allow it to pretend it's port 80.
You would need to put the restrictions in the global section outside of the virtualhost sections to deal with this.
On 2020-02-23 00:15, Trevor Cordes wrote:
Doh. I can also confirm that you can exploit this "flaw" to read any file in /var/www/html and its subdirs even if other virthost <Location> and <Directory> rules forbid it. Further, php files get spit out verbatim (as source) without execution. However, you have to guess the exact file paths/names. Luckily I had dirindexes turned off globally!
I guess the moral of the story is global docroot should never point to anywhere that has real files when you use virthosts for everything. However, once I change global docroot, I'll have to make sure every global setting that applies to docroot and below will be duplicated in the virthosts, as they may no longer apply to the subdirs... I'll have to look into that.
Also, having all dir definitions outside of virthosts would have helped. I like to keep things nested though as it makes more sense to me to have dirs inside the only virthosts they can be accessed by.
All this plus the explicit listens on only certain IPs has solved it. Plus, I realized that newer apaches added support for adding "https" to the end of a Listen to force that Listen line (port) to only talk https and not allow it to pretend it's port 80. _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
On 2020-02-17 athompso@athompso.net wrote:
First thought: what other hits come from that IP address previously? Could it be Redirect or rewrite? -Adam
The pattern is 2-3 fuzz hits that get 4xx codes like: 1.2.3.4 - - [17/Feb/2020:14:59:28 -0600] "\x16\x03\x01" 400 226 "-" "-" 80 9-w1.foo.com -
Then the hit that breaks into /var/www/html
On 2020-02-17 Theodore Baschak wrote:
Also, you've got the IP and you say they're persistent, tcpdump/tshark some packets to a file and see the contents of the request in more detail?
I get 4-5 hits total from a single IP, then no more from that IP. Then a while later it'll be the same pattern from another IP. I have dozens of these groups of hits logged, always the similar sequence. Sometimes they just do the \x code hits and not the breakout hit.
Probably a bot net causing this.
So I can't easily dump these packets, at least not based on IP. This is a very busy production server so I'm not sure I want to turn on global port 80 packet capture... although, most traffic is port 443, so maybe it is an option.
I'm also looking into logging more of the request. There doesn't seem a way to log all headers, but I can log specific ones.
If you’re up to adding and configuring it, ModSecurity and the community rule set can provide a lot of information. Besides actively preventing some attacks, you can log complete requests, ideally only for the weird traffic.
On Feb 17, 2020, at 7:03 PM, Trevor Cordes trevor@tecnopolis.ca wrote:
On 2020-02-17 athompso@athompso.net wrote:
First thought: what other hits come from that IP address previously? Could it be Redirect or rewrite? -Adam
The pattern is 2-3 fuzz hits that get 4xx codes like: 1.2.3.4 - - [17/Feb/2020:14:59:28 -0600] "\x16\x03\x01" 400 226 "-" "-" 80 9-w1.foo.com -
Then the hit that breaks into /var/www/html
On 2020-02-17 Theodore Baschak wrote: Also, you've got the IP and you say they're persistent, tcpdump/tshark some packets to a file and see the contents of the request in more detail?
I get 4-5 hits total from a single IP, then no more from that IP. Then a while later it'll be the same pattern from another IP. I have dozens of these groups of hits logged, always the similar sequence. Sometimes they just do the \x code hits and not the breakout hit.
Probably a bot net causing this.
So I can't easily dump these packets, at least not based on IP. This is a very busy production server so I'm not sure I want to turn on global port 80 packet capture... although, most traffic is port 443, so maybe it is an option.
I'm also looking into logging more of the request. There doesn't seem a way to log all headers, but I can log specific ones. _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
On 2020-02-17 Tim Lavoie wrote:
If you’re up to adding and configuring it, ModSecurity and the community rule set can provide a lot of information. Besides actively preventing some attacks, you can log complete requests, ideally only for the weird traffic.
Thanks for that, I'm looking into it. I did try leaving a tcpdump going on port 80 after confirming we get very little traffic on it. I was right, 99.9% of our traffic is 443 now.
Of course, it ran for 24 hours and this is the first span of 24 hours where the attackers/probers didn't trigger the behavior in weeks. Sigh. On the bright side, that should mean they hit it in the next few hours...
Now I'm also trying to figure out why 2 similarly configured apache's respond differently to CONNECT and OPTION methods... Probers seem to like to test CONNECT for open proxies... next up: restrict all my servers to just GET POST HEAD. The fun never ends!
-
Adam Thompson -
athompso＠athompso.net
-
Scott Toderash -
Theodore Baschak -
Tim Lavoie -
Trevor Cordes