Deeper into the rabbit hole...
I started logging a ton of extra stuff on each apache hit to get a better idea of what is going on with these weird hits, and finally after a 30 hour absence one arrived:
1.2.3.4 - - [20/Feb/2020:17:28:35 -0600] "GET / HTTP/1.1" 403 199 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" 80 foo.com /var/www/html/ HTTP/1.1 Xk8WIxeqhm968vI6R9xOIwAAAIY GET 443 1.2.3.4 1.2.3.4 1.1.1.2
My log setup: LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %p %v %f %H %L %m %{local}p %a %{c}a %A" foo
No, you don't need to wade through all of that. There's 2 interesting bits:
%A says they are hitting our 1.1.1.2 (ip changed to protect the innocent) when our main IP is 1.1.1.1! We don't even have any dns pointing to 1.1.1.2 any more. Yet apache is set to listen on * (all nics). Ok, not the end of the world, but interesting.
Doubly interesting is we don't have a 1.1.1.2:443 virthost in apache! Apache seems to listen to 1.1.1.2:443 anyhow.
Also note that %p ("canonical port") is saying the hit is on port 80 but %{local}p ("local port") says it's port 443! What what what? That's insane!
The clincher was my tcpdump on all interfaces on port 80 was *not* showing any matching traffic for these hits! So they must have been coming in on port 80.
So finally after all of that I've reproduced the problem with simple telnet of all things, after hammering away with openssl and gnutls-cli to no avail (and you'll see why)...
telnet 1.1.1.2 443
GET / HTTP/1.1 Host: foo.com
Voila, I get a log entry on can-port 80 (!!!), local port 443 and it hits my "forbidden" dir of /var/www/html.
So apache will happily talk http (port 80) on port 443, with no ssl/tls handshake/support, and instead of using my sole virthost on 443 (1.1.1.1:443) *or* the *:80 virthost (!!!) it uses no virthost at all and defaults to the global docroot! Ok, it sort of makes sense.
So my next task (ideas appreciated!) is to tell apache to not speak non-ssl on any 443 port. It already seems to do this on the 1.1.1.1:443 vhost, but not on the non-specified ips (!!). I could also be strict with my listens and interfaces to make sure I'm explicitly listing every interface and have matching virthosts for all of them. This may be complicated by the fact the extra ips are on the same physical interface. Lastly, I will still change my global docroot to point to some honeypot dir where I can easily grep for anomalous hits in the future. (Any other ideas?)
In theory, fudging with SNI should allow similar behavior, but I couldn't get our server to work trying to fudge with SNI options in gnutls-cli. Maybe our apache isn't working with SNI because only 1 :443 virthost exists, I'm not sure. Or maybe I'm not using it correct in the client.
The "\x16\x03\x01\x01L\x01" request string hits I reproduced by using gnutls-cli on the 1.1.1.2 IP which immediately barfs every time, probably because apache doesn't know to talk ssl on that ip on any port.
I find much of the above somewhat disturbing given that some of the behavior choices are quite odd and/or dangerous. I'm sure I'm not the only one with a config like this.