RHEL9.5 box, fully updated.
On a working system with no relevant (systemd/php/apache) updates in 2
weeks, all of a sudden php-fpm process(es) cannot write to its systemd
PrivateTmp (forced by default) /tmp/systemd-private-*-php-fpm.service-*
This caused fatal errors in the lowest level of a php site during
simple logging, as tempnam was returning false when it really never
should.
I try really hard to work within the systemd paradigm on many things,
especially when on paper it sounds helpful, like PrivateTmp. And it
worked fine for many years. But now it suddenly does this and screws over
a live web site.
So it's PrivateTmp=false for me now! I've written all scripts using
proper tmp file naming techniques, like php's tempnam (which uses secure
code under the hood, or so I'm told). And there's no local users on the
box anyhow to start trying to race symlinks, and no user-supplied input is
used to create file names. So the whole idea of PrivateTmp to me is
"meh".
But I want to try to understand what happened here. There are a few
oddities. systemd seems to make 2 privatetmp dirs in /tmp for php-fpm(?),
yet when this bug hit only 1 existed(?). I suppose systemd-tmpfiles-clean
or something may have screwed up and removed the dir? Everything is set
to defaults for clean. And the dir seems to have existed via my in-script
tests (see below).
Most weirdly, the bug only hit one of the php-fpm pools we run! The other
main one seemed to be able to write just fine!!?? This is especially
confusing since there is only one "php-fpm: master process". There does
not seem to be a separate master for each pool, so how could one pool work
fine and not the other? There are separate pool worker processes,
however. I guess php-fpm could somehow "speak" to systemd to tell it it
needs 2 privatetmps, one for each pool? But I don't see how that could
happen. Maybe the 2 dirs is a red herring. My personal box doesn't have
more than 1 dir for php-fpm, and I have 3 pools.
I added debugs to my scripts while the problem was still happening just to
confirm the scripts were seeing the tmp dir properly and it was writable,
etc. And it was: drwx as detected by php's is_writable, etc. And the
subdirs systemd creates are rwxt (sticky, like normal /tmp). However, I
couldn't figure out a way for the script to figure out what its real
tmpdir was, as it is tricked by systemd into believe the subdir is just
/tmp. However, the script saw a dir and thought it was writable and
showed me its inode data with ls -l.
Since I cannot reproduce the bug easily now that I've restarted php-fpm,
and now that I've disabled PrivateTmp because I need 100% trustworthy
tools on a production box, I will probably never know what really
happened. That is why I plead for MUUGers to pick their brains trying to
figure out how a script can see a dir inode and detect it's drwx by the
current user and yet any write to it fails.... I know systemd is doing
crazy wacky new-fangled mount or namespaces or cgroup or whatever tricks
to make PrivateTmp work at all. (And it's slightly evil that there can be
a dir sitting right there I can ls at the command line and all perms show
I can write to it and yet even root cannot!)
_______________________________________________
Roundtable mailing list -- roundtable(a)muug.ca
To unsubscribe send an email to roundtable-leave(a)muug.ca
