Just upgraded to F30 with kernel 5.3.12 (not going to F31 quite yet). Hit a weird error that didn't happen in the previous kernel/OS:
If my dir has stickybit set, in this case /tmp, then root can't use tcsh's ">!" redirection on a file owned by another user (permission denied). F29 ">!" allowed it just fine (kernel 5.3.11). I checked and tcsh versions didn't change, except just the bump to F30.
">!" in tcsh is the same as > in bash. tcsh lets you set an option to >! require explicit instruction to clobber existing files so you don't accidentally > an existing file without realizing it. That's the extra ! after the >.
It's funny, because root can still append to the file (>>), rm it, chmod, etc., just not this explicit clobber.
Funnier still is it works in bash (using >) but the OS is still giving it the permission denied. (See below.)
It looks like tcsh is using creat() and that is the syscall returning the error.
as root:
tcsh sudo -u trevor touch /tmp/t echo foo >> /tmp/t # ok echo foo >! /tmp/t # Permission denied.
strace on 5.3.12 F30 (error):
dup(19) = 0 fcntl(0, F_SETFD, 0) = 0 dup2(17, 1) = 1 dup2(18, 2) = 2 creat("t", 0666) = -1 EACCES (Permission denied) write(18, "t: Permission denied.\n", 22l: Permission denied. ) = 22
strace on 5.3.11 F29 (ok):
dup(19) = 0 fcntl(0, F_SETFD, 0) = 0 dup2(17, 1) = 1 dup2(18, 2) = 2 creat("t", 0666) = 3 fcntl(3, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE) fcntl(3, F_SETFL, O_WRONLY|O_LARGEFILE) = 0 ...
strace with bash on 5.3.12 F30 (no error! note the double try!) (command for bash is just: echo foo > /tmp/t)
openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied) openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_TRUNC) = 3 fcntl(1, F_GETFD) = 0 fcntl(1, F_DUPFD, 10) = 10 ...
So bash uses openat instead of creat, but still gets EACCESS, but then retries with slightly different options... almost like bash hit this same bug but has already worked around it.
So is this a kernel bug / change in API? Should I report this to kernel bugs or to tcsh bugs? This is a "bug" in the sense that something changed because I've been doing this type of command for, oh, 20 years, and it's always worked before. And I know Linus doesn't like to change the existing user interface into the kernel.
The strace on bash is more revealing than the one on tcsh, because of the retry on openat() calls...
openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied) openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_TRUNC) = 3
The first call (ignoring the use of openat vs open or creat) is essentially equivalent to what would happen with the creat call in tcsh. The key difference between that and the second call is the use of O_CREAT (and the following mode argument). However, that should have no effect on an existing file, and only come into play when a new file is actually created.
The fact that there's different behaviour between the two openat calls, and the fact that you're getting EACCES when uid==0 (which should almost never happen, especially on file access), and the fact that this is a new error, suggests a kernel bug to me.
It's possible that it's a "feature", but this would seem misguided to me. (Without seeing the actual code diffs, it's hard to say for sure.) But I can't think of any logical reason why you'd get an EACCES error on this as root. (Unless it's the convoluted logic of SELinux that's coming into play?... I'd expect the second call to fail the same way in that case, but no guarantees if the kernel code paths are different.)
The bash vs tcsh differences (the retried openat vs using creat) could be due to a number of reasons, and may make sense in other instances where the O_CREAT could lead to errors (perhaps more likely for non-root users). I wouldn't assume it's a specific workaround for the new kernel behaviour, without seeing comments (or online discussion by the maintainers) to that effect.
Gilbert
On 2019-11-29 5:07 a.m., Trevor Cordes wrote:
Just upgraded to F30 with kernel 5.3.12 (not going to F31 quite yet). Hit a weird error that didn't happen in the previous kernel/OS:
If my dir has stickybit set, in this case /tmp, then root can't use tcsh's ">!" redirection on a file owned by another user (permission denied). F29 ">!" allowed it just fine (kernel 5.3.11). I checked and tcsh versions didn't change, except just the bump to F30.
">!" in tcsh is the same as > in bash. tcsh lets you set an option to >! require explicit instruction to clobber existing files so you don't accidentally > an existing file without realizing it. That's the extra ! after the >.
It's funny, because root can still append to the file (>>), rm it, chmod, etc., just not this explicit clobber.
Funnier still is it works in bash (using >) but the OS is still giving it the permission denied. (See below.)
It looks like tcsh is using creat() and that is the syscall returning the error.
as root:
tcsh sudo -u trevor touch /tmp/t echo foo >> /tmp/t # ok echo foo >! /tmp/t # Permission denied.
strace on 5.3.12 F30 (error):
dup(19) = 0 fcntl(0, F_SETFD, 0) = 0 dup2(17, 1) = 1 dup2(18, 2) = 2 creat("t", 0666) = -1 EACCES (Permission denied) write(18, "t: Permission denied.\n", 22l: Permission denied. ) = 22
strace on 5.3.11 F29 (ok):
dup(19) = 0 fcntl(0, F_SETFD, 0) = 0 dup2(17, 1) = 1 dup2(18, 2) = 2 creat("t", 0666) = 3 fcntl(3, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE) fcntl(3, F_SETFL, O_WRONLY|O_LARGEFILE) = 0 ...
strace with bash on 5.3.12 F30 (no error! note the double try!) (command for bash is just: echo foo > /tmp/t)
openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied) openat(AT_FDCWD, "/tmp/t", O_WRONLY|O_TRUNC) = 3 fcntl(1, F_GETFD) = 0 fcntl(1, F_DUPFD, 10) = 10 ...
So bash uses openat instead of creat, but still gets EACCESS, but then retries with slightly different options... almost like bash hit this same bug but has already worked around it.
So is this a kernel bug / change in API? Should I report this to kernel bugs or to tcsh bugs? This is a "bug" in the sense that something changed because I've been doing this type of command for, oh, 20 years, and it's always worked before. And I know Linus doesn't like to change the existing user interface into the kernel.
On 2019-11-29 Gilbert E. Detillieux wrote:
The fact that there's different behaviour between the two openat calls, and the fact that you're getting EACCES when uid==0 (which should almost never happen, especially on file access), and the fact that this is a new error, suggests a kernel bug to me.
Yes, I noticed the O_CREAT difference in the bash strace too. Very interesting.
I should have mentioned, selinux is in disabled mode on this box (and confirmed with selinuxenabled command) so that's eliminated.
I agree that EACCES should not be a thing when running as root. In fact, I'm a firm believer in "root being able to do everything" on Linux and always hate it when they move more towards Windows "Administrator can't always do everything" model. (Like they did with some of the new capabilities a while ago.) Adam vehemently argues the opposite :-)
I'll file a kernel bug and hit up the mailing list now that you've provided a second sober opinion confirming my suspicions.
Thanks!!
On 2019-11-29 23:39, Trevor Cordes wrote:
I agree that EACCES should not be a thing when running as root. In fact, I'm a firm believer in "root being able to do everything" on Linux [...] Adam vehemently argues the opposite :-)
I most certainly do not! I think SELinux is an abomination whose entire concept needs to die a fiery death, and I'm not exactly stoked about Capabilities, either. What I have said is that many large organizations, especially those subject to periodic audits and/or who need check-box compliance for e.g. PCI et al., *NEED* those capabilities so that they can continue using UNIX.
When you get into a hierarchical, centrally-managed model, things change again - there are valid use cases for wanting the local system "owner" to have root access and be able to do most root-ish things, but not absolutely everything. Network configuration is a common example - it's pretty hard to trust your network when you can't trust that all the systems on it have correct network settings. And if you give the average programmer root access, they WILL screw up the network in some new and interesting never-before-seen way. Go ahead, ask me how I know :-).
Most of the time, technical users "need root" in order to start a daemon that binds to a low port. Or maybe to change printer configuration. There's a lot of crap that still can't be effectively managed by non-root users on modern UNIX desktops. (MacOS being the obvious exception here.) If I want to give a programmer a laptop running Fedora, and have them use it everywhere and anywhere ('cuz, you know, it's a LAPTOP) then I pretty much have no choice but to give them root access to manage networks and printers and displays/peripherals. That's often undesirable.
That's not to say the superuser paradigm is at fault - all those things COULD very well be accomplished without root permissions (not even sudo!), if the relevant subsystems had been designed with that in mind. I'm still perfectly OK with the superuser paradigm, I blame the mid-layers for most of its *supposed* shortcomings.
-Adam
After much kernel bisecting by me that yielded nothing of value, it turns out the bug isn't the kernel, it's a change in Fedora's default sysctl.conf settings between F29 and F30 that enable a new-ish kernel "feature". The "feature" turns on this behavior.
Thanks to Andrew Morton and especially Al Viro for figuring this out for me as I'm pretty sure a sysctl of some obscure feature would have been the last place I would have looked!
The solution is: echo 0 >> /proc/sys/fs/protected_regular
The new feature is (and it may be systemd deciding this):
* The fs.protected_regular and fs.protected_fifos sysctls, which were added in Linux 4.19 to make some data spoofing attacks harder, are now enabled by default. While this will hopefully improve the security of most installations, it is technically a backwards incompatible change; to disable these sysctls again, place the following lines in /etc/sysctl.d/60-protected.conf or a similar file: fs.protected_regular = 0 fs.protected_fifos = 0
The bz is: https://bugzilla.kernel.org/show_bug.cgi?id=205727
See also...
https://www.spinics.net/lists/fedora-devel/msg252452.html
Thanks, Trevor, for bringing this to our attention. I was not even aware of these new sysctl settings and kernel features. I can see why they'd be desirable from a security perspective, but it does break compatibility, possibly for some legitimate but obscure use cases.
Gilbert
On 2019-12-30 11:57 p.m., Trevor Cordes wrote:
After much kernel bisecting by me that yielded nothing of value, it turns out the bug isn't the kernel, it's a change in Fedora's default sysctl.conf settings between F29 and F30 that enable a new-ish kernel "feature". The "feature" turns on this behavior.
Thanks to Andrew Morton and especially Al Viro for figuring this out for me as I'm pretty sure a sysctl of some obscure feature would have been the last place I would have looked!
The solution is: echo 0 >> /proc/sys/fs/protected_regular
The new feature is (and it may be systemd deciding this):
- The fs.protected_regular and fs.protected_fifos sysctls, which were added in Linux 4.19 to make some data spoofing attacks harder, are now enabled by default. While this will hopefully improve the security of most installations, it is technically a backwards incompatible change; to disable these sysctls again, place the following lines in /etc/sysctl.d/60-protected.conf or a similar file: fs.protected_regular = 0 fs.protected_fifos = 0
The bz is: https://bugzilla.kernel.org/show_bug.cgi?id=205727
And see also this potentially useful tutorial, which mentions these sysctl settings, among other tips...
https://blog.frehi.be/2019/01/30/linux-security-hardening-recommendations/
Gilbert
On 2020-01-02 11:22 a.m., Gilbert E. Detilllieux wrote:
See also...
https://www.spinics.net/lists/fedora-devel/msg252452.html
Thanks, Trevor, for bringing this to our attention. I was not even aware of these new sysctl settings and kernel features. I can see why they'd be desirable from a security perspective, but it does break compatibility, possibly for some legitimate but obscure use cases.
Gilbert
On 2019-12-30 11:57 p.m., Trevor Cordes wrote:
After much kernel bisecting by me that yielded nothing of value, it turns out the bug isn't the kernel, it's a change in Fedora's default sysctl.conf settings between F29 and F30 that enable a new-ish kernel "feature". The "feature" turns on this behavior.
Thanks to Andrew Morton and especially Al Viro for figuring this out for me as I'm pretty sure a sysctl of some obscure feature would have been the last place I would have looked!
The solution is: echo 0 >> /proc/sys/fs/protected_regular
The new feature is (and it may be systemd deciding this):
- The fs.protected_regular and fs.protected_fifos sysctls, which were
added in Linux 4.19 to make some data spoofing attacks harder, are now enabled by default. While this will hopefully improve the security of most installations, it is technically a backwards incompatible change; to disable these sysctls again, place the following lines in /etc/sysctl.d/60-protected.conf or a similar file: fs.protected_regular = 0 fs.protected_fifos = 0
The bz is: https://bugzilla.kernel.org/show_bug.cgi?id=205727
On 2020-01-02 Gilbert E. Detilllieux wrote:
See also...
https://www.spinics.net/lists/fedora-devel/msg252452.html
Thanks, Trevor, for bringing this to our attention. I was not even aware of these new sysctl settings and kernel features. I can see why they'd be desirable from a security perspective, but it does break compatibility, possibly for some legitimate but obscure use cases.
Thanks for the links. I see where they are coming from, but it's a bludgeon to try to fix the forever-bugs of thoughtless tmp file/dir usage by programmers, which I don't think is as prevalent today as it used to be.
Systemd already "solved" it for daemons by shadow-masking safe/unique dirs for /tmp usage (ugh, hate that too!).
I guess these new sysctls are trying to solve it for the rest of the programs out there. When really each program should be using well established tmpdir routines available in every single language that solved these problems eons ago.
I see the point, but it irks me they have to do this at all instead of getting programmers who get their code put into distros to just do it the right way.
fs.protected_regular = 0 on all my boxes now! Well, I guess until someone smartens up tcsh like they did bash so I can still use >!
;-)
-
Adam Thompson -
Gilbert E. Detillieux -
Gilbert E. Detilllieux
-
Trevor Cordes