As a follow-up to the opera-ate-my-inodes problem discussed at the meeting, I checked a few things out and realized that this surely is a problem not limited to bad (or lazy) admin choices, for the following reasons:
1. People with separate /home partitions aren't spared: what if an app or malicious user writes to /var/tmp, which on 99% of systems is going to be on the same partition as /. So a separate /home won't spare you from / running out of inodes: all it takes is one sticky-bit dir in the whole tree where a non-root can write to have one non-root user/app starve the system of inodes.
2. The thought that you should create your fs's with massively more inodes than your usage pattern could ever use won't help either, as the user/app could create 0 byte files that eat an inode but never a fs block. You'd run out of inodes before you hit the reserved-root-du limit no matter how many inodes you created at fs creation time. (Not to mention the non-insignificant loss of usable disk space that occurs if you allocate many orders of magnitude more inodes than you will ever need.)
3. Quotas might be able to help, but I'm not talking about public systems here, I'm talking about my own personal home desktop. Surely no one is running quotas against themselves on their personal home system? And surely that can't be offered as a "sane" solution the average Joe is supposed to implement.
OK, here's the weirdest part: I just checked and my / is ext4! I thought Adam mentioned ext4 dynamically creates inode out of thin air (and I thought that too) like XFS? So why did I run out of inodes on a non-full (df) ext4 fs?
If this "bug" indeed affects ext4 then for sure I am filing a bug report. If it only affected "legacy" fs's like ext3 then I probably would just let it pass (and switch to ext4 if I hadn't already).
Thanks for everyone's input!
On 13-12-12 04:21 PM, Trevor Cordes wrote:
- People with separate /home partitions aren't spared: what if an app
or malicious user writes to /var/tmp, which on 99% of systems is going to be on the same partition as /. So a separate /home won't spare you from / running out of inodes: all it takes is one sticky-bit dir in the whole tree where a non-root can write to have one non-root user/app starve the system of inodes.
Playing devil's advocate here: ( / /var /tmp /etc /usr /home ) is the typically-recommended set of partitions. I will point out, however, that most systems set up this way are far more sensitive to running out of file-creation-ability on /var than anywhere else, so your point actually remains valid. (Also, "ln -s /var/tmp /tmp" is fairly common in those setups.)
OK, here's the weirdest part: I just checked and my / is ext4! I thought Adam mentioned ext4 dynamically creates inode out of thin air (and I thought that too) like XFS? So why did I run out of inodes on a non-full (df) ext4 fs?
Actually, what I meant is that it didn't pre-reserve space on disk for the inodes, and only uses up space in the partition as needed, not that it would keep creating inodes out of thin air. Comparatively, ext2/ext3 (IIRC) pre-allocated space in the partition for all the inodes that it would ever need. Not 100% certain of this without checking.
Overall, yes, I agree, there *ought* to be a limit on inodes like there is on disk space, but... even more important would be if inode exhaustion generated a different error than ENOSPC. Um... I just read errno(3), and I take that back: I would never want to be responsible for adding *more* error codes to that list.
On 2013-12-13 Adam Thompson wrote:
Playing devil's advocate here: ( / /var /tmp /etc /usr /home ) is the typically-recommended set of partitions.
Not anymore, not on Linux. I don't know of a single distro, not even enterprise-grade ones, that do this by default during install, or even recommend it in their deployment guides. Yes, this used to be recommended on big-iron, at places like the U of M, but no one realistically does this anymore on standalone desktops or servers (unless for reasons of network-boot, xterms, etc). I'd say having that many partitions would make one as weird as me with my tuning of inode quantities to match my historic usage patterns!
point actually remains valid. (Also, "ln -s /var/tmp /tmp" is fairly common in those setups.)
Fedora, and thus Red Hat (and probably others), are explicitly moving to a shm /tmp and a disk-based /var/tmp, emphatically snubbing the old "ln -s /var/tmp /tmp" paradigm.
Mind you, I'm not saying I agree with them, I'm just reporting the news. (The moment I found out /tmp was shm I changed it back to a real disk partition.)
Actually, what I meant is that it didn't pre-reserve space on disk for the inodes, and only uses up space in the partition as needed, not that it would keep creating inodes out of thin air.
Ah! Thanks for clarifying. As an aside, I am 100% sure XFS does dynamic inode quantity and space allocation, meaning you have unlimited inodes (until disk space runs out). It makes df -i rather interesting (and useless). Not that I'd ever contemplate changing my / to XFS!!!
exhaustion generated a different error than ENOSPC. Um... I just read errno(3), and I take that back: I would never want to be responsible for adding *more* error codes to that list.
:-)
On 12/12/2013 4:21 PM, Trevor Cordes wrote:
- Quotas might be able to help, but I'm not talking about public
systems here, I'm talking about my own personal home desktop. Surely no one is running quotas against themselves on their personal home system? And surely that can't be offered as a "sane" solution the average Joe is supposed to implement.
Quotas should indeed be able to help with this. You can limit the number of blocks and the number of files allowed per user per volume. Yes, it would be odd to implement quotas on your own personal system, but if it helps to treat a misbehaving application the way you would a misbehaving user, that is an option that's available to you.
It's my understanding that even without quotas, most file systems will reserve the last n % of blocks for root, to prevent non-root users from exhausting all blocks on a volume. I find it a bit surprising that they don't seem to do likewise with the last several inodes.
-
Adam Thompson -
Gilles Detillieux -
Trevor Cordes