On 2023-02-05 Chris Audet wrote:
I've got a fairly long lived CentOS server that stubbornly stopped installing updates because the HDD is full.
All great ideas from others here. Essentially all amount to the same thing. I've written my own "du-dirs" ages ago that basically does the same thing.
When disk is full I du-dirs in a likely-culprit dir (usually /var or /home) and start there. Then pick the biggest one and cd/du-dirs in that one; repeat. /var/logs is a great cheap/quick place to immediately start hosing stuff when in "we're 100% disk on production" panic mode.
ll -Srh is your friend once you're in the dir.
Oh ya, and my personal favorite (might not work on super ancient rpm):
rpm -qa --queryformat="%10{SIZE}\t%{NAME}\n" | sort -k1,1n
followed by dnf remove of the biggest one that doesn't look critical (though never include -y until you see the deps!!).
What I mainly wanted to add, though, is an idea to address part of the root cause of your problem: your box probably has enough space for the updates to be installed, just not enough space for that plus the temp space for the rpm files themselves.
I finally ran into a situation for a dist-upgrade type of thing with dnf where it was going to be impossible clear up enough space on / to house both all the rpms and the installed files... at least as how dnf was calculating it.
Luckily dnf (and maybe yum??) have a temp-dir option where you can redirect all of the rpms to be placed on another fs where you have lots of space. This cuts down the space required for any updates (esp system upgrades) drastically.
dnf system-upgrade download --releasever=36 --allowerasing --downloaddir=/some/other/fs/updates-tmp
Conceivably one could modify their update-cron-thing to make sure that option was always present (maybe they let you put it in the .conf).
Of course, that doesn't solve the root issue of other things slowly filling up your disk, but it would probably postpone the "argh" moment by a few days/weeks.
Note: there appears to still be a bug in this option: you must make & specify a subdir of your temp fs area because some guy said --downloaddir=/home and dnf proceeded to update his whole system and then rm -rf /home. Hahahaha.