Ncdu(1). It's in EPEL among other places, and IIRC it's not too hard to compile if you absolutely must. Its defaults are sane, but check out the options especially "-x". -Adam

Get Outlook for Androidhttps://aka.ms/AAb9ysg ________________________________ From: Roundtable roundtable-bounces@muug.ca on behalf of Chris Audet cj.audet@gmail.com Sent: Sunday, February 5, 2023 2:56:04 PM To: roundtable@muug.ca roundtable@muug.ca Subject: [RndTbl] Best ways to find where disk space is being used?

I've got a fairly long lived CentOS server that stubbornly stopped installing updates because the HDD is full.

Can someone share their favourite way to determine where disk space is being used up on a system?

For example, on Windows I'd use Wiztree/Treesize/Windirstat. On Linux desktop I've been using Gnome Disk Usage Analyzer (aka Baobab)https://wiki.gnome.org/Apps/DiskUsageAnalyzer.

But I'm not sure what the best solutions are in cases where there's no GUI available. I could always mount / over SSH and use Baobab to crawl the remote filesystem, but that seems less than optimal 🤔

[root@dogmeat ~]# yum update Loaded plugins: fastestmirror, versionlock Loading mirror speeds from cached hostfile * base: mirror.csclub.uwaterloo.cahttp://mirror.csclub.uwaterloo.ca * epel: ftp.cse.buffalo.eduhttp://ftp.cse.buffalo.edu * extras: mirror.xenyth.nethttp://mirror.xenyth.net * updates: mirror.csclub.uwaterloo.cahttp://mirror.csclub.uwaterloo.ca Excluding 5 updates due to versionlock (use "yum versionlock status" to show them) Resolving Dependencies --> Running transaction check ---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.el7_9.10 will be updated ---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.el7_9.13 will be an update ---> Package bind-libs.x86_64 32:9.11.4-26.P2.el7_9.10 will be updated ---> Package bind-libs.x86_64 32:9.11.4-26.P2.el7_9.13 will be an update ---> Package bind-libs-lite.x86_64 32:9.11.4-26.P2.el7_9.10 will be updated ---> Package bind-libs-lite.x86_64 32:9.11.4-26.P2.el7_9.13 will be an update ---> Package bind-license.noarch 32:9.11.4-26.P2.el7_9.10 will be updated ---> Package bind-license.noarch 32:9.11.4-26.P2.el7_9.13 will be an update ---> Package bind-utils.x86_64 32:9.11.4-26.P2.el7_9.10 will be updated ---> Package bind-utils.x86_64 32:9.11.4-26.P2.el7_9.13 will be an update ---> Package dkms.noarch 0:3.0.9-2.el7 will be updated ---> Package dkms.noarch 0:3.0.10-1.el7 will be an update ---> Package httpd.x86_64 0:2.4.6-97.el7.centos.5 will be updated ---> Package httpd.x86_64 0:2.4.6-98.el7.centos.6 will be an update ---> Package httpd-tools.x86_64 0:2.4.6-97.el7.centos.5 will be updated ---> Package httpd-tools.x86_64 0:2.4.6-98.el7.centos.6 will be an update ---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.352.b08-2.el7_9 will be updated ---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.362.b08-1.el7_9 will be an update ---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.352.b08-2.el7_9 will be updated ---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.362.b08-1.el7_9 will be an update ---> Package kernel.x86_64 0:3.10.0-1160.83.1.el7 will be installed ---> Package kernel-devel.x86_64 0:3.10.0-1160.83.1.el7 will be installed ---> Package kernel-headers.x86_64 0:3.10.0-1160.81.1.el7 will be updated ---> Package kernel-headers.x86_64 0:3.10.0-1160.83.1.el7 will be an update ---> Package kernel-tools.x86_64 0:3.10.0-1160.81.1.el7 will be updated ---> Package kernel-tools.x86_64 0:3.10.0-1160.83.1.el7 will be an update ---> Package kernel-tools-libs.x86_64 0:3.10.0-1160.81.1.el7 will be updated ---> Package kernel-tools-libs.x86_64 0:3.10.0-1160.83.1.el7 will be an update ---> Package python-perf.x86_64 0:3.10.0-1160.81.1.el7 will be updated ---> Package python-perf.x86_64 0:3.10.0-1160.83.1.el7 will be an update ---> Package sudo.x86_64 0:1.8.23-10.el7_9.2 will be updated ---> Package sudo.x86_64 0:1.8.23-10.el7_9.3 will be an update ---> Package xorg-x11-server-Xvfb.x86_64 0:1.20.4-19.el7_9 will be updated ---> Package xorg-x11-server-Xvfb.x86_64 0:1.20.4-21.el7_9 will be an update ---> Package xorg-x11-server-common.x86_64 0:1.20.4-19.el7_9 will be updated ---> Package xorg-x11-server-common.x86_64 0:1.20.4-21.el7_9 will be an update --> Finished Dependency Resolution --> Running transaction check ---> Package kernel.x86_64 0:3.10.0-1160.45.1.el7 will be erased ---> Package kernel-devel.x86_64 0:3.10.0-1160.45.1.el7 will be erased --> Finished Dependency Resolution

Dependencies Resolved

================================================================================ Package Arch Version Repository Size ================================================================================ Installing: kernel x86_64 3.10.0-1160.83.1.el7 updates 52 M kernel-devel x86_64 3.10.0-1160.83.1.el7 updates 18 M Updating: bind-export-libs x86_64 32:9.11.4-26.P2.el7_9.13 updates 1.1 M bind-libs x86_64 32:9.11.4-26.P2.el7_9.13 updates 158 k bind-libs-lite x86_64 32:9.11.4-26.P2.el7_9.13 updates 1.1 M bind-license noarch 32:9.11.4-26.P2.el7_9.13 updates 92 k bind-utils x86_64 32:9.11.4-26.P2.el7_9.13 updates 262 k dkms noarch 3.0.10-1.el7 epel 85 k httpd x86_64 2.4.6-98.el7.centos.6 updates 2.7 M httpd-tools x86_64 2.4.6-98.el7.centos.6 updates 94 k java-1.8.0-openjdk x86_64 1:1.8.0.362.b08-1.el7_9 updates 317 k java-1.8.0-openjdk-headless x86_64 1:1.8.0.362.b08-1.el7_9 updates 33 M kernel-headers x86_64 3.10.0-1160.83.1.el7 updates 9.1 M kernel-tools x86_64 3.10.0-1160.83.1.el7 updates 8.2 M kernel-tools-libs x86_64 3.10.0-1160.83.1.el7 updates 8.1 M python-perf x86_64 3.10.0-1160.83.1.el7 updates 8.2 M sudo x86_64 1.8.23-10.el7_9.3 updates 844 k xorg-x11-server-Xvfb x86_64 1.20.4-21.el7_9 updates 857 k xorg-x11-server-common x86_64 1.20.4-21.el7_9 updates 57 k Removing: kernel x86_64 3.10.0-1160.45.1.el7 @updates 64 M kernel-devel x86_64 3.10.0-1160.45.1.el7 @updates 38 M

Transaction Summary ================================================================================ Install 2 Packages Upgrade 17 Packages Remove 2 Packages

Total size: 144 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test

Transaction check error: installing package python-perf-3.10.0-1160.83.1.el7.x86_64 needs 23MB on the / filesystem installing package sudo-1.8.23-10.el7_9.3.x86_64 needs 26MB on the / filesystem installing package kernel-3.10.0-1160.83.1.el7.x86_64 needs 106MB on the / filesystem installing package bind-export-libs-32:9.11.4-26.P2.el7_9.13.x86_64 needs 109MB on the / filesystem

Error Summary ------------- Disk Requirements: At least 109MB more space needed on the / filesystem.

[root@dogmeat ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.8G 0 3.8G 0% /dev tmpfs 3.9G 148K 3.9G 1% /dev/shm tmpfs 3.9G 11M 3.8G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/mapper/centos_ba--bog--v-root 41G 40G 355M 100% / /dev/sda1 497M 346M 151M 70% /boot /dev/mapper/centos_ba--bog--v-home 20G 99M 20G 1% /home tmpfs 779M 0 779M 0% /run/user/0

On Sunday, February 5, 2023 2:56:04 P.M. CST Chris Audet wrote:

I've got a fairly long lived CentOS server that stubbornly stopped installing updates because the HDD is full.

Can someone share their favourite way to determine where disk space is being used up on a system?

For example, on Windows I'd use Wiztree/Treesize/Windirstat. On Linux desktop I've been using Gnome Disk Usage Analyzer (aka Baobab) https://wiki.gnome.org/Apps/DiskUsageAnalyzer.

But I'm not sure what the best solutions are in cases where there's no GUI available. I could always mount / over SSH and use Baobab to crawl the remote filesystem, but that seems less than optimal 🤔

I use this method from the command line. As root, `cd /` and issue the following command:

find . -maxdepth 1 -type d 2>&1 -print0 | grep -zv '^.$' | xargs -0 du -sm | sort -rn | more

The directory with the greatest usage appears first. `cd` into it and issue the above command again. Clean directories and large files, then repeat as needed.

Two things to note:

1. Some programs in Linux do a trick where they allocate a file and then delete it while keeping the file open. The inode remains busy and the space isn't freed up until the process terminates. The advantage to this is other processes can't open the file to look into its contents. The disadvantage is you can't see the file using 'ls'. However, such files show up in 'lsof' with the tag '(deleted)'.

2. A file system can report "full" if it runs out of inodes. This used to be a problem on old, small systems, but probably isn't any more because file systems today tend to be very large and have loads of spare inodes. The command "df -i" shows the inode counts.

Brian

On 2023-02-05 Chris Audet wrote:

I've got a fairly long lived CentOS server that stubbornly stopped installing updates because the HDD is full.

All great ideas from others here. Essentially all amount to the same thing. I've written my own "du-dirs" ages ago that basically does the same thing.

When disk is full I du-dirs in a likely-culprit dir (usually /var or /home) and start there. Then pick the biggest one and cd/du-dirs in that one; repeat. /var/logs is a great cheap/quick place to immediately start hosing stuff when in "we're 100% disk on production" panic mode.

ll -Srh is your friend once you're in the dir.

Oh ya, and my personal favorite (might not work on super ancient rpm):

rpm -qa --queryformat="%10{SIZE}\t%{NAME}\n" | sort -k1,1n

followed by dnf remove of the biggest one that doesn't look critical (though never include -y until you see the deps!!).

What I mainly wanted to add, though, is an idea to address part of the root cause of your problem: your box probably has enough space for the updates to be installed, just not enough space for that plus the temp space for the rpm files themselves.

I finally ran into a situation for a dist-upgrade type of thing with dnf where it was going to be impossible clear up enough space on / to house both all the rpms and the installed files... at least as how dnf was calculating it.

Luckily dnf (and maybe yum??) have a temp-dir option where you can redirect all of the rpms to be placed on another fs where you have lots of space. This cuts down the space required for any updates (esp system upgrades) drastically.

dnf system-upgrade download --releasever=36 --allowerasing --downloaddir=/some/other/fs/updates-tmp

Conceivably one could modify their update-cron-thing to make sure that option was always present (maybe they let you put it in the .conf).

Of course, that doesn't solve the root issue of other things slowly filling up your disk, but it would probably postpone the "argh" moment by a few days/weeks.

Note: there appears to still be a bug in this option: you must make & specify a subdir of your temp fs area because some guy said --downloaddir=/home and dnf proceeded to update his whole system and then rm -rf /home. Hahahaha.