A disk full alert wakes you up at three in the morning. You SSH into the server, run df -h, and see that the root partition sits at one hundred percent capacity. The server stops accepting new writes. Databases throw disk I/O errors. Background workers crash. Finding out what consumed every byte of storage takes a systematic approach using standard command-line tools. You don't have time to guess. You need precise commands that isolate the problem fast.
Basic disk monitoring tools only give you the top-level view. They tell you that a partition is full. They don't show you which subdirectory or hidden file holds the offending payload. Production environments accumulate garbage in unexpected places like temp directories, abandoned build caches, and unmonitored log streams. If you skip the right diagnostic steps, you'll end up deleting the wrong things. Or rebooting a server that just needs a simple file handle released.
Finding Space with du and ncdu
Your first instinct is likely running du to find large directories. Unguided, du scans every file on the filesystem and floods your terminal with lines you cannot read quickly. You need to sort the output or target specific mount points to get meaningful data. It's easy to waste precious minutes waiting for recursive scans to finish on slow disks.
Run this command to check the immediate subdirectories of the root partition, sorted by size:
sudo du -h --max-depth=1 / | sort -hrThe flags matter here. The --max-depth=1 flag keeps the command from descending into every subfolder. The sort -hr flags sort the human-readable sizes in reverse order. If you want a more interactive experience, install ncdu using your package manager. It gives you a curses-based interface. You can navigate directories, see exact space usage, and delete files directly without dropping back to the shell.
Many engineers forget that du measures disk usage while df measures block allocation. If you mount an overlay filesystem or use network storage, the numbers might not match up cleanly. Always verify you're scanning the actual mount point rather than a masked directory inside an empty mount.
Handling Deleted but Open Files
Sometimes you find large directories using du. When you check the contents, nothing adds up. You deleted a massive log file yesterday, yet the free space on the disk did not change. This happens because a running process still has an open file descriptor pointing to that deleted file. The Linux kernel keeps the blocks allocated on disk until every process closes the file handle. It's a classic trap that catches even senior engineers off guard.
You can list all deleted files that are still held open by processes using the lsof utility. Pipe the output to grep to filter for deleted entries and inspect what's lingering in the background.
sudo lsof +L1The output shows the process ID, the command name, the file descriptor, and the size of the open file. Instead of rebooting the entire server to force the kernel to release those blocks, restart the specific service that holds the file open. Once the process terminates, the operating system reclaims the disk space immediately. You can manage these long-running background services effectively by following Systemd Services for Long-Running Processes to ensure they handle file closures correctly during restarts.
Processes can trap gigabytes of data in unlinked states. Always check open handles before assuming your directory scans are broken.
Identifying Uncontrolled Log Growth
Applications and system daemons write logs continuously. When log rotation fails or isn't configured, these files expand until they consume every available sector. Check your /var/log directory and any application-specific log paths for oversized files. You should also review how your services manage log output to prevent future outages. Proper log rotation prevents these files from growing infinitely, as explained in Mastering Logrotate Configuration.
If a log file is actively written to by a daemon and you need to truncate it right now without stopping the service, do not delete it with the rm command. Instead, empty the file contents directly from the shell:
sudo truncate -s 0 /var/log/application.logThis command shrinks the file size to zero bytes while keeping the inode intact. The running application continues writing to the same file descriptor without throwing errors. Your disk space returns instantly. If you accidentally delete the file instead of truncating it, the process keeps writing to the invisible file descriptor. Disk space remains trapped until you restart that specific daemon. Silent disk exhaustion will strike again if cron jobs or logrotate timers fail to run.
