Where did my disk space go? (df vs du, finding the culprit)

What we're doing

When you hit Start, you land on a Linux box whose disk is full. When you add up the big files you can see, they don't account for all the used space, so something is hidden. You'll use the two commands people reach for here, df and du, see why they sometimes disagree, and find the hidden file.

Watch the video first, then run these commands as you read. Every command and flag is explained. You're root on this box, so use sudo where shown.

Step 1: Which disk is full, and how bad?

df -h /

df means "disk free." -h makes the sizes human-readable (17G, 100%) instead of raw blocks. The / means "just the root filesystem."

Look at the Use% column. It's at 100%, so the disk is full. df tells you how full the disk is, but not what's filling it.

Now look at every disk:

df -h

Check the Mounted on column. A server can have several disks mounted at different paths (/, /boot, maybe /data), and you only want to clean the full one. Here it's /.

df asks the filesystem "how full are you?" It answers instantly, because the filesystem already tracks that number. It doesn't know which files are responsible.

Step 2: What's using the space?

du ("disk usage") walks through your folders and adds up what it finds:

sudo du -xh / 2>/dev/null | sort -h | tail -15

Breaking it down:

• du -xh / measures everything under /. -h is human sizes. -x keeps it on this one filesystem, so it won't wander into /proc and report junk.
• 2>/dev/null hides the error messages. (2> redirects the error stream, and /dev/null discards it.) Otherwise you get a lot of "Permission denied" lines for files you can't read.
• sort -h sorts by those human sizes, so 2G ranks above 900M.
• tail -15 shows the last 15 lines, so the biggest items end up at the bottom, next to your cursor.

du lists folders, not individual files. So you get a list of big directories (ignore base-image folders like /usr; the exact sizes depend on your disk):

...
2.7G   /var/backups
3.8G   /usr
4.2G   /var/log/legacy-app
4.3G   /var/log
7.5G   /var
12G    /

Two folders stand out. To see the files inside them, go one level deeper and add -a, which makes du list files too:

sudo du -ah /var/log/legacy-app /var/backups 2>/dev/null | sort -h | tail

2.7G   /var/backups/db-dump-2024.sql.bak
4.2G   /var/log/legacy-app/requests.log

A log file that grew too big, and a database dump someone left behind. Now you have the real paths, so clear them:

sudo rm /var/backups/db-dump-2024.sql.bak
sudo truncate -s 0 /var/log/legacy-app/requests.log    # empty it without deleting it
df -h /

truncate -s 0 sets the file's size to zero. It empties the log but keeps the file. That matters for logs, because an app might still be writing to one, and deleting a log while an app has it open is the trap in the next step. The dump is just a plain file, so rm is fine there.

Run df -h / again and Use% drops to about 59%. Looks fixed. It isn't.

du is the opposite of df: it counts the folders one by one. That's why it can point at a specific file. But it can only count files it can see in the folder tree.

Step 3: When df and du disagree

You freed real space, but the disk is still fuller than the files explain. Compare the two:

df -h /                       # still shows about 9.8G used (59%)
sudo du -xsh / 2>/dev/null    # adds up to only 4.4G

(-s means "summarize," so du gives one total for everything under / instead of a long list.)

df says 9.8G is used, but du only finds 4.4G. That's a 5.4G difference. Neither one is wrong. The only way this happens is if something is using space that has no name in any folder, so du never sees it.

A file was deleted while a program still had it open. The name is gone, so du can't see it, but the data is still on disk, so df still counts it. That hidden space is the ghost.

Step 4: Find the deleted file that's still open

lsof lists open files. One flag filters it down to the ones we want:

sudo lsof +L1

+L1 shows files with a link count below 1. Link count is how many names point at a file, so below 1 means no names: deleted, but still open.

COMMAND   PID  USER  FD   TYPE  ...  NLINK  NAME
systemd     1  root  25u  REG        0      /memfd:data-fd (deleted)
agetty    879  root  txt  REG        0      /usr/sbin/agetty (deleted)
sleep    7861  root   3w  REG        0      /var/log/telemetry/agent.log (deleted)

A few small rows (the systemd and agetty ones) are normal, so ignore them. The one that matters is the big one: a process holding /var/log/telemetry/agent.log, marked (deleted). That file is about 5.4G, which is exactly our missing space. Note the PID and the FD number (here 3).

A quick word on the FD, the file descriptor. When a program opens a file, the system gives it a small number to refer to it by, and that's the FD. Numbers 0, 1, and 2 are always input, output, and errors, so the first file a program opens is usually 3. So FD 3 is the handle this process is using to hold the deleted log open.

Step 5: Get the space back, no reboot

Two ways. Pick one.

Option A: empty the file through the open handle (keeps the service running):

sudo truncate -s 0 /proc/<PID>/fd/3      # use the PID from Step 4
df -h /

/proc/<PID>/fd/3 points straight at the file the process has open, even though it has no name anymore. Truncate it to zero and the space frees right away, while the service keeps running.

Option B: stop the process holding it:

sudo systemctl stop telemetry-agent      # or: sudo kill <PID>
df -h /

Either way, df drops back to a healthy number (about 27%). Run du -xsh / again and it now matches df. No reboot needed.

Other common places space hides

We cleared this box, but on a real server the cause could be somewhere else. Worth checking:

journald (systemd's logs, which grow over time if nobody limits them):

journalctl --disk-usage                 # how much the journal is using
sudo journalctl --vacuum-size=200M      # trim it to the most recent 200M
# sudo journalctl --vacuum-time=7d      # or keep only the last 7 days

Docker (often the biggest user on a build server):

docker system df                        # images, containers, volumes, build cache
docker system prune -a --volumes        # removes unused images and volumes, so check first

The apt cache and old kernels:

sudo du -sh /var/cache/apt              # -s gives one total for the folder
sudo apt clean                          # clears the downloaded .deb cache
sudo apt autoremove --purge             # removes unused packages and old kernels

Out of inodes, not bytes? Sometimes df -h shows free space but writes still fail. That happens when the filesystem runs out of inodes (one per file) from too many tiny files:

df -i        # shows inode usage; if IUse% is 100%, you have too many files, not too few bytes

Cheat sheet

df -h /                                    # 1. how full is it? (Use% and Mounted on)
sudo du -xh / 2>/dev/null | sort -h | tail # 2. which folders are biggest
sudo du -ah <folder> | sort -h | tail      #    add -a to find the actual files
sudo ncdu -x /                             #    or browse it (-x = stay on one filesystem)
sudo lsof +L1                              # 3. df bigger than du? find the deleted-but-open file
sudo truncate -s 0 /proc/<PID>/fd/<FD>     #    free it, no reboot
df -i                                      #    out of inodes instead of bytes?

The one thing to remember: when df shows more used than du can find, a deleted file is still open somewhere. lsof +L1 finds it, and truncate clears it.