Linux command: du

1. The Basics

du walks a directory tree and adds up how much space the files in it occupy. That sentence contains three assumptions worth taking apart, because each one is where people go wrong.

1.1 The Simplest Possible Use

Two options do ninety percent of the work, and you will almost always want both:

$ du -sh /var/log
1.4G    /var/log

Without -s you get a line for every directory underneath, which on a big tree is thousands of lines scrolling past. Without -h you get a number in 1024-byte blocks, which nobody wants to divide in their head.

The single most useful invocation in this whole article is this one, and it is worth committing to memory:

$ du -h -d 1 /var | sort -h -r
8.2G    /var
6.1G    /var/lib
1.4G    /var/log
403M    /var/cache
...

-d 1 limits the output to one level down, and sort -h -r puts the biggest first. Run that, step into the biggest directory, run it again. Three or four rounds and you have found whatever ate the disk.

1.2 du Counts Blocks, Not Bytes

Here is the first assumption to abandon. du does not report how many bytes are in your files. It reports how much space the filesystem allocated to hold them, and those are different numbers.

$ echo -n "x" > tiny.txt          # a one-byte file

$ du --apparent-size -b tiny.txt
1       tiny.txt                  # one byte of content

$ du tiny.txt
4       tiny.txt                  # four KILOBYTES on disk

$ stat -c '%s bytes, %b blocks of %B' tiny.txt
1 bytes, 8 blocks of 512

A filesystem hands out space in blocks, typically 4096 bytes on ext4. A one-byte file gets a whole block, and 4095 bytes of it are wasted. That is not a bug, it is how block allocation works, and it means a directory of many tiny files costs far more than the sum of their contents. Section 7.4 puts numbers on that.

Newer filesystems complicate this in both directions. Extents, inline data for very small files, and tail packing can all pack things more tightly than the flat block model suggests, while compression can make the allocated size smaller than the content. The 4 KB rule is a good default mental model for ext4 and a starting point elsewhere, not a law.

One more thing is an object with a size of its own: the directory. It has to store the names it contains, so it costs space even when it is empty, and it grows as you fill it:

$ mkdir emptydir
$ du -h emptydir
4.0K    emptydir                  # empty, and already 4 KB

$ du -h --apparent-size emptydir
0       emptydir                  # no content at all

$ for i in $(seq 1 3000); do : > emptydir/file-with-a-longish-name-$i; done
$ stat -c '%s bytes' emptydir
167936                            # the directory itself is now 164 KB of names

So du is not adding up file contents. It is adding up allocated space across every object it walks, directories included, which is another small reason its total will not match a sum of file sizes.

1.3 du and df Answer Different Questions

This is the distinction the whole article turns on. Say it out loud once and a lot of confusion disappears:

The speed difference is not subtle, and it follows directly from the method:

$ du -sxh /usr/share       0.68 s        # 207,983 files, every one stat'ed
$ df -h /                  0.00 s        # one question to the filesystem

The right mental model: df asks the filesystem how much room is left. du asks the files how much room they take. When those two disagree, the difference is made of things du cannot see, and section 7.1 is a catalogue of them.

3. A Short History

Both commands are as old as Unix itself. Not "early Unix": the actual first release.

The GNU versions credit their authors, and the overlap between the two lists is not a coincidence:

$ man du | grep -A2 AUTHOR
       Written by Torbjorn Granlund, David MacKenzie, Paul Eggert, and
       Jim Meyering.

$ man df | grep -A2 AUTHOR
       Written by Torbjorn Granlund, David MacKenzie, and Paul Eggert.

One historical detail still shows up in daily use. The default unit is 1024 bytes, but POSIX specifies 512, so the same command gives you numbers twice as large in a POSIX-strict environment:

$ du tiny.txt
4       tiny.txt

$ POSIXLY_CORRECT=1 du tiny.txt
8       tiny.txt                  # same file, 512-byte blocks

And POSIXLY_CORRECT is not the only variable involved. Three more change the unit, and du honours whichever it finds first:

$ du -s target
20484   target                    # the default, 1024-byte blocks

$ DU_BLOCK_SIZE=1M du -s target
21      target
$ BLOCK_SIZE=1M du -s target
21      target
$ BLOCKSIZE=1M du -s target
21      target
$ POSIXLY_CORRECT=1 du -s target
40968   target                    # 512-byte blocks: twice the number

So a bare number from du depends on four environment variables, any of which might be set in a cron environment, a CI runner or somebody's .bashrc. This is why -h, -k or -B is not optional in a script. The number has no unit attached to it, and the unit is decided by an environment you may not control.

The other portability question is which options exist at all. Most of what makes GNU du pleasant is a GNU extension, and inside a minimal container image you get BusyBox instead. Tested against BusyBox 1.38:

The -b row is worth a moment, because the two spellings are not interchangeable in units. On a 50 MB sparse file, du --apparent-size prints 51200 (in 1 KB blocks) while du -b prints 52428800 (bytes). GNU defines -b as --apparent-size --block-size=1, and BusyBox behaves the same way, so -b is the portable choice as long as you expect bytes.

Note the depth rows too, because they matter for what you type every day. -d 1 works everywhere; --max-depth=1 does not. They do exactly the same thing, so there is no reason to learn the long one. That is why every example in this article uses -d.

$ docker run --rm busybox du --max-depth=1 /t
du: unrecognized option '--max-depth=1'

$ docker run --rm busybox du -d 1 /t
ok

macOS and the BSDs are a third dialect, with their own spelling for some of this. On anything that might not be GNU, stick to -s, -h, -k, -x, -a and -d, and check the local manual page for the rest.

5. Moderate Use Cases

5.1 Excluding Things

--exclude takes a glob and skips anything matching it. This matters more than it sounds, because on a real server the interesting number is usually "how big is this, ignoring the parts I already know about":

$ du -sh /var/www/site --exclude='*/cache'
$ du -sh /home/peter --exclude='.cache' --exclude='.local/share/Trash'
$ du -sh /srv/project --exclude='node_modules' --exclude='.git'

For more than two or three patterns, put them in a file and use -X:

$ cat skip.txt
node_modules
.git
*.iso

$ du -sh -X skip.txt /srv/project

5.2 Thresholds

-t (short for threshold) hides everything below a size, which turns a wall of output into a short list:

$ du -h -d 3 -t 1G /var          # only things over a gigabyte
$ du -ah -t 100M /home/peter     # only files over 100 MB

A negative threshold inverts it, which is the less obvious half and occasionally exactly right:

$ du -ah -t -1k /srv/data        # only entries SMALLER than 1K

That last one is how you find the directory full of a hundred thousand tiny files, which section 7.4 explains is a real problem and not just untidiness.

5.3 Depth Without Recursion: -S

-S (short for separate-dirs) reports each directory's own contents without adding in its subdirectories. Compare against the default:

$ du -h sub                     $ du -Sh sub
2.1M    sub/a/b/c               2.1M    sub/a/b/c
4.1M    sub/a/b                 2.1M    sub/a/b
6.1M    sub/a                   2.1M    sub/a
8.1M    sub                     2.1M    sub

The left column accumulates; the right column does not. Use -S when you want to know which single directory holds the files, rather than which branch of the tree is heaviest. They are different questions and people often get the wrong one.

5.4 Counting Files Instead of Bytes

A filesystem can run out of two separate things: space, and inodes. When it runs out of inodes you get "No space left on device" while df -h cheerfully reports plenty free. --inodes finds the culprit using the same walk:

$ du -sh many
12K     many                     # 500 empty files: almost no space

$ du -s --inodes many
501     many                     # but 501 inodes

$ du --inodes -d 1 /var | sort -n -r | head

Check whether that is your problem before you go hunting for big files:

$ df -i /                        # -i for inodes
Filesystem       Inodes  IUsed  IFree IUse% Mounted on
/dev/mapper/...    120M    14M   106M   12% /

Mail queues, session directories and small-file caches are the usual suspects. A hundred thousand one-byte files use almost no space and a hundred thousand inodes.

5.5 The Recipe

Put together, this is the sequence for "the disk is full and I do not know why":

$ df -h                          # 1. which filesystem, and is it really full?
$ df -i                          # 2. or has it run out of inodes instead?
$ du -shx /* 2>/dev/null | sort -h -r | head       # 3. top level
$ du -h -d 1 -x /var | sort -h -r | head           # 4. descend into the winner
$ du -h -d 1 -x /var/lib | sort -h -r | head       # 5. and again
$ du -ah -t 100M /var/lib/docker | sort -h -r      # 6. finally, the files

Steps 1 and 2 take a second and rule out the two problems that look identical from a distance. Steps 3 to 6 are the same command with a different path each time. Section 6.1 explains the -x.

7. Something Most Users Do Not Know

7.1 Why df and du Disagree

This is the question that brings people to this command, and the answer is not one thing. It is four, and they are cumulative.

Reserved blocks. Start with df disagreeing with itself. On this machine, its own three columns do not add up:

$ df --block-size=1 /
size  = 2011859927040
used  = 1808341385216
avail =  101246103552
used + avail = 1909587488768
missing      =  102272438272 bytes, which is 5.1% of size

Around five percent of an ext4 filesystem is reserved for root by default. It is not free space as far as you are concerned, and it is not used by any file, so it belongs to neither column. That is why a disk can report "100% full" while several gigabytes are technically still there. You can reclaim it on a data volume, where nothing needs the safety margin:

$ sudo tune2fs -m 1 /dev/sdb1     # reduce the reserve from 5% to 1%

Do not do that on the root filesystem. The reserve exists so that a full disk does not stop root logging in and syslog writing, which is exactly when you need both.

Deleted files that are still open. This is the big one, and it is the reason for most dramatic disagreements. Unlinking a file removes its name; the space comes back only when the last process holding it open lets go:

$ dd if=/dev/zero of=ghost.bin bs=1M count=300
$ exec 9< ghost.bin               # hold it open
$ rm ghost.bin                    # the name is gone

$ du -sh .
11M     .                          # du sees nothing: there is no name to walk

df used before rm  : 1724868 MB
df used after  rm  : 1724868 MB   # df has not moved
df after closing   : 1724568 MB   # 300 MB back, at once

A log file deleted while the application still has it open is the classic version. The disk stays full until you restart the service, and no amount of du will show you the file. Find them like this:

$ sudo lsof -nP +L1               # +L1: link count below 1, meaning deleted
$ sudo lsof -nP | grep deleted

The fix is to restart the process holding the descriptor, or to truncate through /proc if you cannot: : > /proc/PID/fd/N.

Files hidden under a mount point. If a directory had files in it before something was mounted over it, those files still exist and still take space. They are simply unreachable, so du cannot count them:

$ sudo mkdir /mnt/tmp-check
$ sudo mount --bind / /mnt/tmp-check     # see the real root, unobscured
$ sudo du -shx /mnt/tmp-check/var        # count what is actually there
$ sudo umount /mnt/tmp-check

This one catches people who filled a directory before mounting a disk on it, which is easy to do during installation and invisible afterwards.

Filesystem overhead. Inode tables, journals and metadata occupy space that belongs to no file. On a large filesystem this is gigabytes, and it explains a permanent small gap rather than a sudden one.

And on a modern filesystem, the accounting itself. Everything above assumes ext4, where a block belongs to exactly one file. Btrfs, ZFS and XFS break that assumption in ways that make the question harder rather than the answer wrong:

This machine runs ext4, where those do not apply: cp --reflink=always here fails with "Operation not supported". But on Btrfs or ZFS they are typically the dominant cause of a df/du gap, and no pathname-walking tool can account for them. Use the filesystem's own tools instead: btrfs filesystem usage, btrfs qgroup show, or zfs list -o space.

Put together: df counts everything the filesystem has committed. du counts what it can reach by walking names. The gap is reserved space, plus files with no name, plus files with no path, plus the filesystem's own bookkeeping, plus whatever your filesystem shares or compresses behind the scenes.

7.2 Throwing Away stderr Makes du Lie

Everybody writes 2>/dev/null to silence the permission warnings. Watch what that does to the answer:

$ du -sh /root 2>/dev/null
4.0K    /root                     # looks like an empty directory

$ du -sh /root
du: cannot read directory '/root': Permission denied
4.0K    /root                     # same number, now with a warning

That 4.0K is the directory entry itself. /root on this machine contains a Desktop folder and a snap directory, and neither is in the total. The number is not an error, it is an honest report of what du could see, and the part that told you it could not see everything went to stderr.

So when you run du -shx /* 2>/dev/null as a normal user, treat every number as a lower bound. Run it under sudo when the answer matters, and read the warnings when you do not.

7.3 du Does Not Double-Count Overlapping Arguments

Give du two paths where one contains the other, and it is smarter than you might expect:

$ du -sh sub/a
6.1M    sub/a

$ du -sh sub/a/b
4.1M    sub/a/b

$ du -sh sub/a sub/a/b -c
6.1M    sub/a
6.1M    total                     # not 10.2M

It tracks what it has already visited within a single run and refuses to add the same blocks twice. That is what makes du -shc /home/* reliable, and it is the same mechanism that handles hard links in section 6.3.

The limitation is the same too: this only holds within one invocation. Two separate du commands know nothing about each other.

7.4 Small Files Cost More Than They Weigh

Section 1.2 showed a one-byte file using 4 KB. Multiply that by a real directory and it stops being a curiosity:

100,000 files of 100 bytes each

apparent size:  100,000 x 100    =  10 MB
device usage:   100,000 x 4096   = 410 MB

Forty times larger. A mail spool, a PHP session directory, a thumbnail cache or a Git object store can occupy an order of magnitude more space than the data in it, and du --apparent-size against plain du tells you immediately whether that is what you are looking at:

$ du -sh /var/lib/php/sessions
$ du -sh --apparent-size /var/lib/php/sessions
# a big gap between these two means "many small files", not "much data"

And remember section 5.4: that same directory is also consuming an inode per file, which is a resource you can exhaust independently.

7.5 du Is Slow Because It Has To Be

There is no shortcut. To total a directory, du must stat every file in it, because no filesystem in common use keeps a running total per directory. That is why the cost scales with the number of files rather than their size:

$ find /usr/share -xdev | wc -l
207983

$ du -sxh /usr/share       0.68 s     (cold)
$ du -sxh /usr/share       0.28 s     (warm, from the kernel's cache)
$ df -h /                  0.00 s

Two practical consequences. Running du on a large tree warms the cache, so a second run is much faster and any timing you take after the first is optimistic. And on a busy production server, a full du / is not free: it evicts useful data from the cache and generates a lot of metadata I/O. Prefer -x, prefer a narrow starting point, and consider ionice -c3 if you must do it during business hours.

7.6 Knowing Where du Stops

ncdu deserves the top row and is worth installing everywhere. It does the same walk du does, once, and then gives you an interactive tree you can move around in and delete from, which turns the four-round descent from section 5.5 into arrow keys.

9. Common Mistakes

9.1 Myth Versus Reality

9.2 Other Traps to Avoid

• Running du / without -x on a desktop. It will descend into /run/user/1000/gvfs and whatever that is mounting, and may never come back.
• Reading du output without sorting it. The order is directory-walk order, not size order. sort -h -r is not optional.
• Forgetting -h on one side of a pipe. sort -h needs the suffixes that du -h produces. du -k | sort -n also works; mixing the two does not.
• Using awk '{print $1}' to extract the size. Works until a path contains a space. The field separator is a tab, so cut -f1 is correct.
• Deleting from ncdu without looking. It deletes immediately and permanently, with no trash and no undo.
• Using -L without thinking about where the links go. A symlink can point outside the tree, onto another filesystem, or back at a parent directory. -L follows all of them, so a scan you expected to take a second can wander a long way.
• Blaming du on a Btrfs or ZFS server. If snapshots exist, the missing space is in them and no pathname-based tool will find it. btrfs filesystem usage or zfs list -o space answer the question du structurally cannot.
• Assuming du on a network share is quick. Every stat is a round trip. A tree that takes two seconds locally can take twenty minutes over NFS or SMB.
• Comparing du totals from two separate runs. Hard links are deduplicated within a run only, so two runs over trees that share files will double-count between them.
• Trusting a total taken while something is writing. du is not atomic. On an active server the number is a smear across the time the walk took.
• Freeing space by deleting inside a directory that is mounted over. You are deleting from the mounted filesystem, not the one whose files are hidden underneath.
• Filling a filesystem to exactly 100 percent. Once the root reserve is gone too, services that cannot write their logs or state files fail in confusing ways. Keep headroom.