Skip to main content

Linux command: top

05 September 2026

A server is slow, so you type top. Almost everybody does. Then almost everybody reads the load average on the first line, sees a number bigger than they expected, concludes the CPU is overloaded, and spends the next hour looking in the wrong place. The load average is not a CPU measurement. The memory line does not say what most people think it says. And a %CPU of 400 is not a bug. top has been part of Unix since 1984 and it is still the most confidently misread program on the box.

1. The Basics

top shows you a list of processes, sorted by CPU use, refreshed every few seconds, with a summary of the whole machine above it. That is the entire idea. Everything worth knowing is in what those numbers actually mean, because several of them mean something quite different from what they look like.

1.1 The Simplest Possible Use

Type the name, nothing else:

$ top

The screen clears and fills with a fixed layout. Press q to quit, or h for help. Those are the only two keys you must know before you start.

top - 17:21:38 up  6:11,  1 user,  load average: 1.63, 1.39, 1.29
Tasks: 517 total,   2 running, 515 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.7 us,  2.6 sy,  0.0 ni, 92.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64026.3 total,  34464.3 free,  14897.2 used,  16868.6 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  49129.1 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 118289 pe7er     20   0 1448.5g 557844 260284 R  91.7   0.9   1:42.29 chrome
  10681 pe7er     20   0   10.6g   3.7g   1.3g S  16.7   5.9  87:49.77 phpstorm
 106453 pe7er     20   0 1448.4g 251576 157480 S   8.3   0.4  16:10.38 chrome
 120269 pe7er     20   0   14868   5584   3536 R   8.3   0.0   0:00.02 top
      1 root      20   0   24500  15008   9248 S   0.0   0.0   0:04.40 systemd

All the output in this article comes from that machine: sixteen CPU cores, 62.5 GiB of memory, Ubuntu, top from procps-ng 4.0.4.

1.2 The Five Summary Lines

The block at the top describes the machine, not any single process. Each line has its own toggle key, and each one is misread in its own particular way.

LineShowsKeyThe usual misreading
top - 17:21:38 up 6:11 ... Clock, uptime, logged-in users, load average l That the load average is a CPU percentage. It is not (section 4.1).
Tasks: 517 total ... Process counts by state t That "running" means "busy". It means ready to run.
%Cpu(s): 4.7 us ... CPU time split by category t That this is one CPU. It is the average of all sixteen (section 5.1).
MiB Mem : ... Physical memory m That low free is a problem. Read avail Mem instead (section 4.3).
MiB Swap: ... Swap, plus avail Mem m That avail Mem belongs to swap. It sits on the swap line but describes RAM.

1.3 The Task Columns

Below the header comes one row per process. The default columns are these:

ColumnMeans
PID Process ID.
USER Effective user name of the owner.
PR Kernel scheduling priority. Normally NI plus 20 (section 6.4).
NI Nice value, -20 to 19. Lower is greedier.
VIRT Everything mapped into the address space. Mostly meaningless (section 7.3).
RES Resident memory: what is actually in RAM. The number that matters, but it cannot be added up (section 7.2).
SHR The part of RES that may be shared with other processes.
S State: R, S, D, I, T, t or Z.
%CPU Share of CPU time since the previous refresh. Not a lifetime average (section 9.1).
%MEM Simply RES divided by total physical memory.
TIME+ Total CPU time used since the process started, to hundredths of a second. Not wall clock time.
COMMAND Program name, or the full command line with c.

The state letters come up constantly in support conversations, so they are worth having straight:

R   running, or ready to run and waiting for a CPU
S   sleeping, waiting for something that can be interrupted
D   uninterruptible sleep, nearly always waiting for disk or network storage
I   idle kernel thread
T   stopped by a job control signal (Ctrl-Z)
t   stopped by a debugger
Z   zombie: finished, but its parent has not collected the exit status

D is the interesting one. A process in D cannot be killed, not even with kill -9, because it is sitting inside a kernel call that has not returned. It is also counted in the load average, which is where most load-average confusion begins.

1.4 The Mental Model

One sentence explains almost every surprise in this article.

top samples. Every field with a percent sign is a difference between two readings, taken over the refresh interval. Every field without one is a total, accumulated since the process started or since boot.

That is why %CPU and TIME+ can disagree so violently. TIME+ says a process has used ninety minutes of CPU since Tuesday. %CPU says it used none in the last three seconds. Both are true, and they answer different questions.

It also explains why the very first screen is not like the ones that follow, and why a script that runs top once gets an answer it should not trust. Section 6.2 measures exactly how bad that is.

Back to top

2. Where the Name Comes From

The name is a description of the output. top sorts processes by CPU use and shows you the ones at the top of that list. Not the top of the screen, not "top" as in best: the heaviest consumers, ranked.

The program was written in 1984 by William LeFebvre, then at Rice University. His reason was straightforward, and he has described it plainly: Unix had no good real-time monitoring, VMS did, and he wondered why. So he wrote one and posted it to a USENET software distribution group for anyone to download and compile. That is how software spread in 1984, and it worked well enough that his program is still on your server forty years later.

One detail from that first version survives untouched and you can see it today. LeFebvre's top refreshed every three seconds. So does the one on this machine, which will tell you so if you ask it for help:

$ top
h                                  # then press h for the help screen

Window 1:Def: Cumulative mode Off.  System: Delay 3.0 secs; Secure mode Off.

Three names are worth keeping apart, because package searches and error messages use all of them:

procps      the original Linux package of /proc tools: ps, top, free, uptime, watch
procps-ng   the "next generation" fork from 2011, which is what you now run
top(1)      the manual page; note the 1, because there is no top(8)

And top is not one program. At least three unrelated implementations share the name, which is why an option you learned on a Linux server may not exist on a BSD or inside a container. Section 3 explains how that happened.

Back to top

3. A Short History

The Linux top is not a descendant of LeFebvre's. It is a re-implementation that borrowed the name, the layout and the three-second beat, and then went its own way. You can read the second half of that story in the copyright file on any Debian or Ubuntu machine:

$ grep -B1 -A1 "James C. Warner" /usr/share/doc/procps/copyright
Files: src/top/*
Copyright: 2002-2022 James C. Warner
License: LGPL-2.0+
EraMilestone
1984 William LeFebvre writes top and distributes it over USENET. It becomes the standard Unix process monitor and is still maintained separately as "unix-top".
1992 Roger Binns writes a top for Linux, which is quickly taken over by others and folded into the procps package. A separate program, sharing only the name and the idea.
2002 James C. Warner rewrites the Linux top from scratch. Windows, field groups, the configuration file and the interactive help all date from this version, and his copyright still heads the source.
2011 The procps-ng fork takes over maintenance after the original procps stalls. Every mainstream distribution follows it.
2022 procps-ng 4.0.0 adds long options (--batch, --delay), the EXE and LOGID fields, four I/O accounting fields, and the %CUU / %CUC lifetime CPU columns.
2023 procps-ng 4.0.4, the version in this article. Bad command-line arguments now exit non-zero, and virtual-machine guest time is counted into system overhead.

Two practical consequences follow, and both catch people out.

First, top is not in POSIX and never has been. There is no standard for it, so there is no guarantee that the flags in your notes work on the next machine. On FreeBSD or macOS you get a descendant of LeFebvre's version, with its own option set and its own field names. Inside a minimal container image you often get BusyBox top, a third implementation again with a different output format entirely. Check what you are actually running before you trust a flag, and read the local manual page rather than one you found online:

$ top -V
top from procps-ng 4.0.4

Second, the options really did keep arriving. -E, -e, -O and the long forms are recent. A five-year-old server may not have them, and a tutorial from 2015 may be describing a program that no longer behaves the same way.

Back to top

4. Simple Use Cases

4.1 The Load Average, Which Nearly Everybody Reads Wrong

This is the single most misunderstood number in Linux administration, so it is worth being precise. The three figures are the load averaged over the last 1, 5 and 15 minutes:

$ uptime
 17:21:38 up  6:11,  1 user,  load average: 1.63, 1.39, 1.29

top does not calculate this. It reads it, unchanged, from a kernel file:

$ cat /proc/loadavg
1.63 1.39 1.29 2/2953 120271

And here is the definition, from the manual page for that file. Read the middle of it twice:

$ man proc_loadavg

       The first three fields in this file are load average figures giving
       the number of jobs in the run queue (state R) or waiting for disk I/O
       (state D) averaged over 1, 5, and 15 minutes.

Or waiting for disk I/O. That clause is what makes Linux different from every other Unix, and it is why the load average is not a CPU metric. A machine with a failing disk, an NFS mount that has gone away, or a database doing heavy synchronous writes can show a load average of 40 with the CPUs almost entirely idle. Every one of those blocked processes is in state D and every one of them counts.

So the load average answers "how many tasks want to make progress right now", not "how busy is the processor". Three rules make it usable:

  • Divide by the number of cores. A load of 8.0 on this sixteen-core machine is half capacity. The same 8.0 on a two-core VPS means work is queueing badly. nproc tells you the divisor.
  • Read the three numbers as a direction, not three facts. 8.0, 2.0, 1.0 is a spike that just started. 1.0, 2.0, 8.0 is a problem that is clearing up. The one-minute figure alone tells you nothing about which.
  • Check the CPU line before blaming the CPU. High load with high id (idle) means the queue is waiting for something else, and that something else is almost always storage.

The fourth field of /proc/loadavg is the instantaneous version of the same thing, and it is free of the averaging: 2/2953 means two kernel scheduling entities are runnable out of 2953 that exist.

4.2 The CPU Line

The percentages on the %Cpu(s) line are shares of the CPU time that passed since the previous refresh, added up across every core and divided by the core count.

%Cpu(s):  4.7 us,  2.6 sy,  0.0 ni, 92.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
FieldShort forMeans
us user Running normal user code. Your application.
sy system Running kernel code on somebody's behalf: syscalls, filesystem, network stack.
ni nice Running user code that has been niced to a lower priority. Usually 0.
id idle Doing nothing at all.
wa wait Idle time during which at least one task was blocked waiting for I/O.
hi hardware interrupt Servicing hardware interrupts.
si software interrupt Servicing software interrupts. Rises with heavy network traffic.
st steal Time the hypervisor gave to somebody else's virtual machine instead of yours.

Three of these deserve more than a table row.

High sy means the kernel is doing the work, not your code. On an application server that is usually a signal rather than a cause. The kernel is busy on somebody's behalf, and the question is whose. The common sources are heavy networking, filesystem traffic, a program making an enormous number of small system calls, and process or thread churn: something forking a worker per request, or a script spawning a command inside a loop. A web server showing 40 percent sy and 10 percent us is rarely computing anything; it is usually doing an enormous amount of small I/O. strace -c -p PID counts system calls by name and will normally name the culprit in one run.

wa is a kind of idle, not a kind of busy. The CPU is doing nothing; it is merely doing nothing while something waits for a disk. That has a consequence people find counter-intuitive: wa is capped by how many cores are free, so on a sixteen-core box a single process stuck on I/O can push it no higher than about 6 percent. Here is what twelve parallel direct reads did to it on this machine:

$ top -b -n 2 -d 1 | grep '^%Cpu' | tail -1
%Cpu(s):  3.6 us,  3.5 sy,  0.0 ni, 73.1 id, 19.8 wa,  0.0 hi,  0.1 si,  0.0 st

# and one second after the readers finished:
%Cpu(s):  6.3 us,  1.3 sy,  0.0 ni, 92.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

Nineteen percent, from twelve processes hammering an NVMe drive. On a busy web server with a slow disk you can see genuine I/O starvation while wa sits at 3 percent. Treat it as a hint, and confirm with iostat -x or the kernel's pressure files:

$ cat /proc/pressure/io
some avg10=8.31 avg60=2.19 avg300=0.48 total=12829609
full avg10=8.07 avg60=2.13 avg300=0.47 total=11957041

Those files are worth a paragraph of their own, because they answer the question wa only gestures at. Pressure Stall Information measures the share of time that tasks were delayed waiting for a resource, rather than the share of time the resource was busy. There are three:

/proc/pressure/cpu       waiting for a CPU to become free
/proc/pressure/memory    stalled on reclaim, page faults, swapping
/proc/pressure/io        stalled waiting for storage

Each has two lines. some is the share of time at least one task was stalled; full is the share of time every task was stalled, which is the one that means real trouble. The three numbers are 10, 60 and 300 second averages, read the same way as the load average.

The advantage over wa is that PSI is not diluted by core count. A single process stuck on I/O can push wa no higher than 6 percent on a sixteen-core box while io pressure climbs honestly. On the readers above, wa reached 19.8 while some io pressure went from 0.50 to 8.31. When a server feels slow and every utilisation figure looks reasonable, these three files are the quickest place to find out what is actually being waited on.

st is your hosting provider. On a virtual machine, steal time is CPU that the hypervisor scheduled away from you and gave to another tenant. A persistent st above a few percent means the physical host is oversubscribed, and no amount of tuning inside your VM will fix it. On bare metal it stays at 0.0.

4.3 The Memory Lines, and "Linux Is Eating My RAM"

These two lines start more unnecessary support tickets than anything else in the output:

MiB Mem :  64026.3 total,  34464.3 free,  14897.2 used,  16868.6 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  49129.1 avail Mem

Every one of those numbers comes straight out of /proc/meminfo, and the arithmetic is worth doing once so you never doubt it again:

used       = MemTotal - MemAvailable
buff/cache = Buffers + Cached + SReclaimable
avail Mem  = MemAvailable

Checked against the file on this machine, to a rounding error:

$ awk '/^MemTotal|^MemAvailable|^Buffers|^Cached:|^SReclaimable/ {printf "%-14s %10.1f MiB\n", $1, $2/1024}' /proc/meminfo
MemTotal:         64026.3 MiB
MemAvailable:     49164.1 MiB
Buffers:           1445.1 MiB
Cached:           14704.2 MiB
SReclaimable:      1185.2 MiB

MemTotal - MemAvailable      = 14852.2 MiB    # matches "used"
Buffers + Cached + SRecl     = 17334.1 MiB    # matches "buff/cache"

Now the important part. buff/cache is not lost memory. It is the page cache: copies of files the kernel is keeping in RAM because RAM would otherwise be sitting empty, and it is handed back the instant a program asks for it. A healthy long-running server with plenty of memory should show a small free and a large buff/cache. Unused memory is wasted memory.

So read avail Mem, not free. It sits at the end of the swap line, which is an unfortunate place for it, but it is a physical-memory figure and it is the honest answer to "how much can I still start?". Here free is 33.7 GiB and avail Mem is 48 GiB, and the second number is the one to plan with.

The swap line is simpler. Swap used above zero is not automatically bad; the kernel moves genuinely idle pages out on purpose. Swap that is actively churning is bad, and top will not tell you the difference. vmstat 1 and its si/so columns will.

If the units bother you, change them. -E (short for scale-summary-mem) rescales the header and -e (short for scale-task-mem) rescales the process rows:

$ top -b -n 1 -E g -e m
GiB Mem :     62.5 total,     33.4 free,     14.7 used,     16.5 buff/cache
GiB Swap:      8.0 total,      8.0 free,      0.0 used.     47.9 avail Mem

Interactively the same toggles are the E and e keys, cycling KiB through EiB.

4.4 Sorting

top starts sorted by %CPU, high to low. Four single keys switch that. They are kept for compatibility with the older Linux top, and because of that the manual notes they deliberately do not appear on any help screen, so you will not find them by pressing h:

P    sort by %CPU     (the default)
M    sort by %MEM     the one you want when memory is the question
T    sort by TIME+    total CPU burned since the process started
N    sort by PID

Press x first. It highlights the column currently being sorted, which removes any doubt about what you are looking at. R reverses the direction, and < and > move the sort one column left or right.

T is underrated. %CPU shows you who is busy in this instant; TIME+ shows you who has been busy all week. A process that has quietly accumulated forty hours of CPU on a server that should be idle is a much better lead than whatever happens to be at the top right now.

4.5 The Keys Worth Learning First

The full list is long. These eight cover almost everything a first session needs:

KeyDoes
h Help. Press it again for the second help screen.
q Quit.
M / P / T Sort by memory / CPU / total CPU time.
c Toggle between program name and full command line.
1 Split the CPU line into one line per core (section 5.1).
u Show one user only. Enter with an empty value to clear.
k Kill a process by PID.
i Hide tasks that used no CPU since the last refresh. Turns 500 rows into a handful.
d Change the refresh interval.

i is the one people are most pleased to discover. Most of a server's process list is asleep and always will be, and hiding it leaves only the tasks that have actually done something since the last refresh. It is also available as -i on the command line, and = puts everything back.

One key deserves a warning: W writes your current settings to a configuration file and they become permanent, including for scripts. Section 6.3 explains why that matters more than it sounds.

Back to top

5. Moderate Use Cases

5.1 One Core Pegged, Fifteen Idle

The %Cpu(s) line is an average, and averages hide exactly the problem you are usually hunting. A single-threaded process saturating one core on a sixteen-core machine can only ever raise the average by 6 percent, so the header looks calm while one core is on fire.

Press 1. It splits that one line into one line per core. Here is a machine with a single busy loop running, seen both ways:

# the default, combined view
%Cpu(s): 16.2 us,  0.7 sy,  0.0 ni, 83.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st

# after pressing 1
%Cpu0  :  3.0 us,  1.0 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  5.1 us,  1.0 sy,  0.0 ni, 93.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  3.0 us,  1.0 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 98.0 us,  0.0 sy,  0.0 ni,  2.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  3.0 us,  1.0 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
...
%Cpu15 :  5.0 us,  1.0 sy,  0.0 ni, 94.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

Sixteen percent in the summary. Ninety-eight percent on core 3. This is the signature of a single-threaded bottleneck, and it is one of the most useful things top can show you: a PHP script, a cron job or a database query that cannot be spread across cores no matter how many you buy.

On a machine with many cores the per-core lines will not all fit. 4 puts several CPUs on each line, and ! combines them in pairs, then fours, then eights. -1 on the command line starts with the toggle already flipped.

5.2 The Cores a Process Is Not Allowed to Use

Section 5.1 showed a process that could only use one core because it had only one thread. There is a second reason a busy process ignores fifteen idle cores, and it is invisible in every column top shows by default: the process may not be allowed on them.

Linux lets you pin a task to a subset of CPUs. taskset reads and sets that mask:

$ taskset -cp 130667                 # -c for a CPU list, -p for an existing pid
pid 130667's current affinity list: 2

$ taskset -c 0,1 ./import.sh         # start something on CPUs 0 and 1 only
$ taskset -cp 0-3 130667             # move a running process onto CPUs 0 to 3

A busy loop pinned to CPU 2 looks completely normal in the task list. It is using an entire core, and top says so:

$ top -b -n 2 -d 1 -p 130667 | tail -1
 130667 pe7er     20   0   10308   3508   3252 R  99.0   0.0   0:12.48 bash

Press 1 and the constraint becomes obvious. One core is flat out and its neighbours are asleep, and no amount of spare capacity elsewhere will help:

%Cpu1  :  1.7 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.3 us,  0.6 sy,  0.0 ni, 97.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
...

That picture is identical to the single-threaded one in section 5.1, which is exactly why it is worth knowing about. The two causes look the same on screen and have completely different fixes: threading the application, or removing the pin. taskset -cp is the one command that tells them apart.

Affinity is rarely set by accident, but it is often set and then forgotten. The usual sources on a server:

Where the pin came fromHow it was set
A systemd service unit CPUAffinity= in the unit file, or in /etc/systemd/system.conf for everything
A container docker run --cpuset-cpus 0,1
A tuning script somebody wrote in 2019 taskset in a wrapper, still running years later on a bigger machine
The application itself A configuration option, common in databases and JVM tuning

top can show you which core a task last ran on. Turn on the P field with f, and watch it stay still for a pinned process and wander for a normal one. The manual adds an honest warning about that column: "the very act of running top may break this weak affinity and cause more processes to change CPUs more often".

One distinction to keep straight, because the two get confused constantly. Affinity says where a task may run. A cgroup CPU quota, the kind a container gets, says how much it may run, without restricting it to particular cores. A process can be throttled to half a CPU while still being scheduled across all sixteen. Section 7.1 covers what top can and cannot see of that.

5.3 A %CPU of 400 Is Not a Bug

By default top runs in what the manual calls Irix mode, where 100 percent means one fully busy core. A process with four busy threads therefore shows 400. This is a compression job with four worker threads:

$ top -b -n 2 -d 1 -p 123850 | tail -1
 123850 pe7er     20   0  372564  70144   2304 S 401.0   0.1   0:16.83 xz

Press H (short for threads-show) and the same process breaks into its threads, each with its own row and its own share:

$ top -b -n 2 -d 1 -H -p 123850 | tail -5
 123852 pe7er     20   0  372564  70144   2304 R  99.9   0.1   0:05.40 xz
 123853 pe7er     20   0  372564  70144   2304 R  99.9   0.1   0:05.39 xz
 123854 pe7er     20   0  372564  70144   2304 R  99.9   0.1   0:05.39 xz
 123855 pe7er     20   0  372564  70144   2304 R  99.9   0.1   0:05.39 xz
 123850 pe7er     20   0  372564  70144   2304 S   1.0   0.1   0:00.05 xz

Four threads at 100 percent and a parent doing the coordination. The 401 was never mysterious; it was four cores plus rounding.

That is the answer you want when a process sits at 100 percent and refuses to go higher: is it one saturated thread, or forty threads doing a little each? Two commands outside top answer it more comfortably, and both show the CPU each thread last ran on:

$ ps -L -p 130929 -o pid,tid,psr,pcpu,comm
 130929  130929   4  0.6 xz            # the parent, idle
 130929  130931   6  102 xz            # tid, psr = which core, pcpu
 130929  130932   3  102 xz
 130929  130933   2  102 xz

$ pidstat -t -p 130929 1 1
Average:  1000  130929       -  300.00  1.00 ...   xz
Average:  1000       -  130931  100.00  0.00 ... |__xz
Average:  1000       -  130932  100.00  0.00 ... |__xz
Average:  1000       -  130933  100.00  0.00 ... |__xz

pidstat -t is the clearer of the two: it prints the process total on one line and indents each thread underneath it. Three threads at 100 and a total of 301 is a healthy parallel job. One thread at 100 and a total of 101, on a process with forty threads, is a bottleneck with a name.

If you prefer the other convention, I switches to Solaris mode, which divides by the number of cores so that 100 percent means the whole machine. The same process, one keystroke apart:

Irix mode Off
 123978 pe7er     20   0  372564  70272   2304 S  25.1   0.1   0:25.04 xz

401 divided by sixteen cores is 25.1. Neither number is wrong; they answer different questions. Irix mode tells you how many cores a process is using, Solaris mode tells you what fraction of the machine it is using. Know which one you are looking at before you quote a figure to somebody, because I is a persistent toggle and a colleague may have flipped it.

5.4 Showing Less

Five hundred rows scrolling past is not diagnosis. top has three separate ways to narrow the list, and they behave differently.

-u (short for filter-only-euser) matches the effective user; -U (short for filter-any-user) matches real, effective, saved or filesystem user. Prefix with ! to invert:

$ top -u www-data           # only the web server's processes
$ top -u mysql              # only the database
$ top -U '!root'            # everything that is not root

-p (short for pid) watches specific processes, up to twenty of them:

$ top -p 1234,5678
$ top -p $(pgrep php-fpm | head -20 | paste -sd,)     # the 20-pid limit, safely

And interactively, o and O build a filter on any field at all. The syntax is a field name, an operator, and a value:

o     then type    COMMAND=chrome       only chrome processes (case ignored)
O     then type    RES>500000           only processes above ~500 MiB resident
o     then type    !USER=root           everything except root
^O                                      show which filters are active
=                                       clear all filters in this window

Now the trap, and it catches everybody once. Filtering the task list does not filter the summary. With a filter showing only chrome, the header still counts the whole machine:

$ top -b -n 1 -u root | sed -n '2p'
Tasks: 515 total,   1 running, 514 sleeping,   0 stopped,   0 zombie

$ top -b -n 1 -u root | tail -n +8 | wc -l
305                              # only 305 root processes are listed

515 in the header, 305 in the list. The one exception is -p, which does restrict the count, so a filtered Tasks: line means you used -p and nothing else. Never quote a summary figure from a filtered screen.

5.5 Seeing What a Process Actually Is

COMMAND shows the program name by default, which is useless when twenty rows all say php-fpm. c (also available as -c, short for cmdline-toggle) swaps it for the full command line:

$ top -b -n 1 -c -w 200 -o +%MEM | sed -n '8p'
  10681 pe7er     20   0   10.6g   3.8g   1.3g S  45.5   6.0  89:33.95 /home/pe7er/.local/share/JetBrains/Toolbox/apps/phpstorm/bin/phpstorm

Kernel threads have no command line, so they appear in square brackets: [kthreadd], [kworker/0:0H]. That is how you tell a kernel thread from a process at a glance, and kernel threads always show 0 for VIRT, RES and SHR.

V turns on forest view, which reorders the list by parentage and draws the tree:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    871 root      19  -1   84512  22256  21092 S   0.0   0.0   0:02.22  `- systemd-journal
    987 root      20   0   31520   9372   5148 S   0.0   0.0   0:00.51  `- systemd-udevd
   1743 systemd+  20   0   17564   7528   6760 S   0.0   0.0   0:09.54  `- systemd-oomd
   1749 systemd+  20   0   21872  13264  11088 S   0.0   0.0   0:03.22  `- systemd-resolve
   1977 avahi     20   0   11544   6396   3836 S   0.0   0.0   1:34.01  `- avahi-daemon
   2108 avahi     20   0    8480   1440   1164 S   0.0   0.0   0:00.00      `- avahi-daemon
   1985 root      20   0   13692   6328   6072 S   0.0   0.0   0:00.19  `- bluetoothd

This is how you find out that the mystery process eating CPU was forked by a cron job rather than by the web server. v collapses or expands the children of the top process, and any sort key exits forest view, which is worth remembering before you wonder where your tree went.

5.6 Changing the Columns, and Keeping the Change

f opens the field manager. Arrow keys move, d or space toggles a field on or off, s makes it the sort field, q returns. There are seventy-odd fields available, far more than the dozen shown by default:

$ top -O | tr '\n' ' '
PID PPID UID USER RUID RUSER SUID SUSER GID GROUP PGRP TTY TPGID SID PR NI nTH P
%CPU TIME TIME+ %MEM VIRT SWAP RES CODE DATA SHR nMaj nMin nDRT S COMMAND WCHAN
Flags CGROUPS SUPGIDS SUPGRPS TGID OOMa OOMs ENVIRON vMj vMn USED nsIPC nsMNT
nsNET nsPID nsUSER nsUTS LXC RSan RSfd RSlk RSsh CGNAME NU LOGID EXE RSS PSS
PSan PSfd PSsh USS ioR ioRop ioW ioWop AGID AGNI STARTED ELAPSED %CUU %CUC ...

A handful of these are worth turning on for real work:

FieldWhy you would add it
nTH Thread count. Explains a %CPU above 100 without switching to H.
PPID Parent PID. Who started this?
OOMs Out-of-memory score, 0 to 1000. Who the kernel will kill first.
SWAP How much of this process has been pushed out to swap.
ioR / ioW Bytes read and written. The closest top gets to per-process disk activity.
%CUU Lifetime CPU average, the way ps computes it. Useful next to %CPU (section 9.1).
ELAPSED How long the process has been alive.
USED RES plus SWAP, which is the honest memory footprint of a process that has been swapped.

Everything you change lasts until you quit, unless you press W. That writes a configuration file and reports where it went:

W

Wrote configuration to '/home/pe7er/.config/procps/toprc'

The legacy location was ~/.toprc, with a leading dot, and it is still honoured. The name actually follows the name of the program, so an alias called t reads and writes procps/trc. The file records the sort field, the visible columns, the colours, the delay and the state of every toggle:

$ head -2 ~/.config/procps/toprc
top's Config File (Linux processes with windows)
Id:k, Mode_altscr=0, Mode_irixps=1, Delay_time=1.0, Curwin=0

Note Delay_time=1.0 there, saved from a session started with -d 1. That single line has consequences well beyond the interactive screen, which is section 6.3.

Back to top

6. Advanced Use Cases

6.1 Batch Mode

top is a full-screen program that takes over the terminal, so it cannot simply be piped. Try and it refuses, with an error and a non-zero exit:

$ top -n 1 | head -3
top: failed tty get

$ top -n 1 > out.txt; echo $?
top: failed tty get
1

-b (short for batch) is the fix. It prints plain lines, accepts no input, and stops after the number of frames you ask for with -n (short for iterations):

$ top -b -n 1                       # one frame, then exit
$ top -b -n 5 -d 2                  # five frames, two seconds apart
$ top -b -n 1 -o +%MEM              # sorted by memory, no config needed
$ top -b -n 1 -u www-data           # one user
$ top -b -n 1 >> /var/log/top.log   # a crude sampler for cron

-o (short for sort-override) is the one to remember, because it makes a batch run independent of whatever is in the configuration file. A leading + sorts high to low, - sorts low to high, and -O lists the legal field names.

There is one more trap in batch mode, and it is a quiet one. Without a terminal to measure, top falls back to 80 columns and truncates COMMAND to something like kworker+. -w (short for width) fixes it, up to 512:

$ top -b -n 1 -c | awk '{print length}' | sort -n | tail -1
80                                 # truncated

$ top -b -n 1 -c -w 512 | awk '{print length}' | sort -n | tail -1
511                                # the whole command line

6.2 The Two-Frame Rule

This is the most useful thing in the article for anybody who scripts top, and it is not in the manual.

Every percentage in top is a difference between two readings. On the first frame there is no previous reading, so top takes one, waits, and takes another. You can measure how long it waits:

$ time top -b -n 1 -p 1 >/dev/null
real    0m0.20s

$ time top -b -n 2 -p 1 >/dev/null
real    0m3.20s                    # 0.20 + one 3.0s delay

$ time top -b -n 3 -p 1 >/dev/null
real    0m6.20s                    # 0.20 + two 3.0s delays

A fifth of a second, every time. So top -b -n 1 does not give you a three-second average, and it does not give you a lifetime average either. It gives you a 0.2 second sample, and the kernel only accounts CPU time in ticks of 10 milliseconds (getconf CLK_TCK returns 100), so a fifth of a second contains about twenty countable units.

That is as unreliable as it sounds. Here is a process that burns CPU for a tenth of a second and then sleeps for nine tenths, so its true usage is roughly 14 percent. Twelve consecutive first-frame readings:

$ for i in $(seq 12); do top -b -n 1 -p $PID | tail -1 | awk '{print $9}'; done
27.3  0.0  0.0  0.0  10.0  20.0  0.0  0.0  0.0  0.0  60.0  0.0

Zero most of the time, 60 percent once. Now the same process, read from the second frame of a two-frame run with a one-second delay:

$ for i in $(seq 6); do top -b -n 2 -d 1 -p $PID | tail -1 | awk '{print $9}'; done
16.0  15.0  15.8  18.0  10.0  15.0

Stable, and correct. The rule follows directly:

Never act on the first frame. For anything scripted, monitored or reported, run top -b -n 2 -d 1 and read the second frame. The first one is a 0.2 second guess.

# the second frame only, and from it the five heaviest processes:
$ top -b -n 2 -d 1 | awk '/^ *PID/{f++} f==2' | tail -n +2 | head -5

# anything over 50%, again from the second frame only:
$ top -b -n 2 -d 1 | awk '/^ *PID/{f++} f==2 && $9+0>50'
 127695 pe7er     20   0   10308   3452   3196 R 100.0   0.0   0:02.22 bash

Memory fields do not suffer from this. RES, VIRT and %MEM are read straight out of /proc and are correct on the first frame. It is only the deltas, %CPU and the %Cpu(s) line, that need two readings.

6.3 The Configuration File Changes Batch Output

Section 5.6 mentioned that W saves the delay. Here is why that is a genuine operational hazard. The same command, on the same machine, with and without a saved configuration file:

# a user with no configuration file of their own:
$ time top -b -n 2 -p 1 >/dev/null
real    0m3.21s                    # the built-in 3.0s delay

# the same command, after somebody once pressed W in a session started with -d 1:
$ time top -b -n 2 -p 1 >/dev/null
real    0m1.21s                    # Delay_time=1.0, read from the rcfile

Batch mode reads the configuration file. Delay, sort column, visible fields, Irix mode, filters, threads mode: all of it. So a monitoring script that worked for a year can start returning different numbers because somebody logged in, pressed I and W, and logged out again. Nothing in the script changed.

Two habits remove the risk. State everything the script depends on, explicitly, on the command line:

$ top -b -n 2 -d 1 -o +%CPU -w 512

Or take the configuration file out of the picture entirely by pointing it somewhere empty:

$ XDG_CONFIG_HOME=/nonexistent top -b -n 2 -d 1

The same reasoning applies to /etc/topdefaultrc, which supplies defaults to every user who has no file of their own.

6.4 Killing, Renicing, and Reading PR

k prompts for a PID and then for a signal, defaulting to SIGTERM. The PID defaults to the first process on screen, which is the top CPU consumer, so pressing k and then Enter twice is a fast way to kill the wrong thing. Type the PID.

r renices. Positive values make a process politer, negative values greedier, and only root may go negative. You do not need top for that, and outside an interactive session you should not use it:

$ nice -n 10 ./nightly-import.sh      # start something at a lower priority
$ renice 10 -p 1234                   # lower the priority of a running process
$ sudo renice -5 -p 1234              # raise it; negative needs root
$ renice 10 -u www-data               # every process owned by a user

One thing to be clear about before you reach for it: a nice value is a weight, not a cap. A process niced to 19 will still use 100 percent of a core if nothing else wants it. Nicing only decides who wins when two runnable tasks compete, so it will not stop a backup from saturating a machine that is otherwise idle. If you need a real ceiling, that is a cgroup CPU quota, which is what a container gets and what systemd exposes as CPUQuota= on a service unit.

Which brings up the two priority columns:

PR    the kernel's scheduling priority
NI    the nice value you can set, from -20 to 19

For ordinary processes the relationship is fixed at PR = NI + 20, and you can watch it hold across a whole machine:

$ top -b -n 1 -w 200 | awk 'NR>=8 {print $3, $4}' | sort -u -k2 -n
0 -20              # PR   NI
9 -11
10 -10
19 -1
20 0               # the normal case
21 1
25 5
39 19

Real-time processes break the pattern, and the two forms confuse people:

rt      real-time priority 99, the highest there is (kernel migration threads)
-51     real-time priority 50, shown as -1 minus the rt priority
-2      real-time priority 1

So a negative PR is not a very nice process, it is a real-time one, and it will pre-empt everything you own. On a normal server, seeing rt or a negative PR next to an application process is worth a second look.

6.5 Secure Mode

On a shared or public-facing machine you may not want every logged-in user killing processes from a full-screen program. Creating /etc/toprc with exactly two lines takes k, r and the delay controls away from ordinary users:

$ sudo tee /etc/toprc <<'EOF'
s
5.0
EOF

# line 1: s = secure mode on
# line 2: the forced delay interval, in seconds

Root is unaffected unless you also pass -s (short for secure-mode), which forces the restrictions even for root. To find out whether a running top is secured, press h and read the second line of the help screen: it says Secure mode Off or On.

Do not confuse this with security. It removes convenience, not capability; anyone who can run top can still run kill. Its real use is preventing accidents on a machine where several people work.

6.6 Four Windows at Once

Almost nobody knows this exists. top maintains four independent field groups, each with its own columns, sort order, filters and colours. A shows all four at once:

A       switch between one window and all four
a / w   move to the next / previous window
g       jump to window 1 to 4 by number
-       hide or show the current window's task list
=       reset the current window
G       rename the current window

The obvious arrangement is a CPU view, a memory view, a view filtered to one user, and one sorted by TIME+. Set them up once, press W, and they come back every time. It is the closest top comes to a dashboard, and it costs nothing extra to run.

Two smaller conveniences in the same family. Long output can be scrolled with the arrow keys, PgUp, PgDn, Home and End, and C turns on a coordinates message so you know where you are. And L searches for a string anywhere in a task row, with & for the next match, which beats staring at five hundred rows looking for a PID.

Back to top

7. Something Most Users Do Not Know

7.1 Inside a Container, top Reports the Host

This one costs real time and real money, because it makes people size containers by a number that was never about their container.

Here is top running inside a container that has been given a hard limit of 256 MiB of memory and one CPU:

$ docker run --rm -m 256m --cpus 1 mariadb:10.6.18 sh -c 'top -b -n 1 | head -5'
top - 15:28:56 up  6:18,  0 users,  load average: 2.52, 1.67, 1.43
Tasks:   3 total,   1 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.4 us,  3.0 sy,  0.0 ni, 71.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64026.3 total,  34062.5 free,  12709.0 used,  17254.8 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  49266.0 avail Mem

62.5 GiB total. The container may use 256 MiB. And the limit really is in force:

$ docker run --rm -m 256m --cpus 1 mariadb:10.6.18 sh -c \
      'cat /sys/fs/cgroup/memory.max; cat /sys/fs/cgroup/cpu.max; nproc'
268435456                          # 256 MiB, exactly as requested
100000 100000                      # one CPU's worth of quota
16                                 # but the machine still looks like 16 cores

Look at the Tasks: 3 total line, though, because it shows what is going on. The PID namespace is respected, so top sees only the three processes inside the container. The memory and CPU cgroups are not, because top reads /proc/meminfo and /proc/stat, which are the host's files, mounted straight through.

So inside a container, top tells the truth about processes and lies about capacity. The load average is the host's. The CPU percentages are shares of all sixteen host cores. The memory total is the host's RAM, which means %MEM is a fraction of the wrong denominator, and a process at 90 percent of its container limit shows up as 0.4 percent.

When you need the container's own numbers, read the cgroup files or ask the runtime:

$ cat /sys/fs/cgroup/memory.current   /sys/fs/cgroup/memory.max
$ cat /sys/fs/cgroup/cpu.stat
$ docker stats --no-stream          # from the host: real per-container figures

There is a second consequence of that PID namespace, and it bites the moment you try to act on what you saw. A PID only means something inside the namespace you read it in. The same process has one number in the container and a different one on the host:

$ C=$(docker run -d --rm -m 256m mariadb:10.6.18 sleep 120)

# inside the container:
$ docker exec $C ps -eo pid,comm
    PID COMMAND
      1 sleep

# on the host, the very same process:
$ docker inspect -f '{{.State.Pid}}' $C
130140

So a PID copied out of a container's top is meaningless in the host's top, and quite possibly belongs to something else entirely. Worse, a kill 1 that would stop a container's application will, run on the host, be aimed at systemd. Decide which side you are on before you act, and use docker inspect or docker top to translate between the two.

The same warning applies to free, uptime and nproc. It is not a top bug; it is what happens when a program written for whole machines runs inside a slice of one.

7.2 You Cannot Add Up RES

RES is the honest per-process memory number, and the obvious next step is to add the column up and see where the RAM went. That does not work, and the error is large.

$ ps -eo rss= | awk '{s+=$1} END {printf "sum of RES over all processes: %.1f MiB\n", s/1024}'
sum of RES over all processes: 23660.6 MiB

$ top -b -n 1 | sed -n 4p
MiB Mem :  64026.3 total,  33770.9 free,  15078.0 used,  17595.8 buff/cache

23.6 GiB of RES against 15.0 GiB actually in use. The sum is more than half again too big, and on a machine running many copies of the same program the gap is far worse.

The reason is sharing. Shared libraries, the program image itself, copy-on-write pages inherited from a fork, shared memory segments: each page is counted in full against every process that maps it. Twenty PHP-FPM workers forked from one parent each report the parent's memory as their own. The SHR column is telling you roughly how much of RES is at risk of being double counted, which is why SHR sits right next to it.

If you need a number that adds up, ask for the proportional set size, where each shared page is divided by the number of processes sharing it:

PSS    proportional set size: shared pages divided by the number of sharers
USS    unique set size: the pages this process alone holds
RSS    a more precise resident figure, from smaps_rollup

All three are available in top through f, and both the manual and reality agree on the catch: "Accessing smaps values is 10x more costly than other memory statistics", and reading them for another user's processes needs root. Turn them on when you are investigating memory, and turn them off again.

The practical rule: use RES to compare processes with each other, and the summary line to ask how much memory the machine is using. Never use one to derive the other.

7.3 VIRT Means Almost Nothing

The widest number on the screen is the one you should care about least:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 106453 pe7er     20   0 1448.4g 251576 157480 S  16.7   0.4  16:48.36 chrome

1448.4 gibibytes of virtual memory, on a machine with 62.5 GiB of RAM and 8 GiB of swap. That is roughly twenty times more memory than exists anywhere on the box, and nothing is wrong.

VIRT is address space, not memory. It counts everything the process has mapped: code, shared libraries, file mappings, memory that was requested but never touched, and large reservations that runtimes make on principle. Java, Go, Chrome, PostgreSQL and anything using a large mmap arena routinely reserve hundreds of gigabytes of addresses they will never use. Reserving an address costs nothing; only touching a page costs memory, and touched pages appear in RES.

There is one case where VIRT is worth a glance: if it climbs steadily and forever while RES stays flat, something is leaking mappings rather than leaking memory. Otherwise, ignore the column, or press f and remove it so that it stops drawing your eye.

7.4 Where the Numbers Actually Come From

top calculates almost nothing about your machine. It opens files, reads them, subtracts one reading from the previous one, and draws the result. Every number in this article is sitting in a file you can read yourself.

The five summary lines come from three files, read once per refresh:

/proc/loadavg      the load average line
/proc/meminfo      both memory lines
/proc/stat         the %Cpu(s) line and the task state counts
/proc/uptime       the "up 6:11" part of the first line

Every process row comes from two more, read once per process per refresh:

/proc/PID/stat     PID, state, PR, NI, utime, stime, last used CPU  <- most columns
/proc/PID/statm    VIRT, RES, SHR
/proc/PID/cmdline  the COMMAND column, but only when `c' is on
/etc/passwd        turning the numeric UID into the USER column

You do not have to take that on trust. strace will show you every file top opens for a single frame:

$ strace -f -e trace=openat -o tr.txt top -b -n 1 >/dev/null
$ grep -c openat tr.txt
2231

$ grep -oE '"/proc/[0-9]+/[a-z]+"' tr.txt | sed 's|/proc/[0-9]*/|/proc/PID/|' \
      | sort | uniq -c
   1042 "/proc/PID/statm"
   1042 "/proc/PID/stat"

Now do the arithmetic, because it confirms something from section 6.2 by accident. There were 521 processes on the machine at the time. 521 times two is 1042. top -b -n 1 read every process twice, which is the priming sample and the frame, and it is why a single-frame batch run still produces a %CPU column at all. The 0.2 seconds measured in section 6.2 is the gap between those two passes.

Switch on command lines and a third file per process appears, opened exactly as often as the other two. The counts are 1044 rather than 1042 only because a process started between the two runs:

$ strace -f -e trace=openat -o tr2.txt top -b -n 1 -c >/dev/null
   1044 "/proc/PID/statm"
   1044 "/proc/PID/stat"
   1044 "/proc/PID/cmdline"     <- new, and only because of -c

None of those files exists on a disk. They are zero bytes, and the kernel generates the answer at the moment you read it:

$ ls -l /proc/meminfo /proc/stat /proc/1/stat
0 /proc/1/stat
0 /proc/meminfo
0 /proc/stat

Three useful things follow from all of this.

You can skip top entirely. For a script, reading the file is better than parsing a screen: no 0.2 second sample, no configuration file, no 80-column truncation, no failed tty get, and nothing that changes when a colleague presses W.

$ cut -d' ' -f1-3 /proc/loadavg            # the load average, three fields
1.63 1.39 1.29

$ awk '/^MemAvailable/ {print $2/1024 " MiB"}' /proc/meminfo
49164.1 MiB

$ awk '{print $1, $2}' /proc/1234/statm     # VIRT and RES, in pages

Anything reading the same files reports the same numbers. htop, btop, free, uptime, ps and most monitoring agents are all readers of /proc. That is why swapping top for a prettier program changes the presentation and never the meaning, which is section 7.8.

Whose /proc you are reading decides what you are told. Inside a container the process directories belong to the container's PID namespace, while meminfo and stat come straight from the host. That single sentence is the whole of section 7.1.

7.5 top Changes What It Measures

top appears in its own output, and not at the bottom:

 120269 pe7er     20   0   14868   5584   3536 R   8.3   0.0   0:00.02 top

Section 7.4 counted the cost: 2231 file opens for a single frame, two per process plus the summary files. That is not free, and it is measurable:

$ ls /proc | grep -c '^[0-9]'
516                                # processes to walk

$ time top -b -n 1 >/dev/null
real 0m0.23s   user 0m0.00s   sys 0m0.02s

Harmless at the default three seconds. Not harmless at -d 0.1 on a busy production server with thousands of processes, where top can become one of the heavier things running. The manual is candid about the side effects too, noting under the P (last used CPU) field that "the very act of running top may break this weak affinity and cause more processes to change CPUs more often".

Two habits follow. Leave the interval alone unless you have a reason, and if you leave top running in a forgotten SSH session for a week, know that it has woken up 1,200 times an hour, every hour, to walk every process on the machine.

7.6 Hiding the Kernel Threads

On this machine top reports over five hundred tasks, and most of them are not processes in any sense that matters to you. One environment variable removes every kernel thread from both the list and the counts:

$ top -b -n 1 | sed -n 2p
Tasks: 518 total,   1 running, 516 sleeping,   0 stopped,   1 zombie

$ LIBPROC_HIDE_KERNEL=1 top -b -n 1 | sed -n 2p
Tasks: 262 total,   1 running, 260 sleeping,   0 stopped,   1 zombie

518 becomes 262. Half of everything top was showing you was kworker, ksoftirqd, rcu_gp and friends. The value you set does not matter, only that the variable exists, and it works for the whole procps family, so ps obeys it too.

Put it in your shell profile on a server you administer often. It makes top noticeably easier to read, and the kernel threads you actually need to see, a kworker pegged at 100 percent or a stuck jbd2, are exactly the ones you would go looking for deliberately.

7.7 Knowing Where top Stops

top answers one question well: what is happening on this machine right now. Almost every frustration with it comes from asking it something else.

When you needReach for
What the machine was doing an hour ago atop, which logs snapshots to disk, or sar from sysstat
A friendlier interactive display htop, or btop for graphs. Neither is installed by default (section 7.8).
Per-process disk I/O pidstat -d 1, or iotop
Whether the disk is the bottleneck iostat -x 2, and /proc/pressure/io
Rates: context switches, swap in and out, page faults vmstat 1
A stable, scriptable one-off list ps, which was designed for exactly that
The real numbers for a container docker stats, or the cgroup files (section 7.1)
Which service or container is using what systemd-cgtop: top for cgroups rather than processes
What a process is actually doing right now strace -p, perf top, or cat /proc/PID/stack
To be told at 3am A monitoring system. top has no history and no alerting.

atop deserves the top row. Installed as a service it samples on a timer and writes every sample to a log, so you can go back and look at the ten minutes before the machine fell over. That is the one thing top structurally cannot do, and it is the thing you want most when somebody reports a problem that has already passed. It has to be installed before the incident, which is the whole difficulty with it.

And before adding a tool, check whether top already has a narrower relative that answers your question in one line: uptime for the load average, free -h for the memory summary, pgrep and pidof for finding a PID, w for who is logged in. They all come from the same procps package and read the same files.

7.8 htop and btop, and Why top Is Still Worth Learning

Ask anyone about top and within a sentence you will be told to use htop instead. That advice is mostly right. It is also, on a server you have just been handed, frequently useless, and the reason is one line of package metadata:

$ dpkg-query -W -f='${Package} ${Priority}\n' procps htop btop
btop optional
htop optional
procps required

procps, which ships top, is a required package. It is in every Debian and Ubuntu install, including the minimal ones. htop and btop are optional: someone has to install them, and atop was not installed on this machine at all.

That distinction only matters in the moments when it matters most. A freshly provisioned VPS, a rescue shell, a minimal container image, a customer's server you are looking at for the first time, a box whose outbound network is exactly the thing that is broken. In all of those, top is what you have, and apt install htop is not an option. Learn top because it is always there; use htop when somebody has been kind enough to install it.

When it is there, here is what it genuinely does better:

htop doesWhere top makes you work for it
Shows a meter per CPU core by default You have to press 1, and most people never learn that (section 5.1).
Scrolls up, down, left and right Scrolling exists but is rarely discovered, and columns run off the edge.
Select a row, press F9, kill it k then type the PID, with the busiest process as a dangerous default (section 9.3).
F5 tree, F3 search, F4 filter as you type V, L and o, all of which need the manual first.
F2 opens a setup screen for meters, columns and colours f, Z and W, spread over three separate screens.
Counts processes, threads and kernel threads separately One number that quietly mixes all three (section 7.6).

That last row is a real improvement, and you can see it by running both at the same moment:

top      Tasks: 521 total,   2 running, 519 sleeping,   0 stopped,   0 zombie
htop     Tasks: 265, 2390 thr, 261 kthr; 3 running

One number against three. top says 521 tasks; htop says 265 real processes, 2390 threads between them, and 261 kernel threads counted apart from the rest. The second line is simply more informative.

btop is a different proposition again: graphs over time, mouse support, and a look that needs a modern terminal to render properly. It is pleasant, and it is a viewer rather than a tool you build anything on.

Two things about htop catch people who arrive from top, and both are worth knowing before you type them on a live machine:

$ top  -d 5        # 5 SECONDS between refreshes
$ htop -d 5        # 0.5 seconds. htop counts in TENTHS of a second.

The manual is explicit that htop clamps that value between 1 and 100 tenths, so its slowest refresh is ten seconds and its fastest is a tenth. And the second difference matters if you were hoping to script it:

$ top -n 1 | head -3
top: failed tty get                # refuses cleanly, exit code 1

$ htop -n 1 | head -3
[?1049h[22;0;0t[1;24r(B[m[4l[?7h[H[2J ...
                                   # raw escape codes into your pipe, exit code 0

htop has no batch mode. It has -n to limit the number of frames, but the frames are still full-screen drawing instructions, so a script that captures them gets a screenful of terminal control sequences and no error to tell it so. For anything unattended, that leaves top -b and ps, which is section 9.1's conclusion arriving from a different direction.

Now the part that matters more than any of the above. htop and btop read the same /proc files that top reads. They are nicer windows onto identical numbers, so every misreading in this article survives the switch intact. The container trap from section 7.1 is the clearest proof, with htop installed inside the same 256 MiB container:

$ docker run --rm -t -m 256m --cpus 1 ubuntu:24.04 \
      sh -c 'apt-get install -y htop >/dev/null; cat /sys/fs/cgroup/memory.max; htop'
268435456                          # the container's real limit: 256 MiB

15.1G/62.5G                        # what htop's memory meter shows
Load average: 1.40 1.27 1.29       # and the host's load average

62.5 GiB, the host's RAM, in a container allowed 256 MiB. Exactly what top did, because it is exactly the same file being read. A prettier meter does not make the number mean something different.

So the honest summary is this. htop is the better program to sit in front of, and it is worth installing on every machine you own. It is not a substitute for knowing what the load average counts, what RES double counts, or whose memory you are looking at, and those are the things that actually cost people an afternoon.

Back to top

8. Best Practices

  • Divide the load average by the core count. Then check whether the CPU line is actually busy. High load with high id means storage, not processors.
  • Remember that the load average counts state D. Processes blocked on disk or a dead NFS mount raise it just as much as processes computing.
  • Read all three load figures as a trend. The one-minute number alone cannot tell a spike from a recovery.
  • Read avail Mem, never free. A low free with a large buff/cache is a healthy server, not a sick one.
  • Press 1 early. One saturated core out of sixteen is invisible in the averaged line and obvious in the per-core lines.
  • Press i to hide the idle tasks. Most of a server's process list is asleep and always will be. Hiding it leaves only what has moved since the last refresh.
  • Check taskset -cp PID before blaming threading. A pinned process and a single-threaded one look identical on screen, and only one of them is fixed by changing the code.
  • Remember a nice value is a weight, not a cap. A process niced to 19 will still take a whole core if nothing else wants it. For a real ceiling you need a cgroup quota.
  • Press M when the question is memory. The default CPU sort answers a different question, and half of memory investigations start by looking at the wrong list.
  • Use c to see the full command line. Twenty rows saying php-fpm tell you nothing; twenty command lines tell you which site.
  • Never act on the first frame. top -b -n 1 is a 0.2 second sample. Use top -b -n 2 -d 1 and read the second frame.
  • In a script, read /proc instead of parsing top. /proc/loadavg, /proc/meminfo and /proc/PID/stat are where the numbers come from anyway, and none of the traps in section 6 apply to them.
  • Reach for /proc/pressure/ when utilisation looks fine but the server does not. PSI measures how long tasks were delayed, and unlike wa it is not diluted by core count.
  • Never carry a PID across a container boundary. PID 1 inside is some five-digit number outside, and kill 1 on the host is aimed at systemd.
  • Spell out every option in a script. Batch mode reads the configuration file, so -d, -o and -w belong on the command line where nobody's saved toggles can change them.
  • Add -w 512 to any batch run you will parse. Without a terminal, top assumes 80 columns and truncates COMMAND.
  • Do not add up RES. Shared pages are counted against every process that maps them. Use it to compare processes, and the summary line for machine totals.
  • Ignore VIRT. It is address space. A number twenty times larger than your RAM is normal for Java, Go and Chrome.
  • Know which mode you are in before quoting %CPU. 400 percent in Irix mode and 25 percent in Solaris mode are the same measurement.
  • Reach for H when a process is over 100 percent. It splits the process into threads and shows you whether one thread or all of them are busy.
  • Remember the summary is never filtered. Only -p changes the Tasks: count. -u, -U and o do not.
  • Set LIBPROC_HIDE_KERNEL=1 on servers you use often. It halved a 518-task list to 262 here, and the kernel threads worth chasing are ones you would go looking for on purpose.
  • Do not size a container from top inside it. It reports the host's memory and CPUs. Use docker stats or the cgroup files.
  • Install htop, and learn top anyway. htop is the nicer program, but procps is a required package and htop is an optional one. On a rescue shell or a fresh VPS, top is what you get.
  • Do not expect htop to fix a misreading. It reads the same /proc files, so it reports the host's RAM inside a container exactly as top does.
  • Install something that keeps history. top only ever shows the present, and the incident you are asked about is always in the past.
  • Press W once you have a layout you like, and remember you did, because it will change what your scripts see.
  • Read the manual properly at least once. It is one of the most detailed manual pages on the system, and most of what people call a top quirk is documented in it.
$ man 1 top                # long, thorough, and worth an evening
$ top -h                   # the full option list, one screen
$ top -V                   # check the version before trusting a flag
$ top -O                   # every available column name
h                          # inside top: help, then h again for page two
$ man 1 ps                 # the scriptable counterpart
$ man proc_loadavg         # the definition that settles the load argument
Back to top

9. Common Mistakes

9.1 top Versus ps, and the %CPU That Disagrees

Both programs print a column called %CPU. They mean completely different things, and the difference is not small. This process burned seven seconds of CPU and then went to sleep:

$ top -b -n 1 -p 120358 | tail -1
 120358 pe7er     20   0    8656   2068   2068 S   0.0   0.0   0:07.04 sleep

$ ps -o pid,%cpu,time,etime,comm -p 120358
    PID %CPU     TIME     ELAPSED COMMAND
 120358 67.4 00:00:07       00:10 sleep

# and it is not the first-frame problem from section 6.2 either,
# the second frame agrees:
$ top -b -n 2 -d 1 -p 120358 | tail -1
 120358 pe7er     20   0    8656   2068   2068 S   0.0   0.0   0:07.04 sleep

Zero, and 67.4, for the same process at the same moment. Neither is wrong:

  • top reports CPU used since the last refresh. The process is asleep now, so 0.0 is correct.
  • ps reports total CPU divided by total lifetime. Seven seconds of CPU in ten seconds of life is 67.4 percent, averaged over its whole existence.

That is why ps aux can show a web server process at 40 percent that is doing nothing at all: it was busy at start-up an hour ago and the average never recovers. If you want the ps figure inside top, the %CUU field is exactly that, and having both columns side by side answers "is this process busy now, or has it always been busy?" in one glance.

Questiontopps
What does %CPU mean Since the last refresh Average over the process lifetime
Correct on a single reading No, the first frame is a 0.2s sample Yes, it is a single-shot tool
Safe to parse in a script Only with -b -n 2 -d 1 -w 512 Yes, that is what it is for
Affected by a config file Yes, even in batch mode No
Interactive Yes, that is the point No
Machine summary Yes, five lines of it No

The clean split: top is for looking, ps is for scripting. When a script needs a process list, ps -eo pid,pcpu,rss,comm --sort=-pcpu is shorter, faster, stable and free of every trap in section 6.

9.2 Myth Versus Reality

MythReality
"Load average 8 means the CPU is 800 percent busy." It is a count of tasks wanting to run, not a percentage. Divide it by the core count, and remember it includes tasks blocked on disk.
"High load always means a CPU problem." Linux counts state D as well as state R. A failing disk or a dead NFS mount produces a high load with idle CPUs.
"free is nearly zero, so the server is out of memory." buff/cache is returned on demand. Read avail Mem, which is MemAvailable from /proc/meminfo.
"%CPU above 100 is a bug." Irix mode counts 100 percent per core. Four busy threads is 400 percent. I switches to Solaris mode and divides by the core count.
"The CPU line shows one processor." It is the average of all of them. One core at 98 percent out of sixteen shows as 6 percent. Press 1.
"wa tells me the disk is busy." wa is idle time while something waits for I/O, so it is capped by how many cores are free. Use iostat -x or /proc/pressure/io.
"Adding up RES shows where the memory went." Measured here: 23.6 GiB of RES against 15.0 GiB in use. Shared pages are counted against every process that maps them.
"VIRT is how much memory the process needs." It is address space. Chrome shows 1448.4g on a machine with 62.5 GiB of RAM, and nothing is wrong.
"top -b -n 1 gives me the CPU usage." It gives you a 0.2 second sample. Measured on a 14 percent process, twelve readings ranged from 0.0 to 60.0. Use -n 2 -d 1.
"Batch mode ignores my settings." It reads the same rcfile. A saved delay changed top -b -n 2 from 3.21s to 1.21s. Put every option on the command line.
"top and ps disagree, so one is broken." top measures the last interval, ps averages over the process lifetime. Both were right when they printed 0.0 and 67.4 for the same process (section 9.1).
"TIME+ is how long the process has been running." It is CPU time consumed, not wall clock. A process alive for a week with TIME+ of 0:02 has done almost nothing. ELAPSED is the wall clock field.
"Filtering by user narrows the whole screen." Only the task list. The Tasks: line still counted 515 while listing 305. Only -p narrows the summary too.
"top inside my container shows my container." Processes yes, capacity no. In a 256 MiB container it reported 62.5 GiB total and 16 CPUs, because /proc/meminfo is the host's.
"State R means the process is using the CPU." It means running or ready to run. On a queued machine most R processes are waiting their turn.
"A negative PR is a very nice process." It is a real-time one. PR is -1 minus the real-time priority, so -51 is rt priority 50, and rt is 99.
"kill -9 will get rid of it." Not a process in state D. It is inside a kernel call that has not returned, and no signal is delivered until it does.
"top is standard, so the flags work everywhere." top is not in POSIX. Linux, BSD and BusyBox ship three unrelated implementations with different options and different output.

9.3 Other Traps to Avoid

  • Pressing k and then Enter. The PID defaults to the first row, which is the busiest process on the machine. That is rarely the one you meant to kill.
  • Leaving -d 0.1 running on a production server. Every refresh reads several files for every process. At 0.1 seconds on a machine with thousands of processes, top becomes part of the load it is reporting.
  • Parsing top output by column number. The columns depend on the rcfile, the H and c toggles and the terminal width. Use ps -o with named fields instead.
  • Trusting %MEM inside a container. The denominator is the host's RAM, so a process at its container limit can read 0.4 percent.
  • Chasing a kworker thread. A busy kworker is the kernel doing work for something. Find the process causing it rather than trying to renice a kernel thread.
  • Reading %CPU immediately after starting top. The first screen is the unreliable one. Wait for the second refresh before believing anything.
  • Assuming Tasks: 517 total means 517 programs. Half are kernel threads (section 7.6), and with H on, the count switches to threads and jumps again.
  • Killing a zombie. A Z process is already dead and holds no memory. It is waiting for its parent to collect the exit status, so the process to look at is the parent.
  • Sorting by %CPU while searching with L. The manual warns about this directly: rows move between refreshes, so a search on an unstable sort column keeps losing its place. Sort by PID while you search.
  • Calling it from a script without -b. top takes over the terminal, so a cron job or a CI step that forgets batch mode produces failed tty get and exit code 1, and nothing else.
  • Reporting swap used as a problem. Pages moved out while idle are the kernel doing its job. Swap that is actively churning is the problem, and vmstat 1 shows it in the si and so columns.
Back to top

10. Summary

top is over forty years old, ships on every Linux server, and is read wrongly more often than any other command on the box. Almost all of that comes down to one sentence: every field with a percent sign is the difference between two readings, and every field without one is a total.

  • Five summary lines describe the machine, then one row per process. h for help, q to quit.
  • The load average is not a CPU metric. The manual for /proc/loadavg defines it as jobs in the run queue or waiting for disk I/O. Divide it by the core count, and read all three figures as a direction.
  • The %Cpu(s) line is the average of every core. One core at 98 percent out of sixteen shows as 6 percent, so press 1.
  • wa is idle time while something waits for I/O, capped by how many cores are free. It is a hint, not a disk meter. st is your hypervisor giving your CPU to somebody else.
  • Read avail Mem, not free. used is MemTotal - MemAvailable and buff/cache is Buffers + Cached + SReclaimable, both straight from /proc/meminfo. Cache is returned on demand.
  • %CPU above 100 is Irix mode counting one core as 100 percent. I switches to Solaris mode: the same job read 401.0 and then 25.1 on a sixteen-core machine. H splits a process into its threads.
  • You cannot add up RES. Shared pages count against every process that maps them: 23.6 GiB of RES against 15.0 GiB actually in use. PSS and USS divide them properly, at ten times the cost.
  • VIRT is address space, not memory. 1448.4g on a 62.5 GiB machine is normal and means nothing.
  • top -b -n 1 is a 0.2 second sample. Measured on a process using 14 percent, twelve first-frame readings ran from 0.0 to 60.0; second-frame readings sat between 10 and 18. Use top -b -n 2 -d 1 and read the second frame.
  • Batch mode reads your rcfile. A saved delay turned top -b -n 2 from 3.21 seconds into 1.21. Put -d, -o and -w 512 on the command line, or point XDG_CONFIG_HOME somewhere empty.
  • Without a terminal, top assumes 80 columns and truncates COMMAND. Without -b it refuses to run at all: failed tty get, exit 1.
  • Filtering with -u, -U or o narrows the list but never the summary. Only -p narrows both.
  • PR is NI + 20 for normal processes. A negative PR is real-time, not polite: -51 is real-time priority 50, and rt is 99.
  • top computes almost nothing: it reads files. /proc/loadavg, /proc/meminfo and /proc/stat for the summary, /proc/PID/stat and /proc/PID/statm per process. strace counted 2231 opens for one frame, and the 1042 stat reads over 521 processes prove it samples everything twice. In a script, read the file instead.
  • An idle core does not help a process that is not allowed on it. A pin from taskset, a systemd CPUAffinity= or --cpuset-cpus looks exactly like a single-threaded bottleneck. taskset -cp PID tells them apart, and a nice value is a weight rather than a cap.
  • /proc/pressure/{cpu,memory,io} measures how long tasks were delayed, not how busy a device was, and it is not diluted by core count the way wa is.
  • Inside a container, top reports the host. A 256 MiB container showed 62.5 GiB total and sixteen CPUs. The PID namespace is honoured; the memory and CPU cgroups are not, and a PID from inside means nothing outside: PID 1 there was 130140 here.
  • LIBPROC_HIDE_KERNEL=1 removed 256 kernel threads from a 518-task list.
  • top and ps printed 0.0 and 67.4 for the same process at the same moment. top measures the last interval; ps averages over a lifetime. top is for looking, ps is for scripting.
  • htop and btop are nicer to sit in front of, and neither is installed by default: procps is a required package, they are optional ones. They also read the same /proc files, so htop in a 256 MiB container showed 15.1G/62.5G too. And htop -d 5 means half a second, not five.
  • top keeps no history. For the ten minutes before the server fell over, you need atop or sar, and you need them installed beforehand.

This is the quick reference worth keeping:

THE ESSENTIALS
top                          start it; q quits, h helps
top -b -n 2 -d 1             batch, two frames  <- read the SECOND one
top -b -n 1 -o +%MEM -w 512  one frame, sorted by memory, full width
top -u www-data              one user only
top -p 1234,5678             specific PIDs (up to 20)
top -H                       threads instead of processes
top -c                       full command lines
top -V                       version, before trusting any flag

KEYS THAT EARN THEIR PLACE
1     one line per CPU core   <- the single-core bottleneck
M P T sort by MEM / CPU / TIME+          x  highlight the sort column
c     program name or full command line  V  forest view (v collapses)
i     hide tasks that used no CPU since the last refresh
H     threads mode            I  Irix (per core) or Solaris (per machine)
u     filter by user          o  filter any field: RES>500000, !USER=root
f     choose columns          e/E  memory units, KiB through EiB
L / & search / search again   =  clear all filters and limits
A     four windows at once    W  save the layout (and change your scripts)
k     kill                    r  renice          d  change the interval

WHERE THE NUMBERS LIVE (read these directly in scripts)
/proc/loadavg          the three load figures
/proc/meminfo          every value on both memory lines
/proc/stat             the %Cpu(s) line
/proc/PID/stat         PID, state, PR, NI, cpu times, last used CPU
/proc/PID/statm        VIRT, RES, SHR
/proc/pressure/{cpu,memory,io}   how long tasks were DELAYED, not how busy

CORES A PROCESS CANNOT USE
taskset -cp PID              show the affinity mask of a running process
taskset -c 0,1 CMD           start something on CPUs 0 and 1 only
CPUAffinity= / --cpuset-cpus systemd units and docker set pins too
nice -n 10 CMD / renice      a WEIGHT, not a cap; CPUQuota= is the cap

READING THE SUMMARY
load average   tasks wanting to run, R *and* D. Divide by nproc.
%Cpu(s)        the average of every core. Press 1 before concluding anything.
  us  your code       sy kernel       ni niced      id doing nothing
  wa  IDLE, waiting on I/O            hi/si interrupts
  st  the hypervisor gave your CPU away
used           = MemTotal - MemAvailable
buff/cache     = Buffers + Cached + SReclaimable, given back on demand
avail Mem      READ THIS ONE. It is MemAvailable, and it is physical RAM.

READING A PROCESS ROW
%CPU     since the LAST REFRESH, not a lifetime average (ps differs)
TIME+    CPU time consumed, not wall clock (ELAPSED is wall clock)
RES      real memory - compare processes with it, never sum it
VIRT     address space - ignore it
SHR      how much of RES may be double counted
S        R ready  S sleeping  D uninterruptible (unkillable)  Z zombie
PR       NI + 20; negative means REAL TIME, rt means priority 99

DO NOT
top -b -n 1 for a CPU number    it is a 0.2s sample
parse top by column number      the rcfile and toggles move them
add up the RES column           shared pages counted many times
size a container from inside    it reports the host's RAM and cores
press k then Enter              the default PID is the busiest process
run it without -b in a script   "failed tty get", exit 1

WHEN top RUNS OUT   (none of these ship by default; procps does)
atop / sar        history: what happened before you logged in
htop              per-core meters, tree, F9 to kill, scrolling
btop              graphs and mouse, needs a modern terminal
pidstat -d 1      per-process disk I/O
pidstat -t -p PID per thread, indented under the process total
ps -L -p PID      threads with the core each last ran on
systemd-cgtop     top for cgroups: which service or container
strace -c -p PID  which system call is behind a high %sy
iostat -x 2       is the disk the bottleneck
vmstat 1          rates: swap in/out, context switches, faults
docker stats      the real numbers for a container
ps -eo ...        anything scripted

htop -d 5   =  0.5s, it counts in TENTHS   (top -d 5 = 5 seconds)
htop piped  =  raw escape codes, exit 0    (top without -b = exit 1)
htop, btop  =  the same /proc files, so the same misreadings

every % is a DELTA between two refreshes; everything else is a TOTAL
the load average counts disk waiters, so it is not a CPU number
the CPU line is an average: press 1
avail Mem, not free

Start with the load average and the CPU line together, because the pair of them is what separates a processor problem from a storage problem in about five seconds. And when a server crawls, the load average sits at 30, and every core is idle, the CPU was never the problem: something is stuck in state D, waiting for a disk that has stopped answering.

Back to top
Linux command: top
Peter Martin
Peter Martin
Joomla Specialist

Peter is a Joomla specialist and a Linux admin for fast, secure and scalable websites.