
Linux command: sort
Almost everybody learns sort in their first week of Linux and then never learns it again. It puts lines in order, and that looks like the whole story. It is not. sort decides the order using rules that come from your environment rather than from your command line, so the same command on the same file can give two different answers on two machines. It sorts numbers wrongly unless you tell it they are numbers. Its -u can quietly throw away a line you wanted to keep, and its -k2 almost never means what people think it means. It is also one of the very few commands in daily use that can run a server out of memory. This article is about what sort really compares, and how to make it compare the thing you meant.
1. The Basics
sort reads lines, puts them in order, and writes them out. Three words in that sentence hide the whole article: lines, order and reads. Each one behaves differently from how people expect.
1.1 The Simplest Possible Use
Give it a file, and it prints the file in order. It does not change the file:
$ cat fruit.txt
banana
Apple
apple
Banana
$ sort fruit.txt
apple
Apple
banana
Banana
With no filename it reads standard input, which is how it spends most of its life:
$ cut -d: -f1 /etc/passwd | sort | head -3
_apt
avahi
backup
Two options cover most casual use, and both are worth learning on day one:
| Option | Short for | What it does |
|---|---|---|
-r |
reverse | Reverse the result of every comparison |
-n |
numeric | Compare the lines as numbers, not as text |
-n is not a nicety. Without it, sort compares digits the way it compares letters, one character at a time, and the answer is wrong in a way that looks almost right:
$ printf '10\n9\n100\n2\n' | sort
10
100
2
9
$ printf '10\n9\n100\n2\n' | sort -n
2
9
10
100
The first output is not a bug. 1 really does come before 2, and 10 before 100 before 2, if you are comparing text. sort has no idea those lines are numbers until you tell it.
1.2 sort Reads Everything Before It Prints Anything
This is the single most important structural fact about the command, and it explains almost everything else in this article. sort cannot print its first line until it has read its last one, because the last line of the input might belong at the top.
You can watch it refuse:
$ yes hello | sort | head -1
# nothing, forever
$ yes hello | grep -m1 hello
hello # instant
grep streams: it decides about each line as it arrives. sort buffers: it holds the whole input before it can decide anything. Four consequences follow directly, and each gets its own section later:
- It uses memory in proportion to the input, roughly eleven times the file size (section 7.5).
- When that memory runs out it writes temporary files to
/tmp, which can fill up (sections 6.8 and 6.9). - You cannot put it in the middle of a live pipeline.
tail -f app.log | sortwill print nothing until the log ends, which is never. - Its input must be finite. Anything you pipe into
sorthas to stop on its own.
grepis a filter.sortis a container. Everything that is inconvenient aboutsortfollows from the fact that it has to hold your data before it can hand any of it back.
1.3 The Locale Decides the Order
"Alphabetical order" is not one order. Which letters count, whether case matters, and what happens to punctuation are all decided by your locale, and specifically by the LC_COLLATE setting. The same file, the same command, two answers:
$ sort fruit.txt
apple
Apple
banana
Banana
$ LC_ALL=C sort fruit.txt
Apple
Banana
apple
banana
Neither is broken. The first is linguistic order: it groups apple and Apple together, because a reader looking up a word in a dictionary does not care about capitals. The second is byte order: A is byte 65 and a is byte 97, so every capital comes before every lower-case letter.
Punctuation is where the two really part company:
$ printf 'a-b\nab\na_b\naa\nAb\n' > punct.txt
$ sort punct.txt # en_US.UTF-8
aa
a-b
a_b
ab
Ab
$ LC_ALL=C sort punct.txt
Ab
a-b
a_b
aa
ab
In the linguistic order the hyphen and the underscore are almost ignored, so a-b lands next to ab. In byte order the hyphen is byte 45 and sorts before every letter, so a-b lands near the top. Nothing in the command line hints at any of this.
The right mental model:
sortdoes not have an order. It borrows one from the environment. If two machines disagree about the output, look atLC_COLLATEbefore you look at anything else.
The practical rule is short. When a human will read the output, let the locale do its work. When a program will read the output, or when a second command has to agree about the order, force LC_ALL=C. Section 7.1 shows the failure this prevents, and it is not a theoretical one.
2. Where the Name Comes From
There is no puzzle here, and that is itself worth noticing. Its neighbours from the same era were cut to the bone: ls, cp, mv, rm, du, df, wc. sort kept all four of its letters because there was no obvious way to shorten it and still be understood. srt would have saved a keystroke and cost a great deal of clarity.
The vocabulary is where the real confusion lives, because sort uses several ordinary words in a very specific way. Every one of these terms causes a misunderstanding somewhere in this article:
line what sort moves around; the trailing newline is not compared
key the part of a line sort actually compares (default: all of it)
field a numbered piece of a line, counting from 1
separator what divides fields; by default a run of blanks, -t changes it
collation the rules that decide which string comes first
locale where those rules come from: LC_COLLATE, or LC_ALL, or LANG
stable lines that compare equal keep their input order (-s)
last resort when every key ties, sort compares the whole line anyway
merge combining already-sorted inputs without sorting again (-m)
external sort sorting more data than fits in memory, using temporary files
Two of those deserve a warning right now. A key is not the same as a field: -k2 names a key that starts at field 2 and runs to the end of the line, which is section 5.2 and the most common mistake with this command. And last resort means sort quietly compares things you never asked it to compare, which is section 6.1 and the reason its output is deterministic when you expected a tie.
The manual page is honest about the locale trap from section 1.3, and it shouts, which the GNU manuals almost never do:
$ man sort | grep -A3 WARNING
*** WARNING *** The locale specified by the environment affects sort
order. Set LC_ALL=C to get the traditional sort order that uses native
byte values.
Back to top3. A Short History
sort is as old as Unix itself. A sorting facility existed in Multics before Unix did, and sort appeared in the very first release of Unix, written by Ken Thompson.
| Era | Milestone |
|---|---|
| Before Unix | Multics provides a sorting facility. The idea arrives at Bell Labs already formed. |
| 3 November 1971 | sort appears in Version 1 Unix, written by Ken Thompson. |
| Version 4 Unix | Thompson reworks it to fit into pipes, while keeping the output-file option that lets it overwrite its own input. That option is still -o today (section 4.6). |
| Version 5 Unix | The convention that a filename of - means standard input arrives here. |
| 1987 | Standardised in the X/Open Portability Guide Issue 2, and later in POSIX. -b -c -d -f -i -m -n -o -r -t -u and -k all date from that settlement and work everywhere. |
| GNU coreutils | Reimplemented by Mike Haertel and Paul Eggert using a merge sort, which is why it can sort files larger than memory and why it can use several CPU cores. |
| 6.8 (2007-02-24) | --compress-program for temporary files, and -C: check quietly, report only through the exit code. |
| 7.0 (2008-10-05) | -V version sort, and --files0-from for NUL-separated file lists. |
| 7.5 (2009-08-20) | -h, human-numeric sort, so du -h | sort -h finally works. |
| 8.6 (2010-10-15) | --debug, and parallel sorting using every available processor. |
| 8.8 (2010-12-22) | The default thread count is capped at 8, "due to diminishing performance gains". Section 7.4 measures exactly that. |
| glibc 2.28 (2018-08-01) | Not a coreutils change at all, but the biggest change to sort output in decades: the C library's collation data is updated to ISO 14651 and Unicode 9, and the order of strings containing punctuation and mixed case changes in most locales. |
The authors are named in the manual, and the first name on that list belongs to the same person who wrote GNU grep:
$ man sort | grep -A2 AUTHOR
Written by Mike Haertel and Paul Eggert.
3.1 The glibc 2.28 Break
That last table row is worth a paragraph of its own, because it caught a lot of people and it will catch more. In August 2018 glibc merged roughly eighteen years of accumulated locale updates in a single release. Sorting rules that had been stable since the late 1990s changed, and they changed for ordinary ASCII strings, not only for exotic scripts.
Anything that had stored a sorted order and expected it to stay valid was now wrong. The most public casualties were PostgreSQL indexes: a database moved from a CentOS 7 machine (glibc 2.17) to a CentOS 8 machine (glibc 2.28) silently had most of its text indexes in the wrong order, with queries returning incomplete results and unique constraints no longer guaranteeing anything. The distributions picked the change up over about a year, starting with Fedora 29 and Ubuntu 18.10 in October 2018.
Your own shell scripts have the same exposure in a smaller way. If a script sorts a list on one server and compares it against a list sorted on another, an operating system upgrade on either side can change the answer. LC_ALL=C is immune to this, because byte values do not get updated.
3.2 GNU Versus BusyBox
Inside a minimal container image you do not get GNU sort, you get BusyBox. Most of the everyday options are there. Tested against BusyBox 1.38:
| Option | GNU coreutils | BusyBox 1.38 |
|---|---|---|
-b -c -d -f -g -h -i -k -M -n -o -r -s -t -u -V -z |
Yes | Yes |
-C (check quietly) |
Yes | No: invalid option |
-R (random sort) |
Yes | No: invalid option |
--debug |
Yes | No: unrecognized option |
--parallel, --files0-from, --batch-size, --compress-program, -T |
Yes | No |
-m (merge), -S (buffer size) |
Yes | Accepted and ignored. This is worse than rejecting them. |
That last row is the one to remember, because it fails silently. BusyBox takes -m without complaint and then does a full sort anyway:
$ printf '1\n3\n5\n' > a ; printf '5\n1\n' > b
$ sort -m a b # GNU: a straight merge, so b's disorder shows through
1
3
5
5
1
$ docker run --rm -v "$PWD:/d" busybox sort -m /d/a /d/b
1
1
3
5
5 # BusyBox sorted everything instead
On this occasion BusyBox produced the nicer-looking output, which is exactly why the difference is dangerous. A script that uses -m to merge pre-sorted chunks cheaply is silently doing the expensive thing, and a script that relies on merge semantics gets a different answer.
BusyBox also has no locale support to speak of, so it behaves like LC_ALL=C whatever the environment says:
$ printf 'banana\nApple\napple\n' | sort # host, en_US.UTF-8
apple
Apple
banana
$ printf 'banana\nApple\napple\n' | docker run --rm -i busybox sort
Apple
apple
banana
If you have ever had a build produce a different file list inside a container than on the developer's laptop, this is a good first suspect. macOS and the BSDs are a third dialect again, with their own differences in the corners. On anything that might not be GNU, stay with the POSIX set from the 1987 row of the table above and check the local manual for the rest.
Back to top4. Simple Use Cases
4.1 Alphabetical, and Backwards
-r (short for reverse) does not sort backwards. It reverses the result of every individual comparison, which amounts to the same thing for a plain sort but matters once you have several keys (section 5.4):
$ sort -r fruit.txt
Banana
banana
Apple
apple
Multiple files are treated as one long file, concatenated and then sorted. This is different from sorting each one:
$ sort file1.txt file2.txt file3.txt # one merged, sorted stream
4.2 Numbers: -n, -g and -h
There are three numeric sorts and they are not interchangeable. Picking the wrong one gives a plausible-looking wrong answer, which is the worst kind.
| Option | Short for | Understands | Use it for |
|---|---|---|---|
-n |
numeric | Plain integers and decimals, with an optional sign, plus the thousands separator of your locale | Almost everything: byte counts, IDs, port numbers, columns from awk |
-g |
general numeric | Everything -n does, plus scientific notation, inf and nan |
Scientific output. Slower, and it does not understand thousands separators |
-h |
human readable | Numbers with a K, M, G, T suffix |
The output of du -h, df -h, ls -lh |
The difference between -n and -g is visible in one example:
$ printf '1e3\n5\n0.5\n2\n-1\ninf\nnan\n' > gen.txt
$ sort -n gen.txt
-1
inf # -n cannot read these, so both count as 0
nan
0.5
1e3 # and this counts as 1, not 1000
2
5
$ sort -g gen.txt
nan
-1
0.5
2
5
1e3 # 1000, correctly at the top
inf
The reverse trap is just as real. -n honours the thousands separator of your locale and -g does not:
$ printf '1,200\n999\n1,100\n' | sort -n # en_US.UTF-8
999
1,100
1,200
$ printf '1,200\n999\n1,100\n' | sort -g
1,100 # both read as "1", then compared as text
1,200
999
-h exists because -n makes a mess of human-readable sizes. It reads the suffix instead of stopping at it:
$ printf '1.4G\n900M\n2K\n1023\n1.1T\n' > sizes.txt
$ sort -h sizes.txt
1023
2K
900M
1.4G
1.1T
$ sort -n sizes.txt
1.1T # 1.1, 1.4, 2, 900, 1023: the suffix is ignored
1.4G
2K
900M
1023
The pairing rule is worth memorising, because mixing the two is a common pipeline bug: -h goes with output that has suffixes, -n goes with output that does not. du -h | sort -h is right and du -k | sort -n is right; du -h | sort -n is the wrong answer, dressed up correctly.
4.3 Versions: -V
-V (short for Version) sorts the way a human reads a release number, which is neither text order nor numeric order:
$ printf 'v1.10\nv1.9\nv1.2\nv1.21\n' > vers.txt
$ sort vers.txt
v1.10
v1.2
v1.21
v1.9
$ sort -V vers.txt
v1.2
v1.9
v1.10
v1.21
It is not only for version numbers. It is the right tool for anything where digits are embedded in text: log.2 before log.10, eth1 before eth10, photo-9.jpg before photo-10.jpg. It also happens to sort IPv4 addresses correctly, which surprises people:
$ printf '10.0.0.9\n10.0.0.100\n10.0.2.1\n9.1.1.1\n10.0.0.10\n' | sort -V
9.1.1.1
10.0.0.9
10.0.0.10
10.0.0.100
10.0.2.1
4.4 Removing Duplicates: -u
-u (short for unique) keeps only the first line of each run of equal lines. It replaces the older sort | uniq and saves a process:
$ printf 'a\nb\na\nb\na\n' | sort -u
a
b
Two things about it will bite you later, and both have their own sections. -u decides "equal" using whatever comparison is currently in force, not by comparing bytes, so sort -uf and sort -un throw away much more than you expect (section 7.3). And sort -u destroys the information that uniq -c would have given you, which is why the frequency counter in section 5.6 uses uniq rather than -u.
4.5 Sorting Is Not the Same as Counting
The relatives of sort all need sorted input, and that is the whole reason sort turns up at the front of so many pipelines. uniq only ever compares a line with the one directly before it:
$ printf 'a\nb\na\nb\na\n' | uniq
a
b
a
b
a # nothing removed: no two neighbours were equal
$ printf 'a\nb\na\nb\na\n' | sort | uniq
a
b
The same applies to comm and join, which is section 7.1.
4.6 Writing the Result: -o, and the Trap Next to It
The obvious way to sort a file in place is the one that destroys it:
$ cat danger.txt
c
a
b
$ sort danger.txt > danger.txt
$ wc -c < danger.txt
0 # the file is gone
This is not sort's fault. The shell creates and truncates danger.txt before sort ever starts, so sort opens an empty file, reads nothing, and writes nothing. Every command in the shell behaves this way; sort just happens to be the one people try it with.
-o (short for output) is the fix, and it has known about this case since Version 4 Unix. sort reads the input fully before it opens the output, so naming the same file is explicitly safe:
$ printf 'c\na\nb\n' > safe.txt
$ sort -o safe.txt safe.txt
$ cat safe.txt
a
b
c
It is safe even when the output file is one of several inputs:
$ printf 'c\na\n' > o1.txt ; printf 'd\nb\n' > o2.txt
$ sort -o o1.txt o1.txt o2.txt
$ cat o1.txt
a
b
c
d
Back to top5. Moderate Use Cases
Everything so far sorted whole lines. Real data has columns, and the moment you want to sort by one of them you meet -k, which is where most people's understanding of sort quietly goes wrong.
5.1 Sorting by a Column: -k
-k (short for key) says which part of the line to compare. The full syntax looks alarming and is mostly optional:
-k F[.C][OPTS][,F[.C][OPTS]]
F field number, counting from 1
.C character position inside that field, counting from 1
OPTS ordering letters for this key only: b d f g i M h n R r V
, separates the START of the key from its END
Start with a file and the two everyday forms:
$ cat staff.txt
sales kim 42
admin ali 7
sales bob 130
admin zoe 9
support eve 42
$ sort -k2,2 staff.txt # by the name column
admin ali 7
sales bob 130
support eve 42
sales kim 42
admin zoe 9
$ sort -k3,3n staff.txt # by the number column, numerically
admin ali 7
admin zoe 9
sales kim 42
support eve 42
sales bob 130
Note where the n goes in that second command: inside the key, attached to it. -k3,3n means "compare field 3, numerically". You can also write sort -n -k3,3 and get the same result here, but that global -n applies to every key, and section 5.4 shows why that matters.
5.2 The Comma Is Not Optional
This is the single most common mistake with sort, and it is so quiet that people carry the misunderstanding for years. -k2 does not mean "field 2". It means "from the start of field 2 to the end of the line".
Most of the time the difference is invisible, because the first field usually decides the comparison anyway. It shows up the moment field 2 ties:
$ cat k2.txt
xray beta 1
yankee beta 0
$ sort -k2 k2.txt # key is "beta 1" against "beta 0"
yankee beta 0
xray beta 1
$ sort -k2,2 k2.txt # key is "beta" against "beta": a tie
xray beta 1
yankee beta 0
The first command sorted by the trailing 1 and 0, which nobody asked for. The second saw a genuine tie and fell back on comparing the whole line, which put xray first. Both are correct; only one of them is what you meant.
The habit to build is simple and costs one character: always write the end field. -k1,1, -k3,3n, -k2,4. Write -k2 only when you genuinely mean "field 2 onwards", which is rare.
A key is a range, not a column.
-k2is the range "field 2 until the end of the line";-k2,2is the range "field 2 until field 2". The comma is not decoration.
5.3 Choosing a Separator: -t
By default sort does not split on a character at all. It splits at every transition from a blank to a non-blank, which means a run of ten spaces counts as one separator. That is usually what you want for command output, and it is why the aligned columns in staff.txt worked without any setup.
-t (short for field terminator) replaces that rule with a single literal character, and it is stricter than people expect. With -t, every single occurrence of the character is a separator, so consecutive separators produce empty fields:
$ cat sp.txt
a 3 x
b 1 y
c 2 z
$ sort -k2,2n sp.txt # default splitting: field 2 is "3", "1", "2"
b 1 y
c 2 z
a 3 x
$ sort -t' ' -k2,2n sp.txt # single space: field 2 is EMPTY
a 3 x
b 1 y
c 2 z # nothing sorted at all
The second command produced the input order unchanged, which looks like it worked on a file that happened to be almost sorted. It did not work: the key was empty on every line, so every line tied. Section 5.5 shows how to see this instead of guessing.
For data with real single-character separators, -t is exactly right:
$ sort -t: -k3,3n /etc/passwd | head -4 # by numeric user ID
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
$ sort -t$'\t' -k2,2n data.tsv # a tab, in bash
One more surprise: with -t, a line that starts with the separator has an empty field 1, because there is a field before that first separator. :a:10 has field 1 empty, field 2 a and field 3 10.
5.4 Several Keys at Once
sort compares keys in the order you write them and stops at the first difference. This is how you get "group by department, then biggest number first":
$ sort -k1,1 -k3,3nr staff.txt
admin zoe 9
admin ali 7
sales bob 130
sales kim 42
support eve 42
Read that command left to right: field 1 as text ascending, then field 3 as a number descending. The r is attached to the second key, so it reverses only that key. A global -r would reverse everything:
$ cat two.txt
alpha 3
beta 1
alpha 1
beta 3
$ sort -k1,1 -k2,2nr two.txt # names up, numbers down
alpha 3
alpha 1
beta 3
beta 1
$ sort -r -k1,1 -k2,2n two.txt # everything down
beta 1
beta 3
alpha 1
alpha 3
The same rule catches people with -n. A global -n applies to every key, including the ones that hold text, and a text field parsed as a number is simply zero:
$ cat g2.txt
b 10
b 9
a 100
a 20
$ sort -k1,1 -k2,2n g2.txt # correct: letter, then number
a 20
a 100
b 9
b 10
$ sort -n -k1,1 -k2,2 g2.txt # the global -n ruins key 1
b 9
b 10
a 20
a 100
In that last command both a and b parsed as 0, so key 1 tied on every line and key 2 did all the work. The output is sorted, just not by anything you asked for. Put the ordering letter on the key, not in front of it.
5.5 Stop Guessing: --debug
Since coreutils 8.6 you do not have to reason about any of this. --debug underlines the exact bytes used as the key on every line, and warns about the mistakes above:
$ sort --debug -k2 staff.txt
sort: text ordering performed using 'en_US.UTF-8' sorting rules
sort: leading blanks are significant in key 1; consider also specifying 'b'
admin ali 7
_____________
__________________
sales bob 130
_______________
____________________
...
Two underlines appear per line. The first marks the key, the second marks the last-resort comparison, which is the whole line (section 6.1). Here the key visibly runs to the end of the line, which is the -k2 problem from section 5.2 made visible.
The failure mode from section 5.3 is even clearer:
$ sort --debug -t' ' -k2,2n sp.txt
sort: text ordering performed using 'en_US.UTF-8' sorting rules
sort: note numbers use '.' as a decimal point in this locale
a 3 x
^ no match for key
_______
b 1 y
^ no match for key
_______
no match for key is sort telling you it found nothing to compare. It is the message worth learning to recognise.
--debug also names the locale in its first line every single time, which is the fastest way to answer "why does this sort differently on the server":
$ sort --debug fruit.txt 2>&1 | head -1
sort: text ordering performed using 'en_US.UTF-8' sorting rules
$ LC_ALL=C sort --debug fruit.txt 2>&1 | head -1
sort: text ordering performed using simple byte comparison
The 2>&1 is needed because the annotations go to standard error, which is deliberate: it means you can leave --debug in a pipeline and the data still flows normally down standard output.
The warnings it produces are worth knowing on sight:
| Message | What it means |
|---|---|
no match for key |
The key is empty on this line. Usually a wrong -t or a field number that does not exist. |
leading blanks are significant in key 1 |
Your key includes the whitespace in front of it. Add b to the key. |
key 1 is numeric and spans multiple fields |
You wrote -k2n where you meant -k2,2n. |
note numbers use '.' as a decimal point in this locale |
A hint that a numeric sort is in force and the locale decides how numbers are parsed. See section 9.1. |
text ordering performed using simple byte comparison |
You are in the C locale, by choice or by accident (section 7.1). |
5.6 The Frequency Counter
This is the one pipeline everybody should know by heart. It answers "what is the most common thing in this file", and it works on any column of any log:
sort | uniq -c | sort -rn | head
Read it as four steps: group equal lines together, count each group, put the biggest count first, show the top of the list. On an Apache or Nginx access log it answers most of the questions you would otherwise open a dashboard for:
$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -5
125 203.0.113.7
40 198.51.100.22
35 192.0.2.44
$ awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -5
81 /index.php
44 /index.php?option=com_content
40 /administrator/index.php
35 /images/logo.png
$ awk '{print $9}' access.log | sort | uniq -c | sort -rn
125 200
40 404
35 500
The first sort is doing something specific: it is not there to order the output, it is there so that uniq can work at all. uniq only compares neighbours, so identical lines have to be brought together first.
The second sort uses -rn, which is -r and -n written together. Order does not matter, so -nr is the same command.
Do not be tempted to replace the first sort | uniq -c with sort -u. It looks shorter and it deletes the answer:
$ printf 'a\nb\na\nb\na\n' | sort | uniq -c
3 a
2 b
$ printf 'a\nb\na\nb\na\n' | sort -u
a
b # the counts are gone
uniq has two more modes that pair with sort and save a lot of scripting:
$ printf 'a\na\nb\nc\nc\nc\n' | sort | uniq -d
a
c # only the values that appear more than once
$ printf 'a\na\nb\nc\nc\nc\n' | sort | uniq -u
b # only the values that appear exactly once
uniq -d is how you find duplicate entries in an export, a mailing list or a database dump in one line.
6. Advanced Use Cases
6.1 Ties, Stability and the Last-Resort Comparison
What does sort do when two lines compare equal on every key you gave it? Most people assume it leaves them alone. It does not. GNU sort falls back on comparing the entire line, as if you had given no options at all except -r. The manual calls this the last-resort comparison.
$ cat stab.txt
2 zebra
1 apple
2 apple
1 zebra
$ sort -k1,1n stab.txt
1 apple
1 zebra
2 apple
2 zebra # the second column got sorted too, uninvited
Field 2 was never mentioned on that command line. sort compared it anyway, because both lines tied on field 1. -s (short for stable) turns the fallback off, and equal lines then keep their original order:
$ sort -s -k1,1n stab.txt
1 apple
1 zebra
2 zebra
2 apple # input order preserved within each group
This matters more than it looks. The last-resort comparison is what makes GNU sort deterministic: run it twice on the same input and you get the same output, regardless of how the merge happened to interleave things. Turn it off with -s and the order within a group becomes the input order, which is what you want when you are sorting data that already carries a meaningful sequence, such as a log you have sorted by severity but want to keep in time order within each severity.
The awkward part is that -u also switches the fallback off, silently. That combination is section 7.3, and it can lose data.
6.2 Checking Instead of Sorting: -c and -C
-c (short for check) does not sort. It reports whether the input is already in order and tells you the first line that is not:
$ printf 'a\nc\nb\n' > unsorted.txt
$ sort -c unsorted.txt
sort: unsorted.txt:3: disorder: b
$ echo $?
1
$ printf 'a\nb\nc\n' | sort -c
$ echo $?
0
-C is the same check with no message, for scripts that only want the exit code. The three exit codes are worth knowing because they are not the usual two:
| Exit code | Meaning |
|---|---|
0 |
Success. With -c or -C: the input was in order. |
1 |
Only from -c or -C: the input was not in order. Not an error. |
2 |
A real error: file not found, cannot write the output, out of temporary space. |
$ sort /nonexistent
sort: cannot read: /nonexistent: No such file or directory
$ echo $?
2
Adding -u to a check turns it into a check for strict ordering, which means duplicates now count as disorder. That is a one-line duplicate detector:
$ printf 'a\na\nb\n' | sort -c
$ echo $?
0 # sorted: duplicates are allowed
$ printf 'a\na\nb\n' | sort -cu
sort: -:2: disorder: a
$ echo $?
1 # not strictly sorted: there is a duplicate
In a script, sort -C is much cheaper than sorting a large file you suspect is already sorted, because it never buffers anything: it streams, compares each line with the previous one, and stops at the first problem.
6.3 Merging Already-Sorted Files: -m
-m (short for merge) combines files that are already sorted, without sorting them again. It streams, so it needs almost no memory whatever the file sizes:
$ printf '1\n3\n5\n' > m1.txt ; printf '2\n4\n6\n' > m2.txt
$ sort -m m1.txt m2.txt
1
2
3
4
5
6
It trusts you completely. Give it unsorted input and it produces unsorted output with no warning at all:
$ printf '5\n1\n' > m3.txt
$ sort -m m1.txt m3.txt
1
3
5
5
1 # garbage, silently
There is a second requirement that is easier to miss, because every input can be perfectly sorted and the merge still produces nonsense. The inputs must be sorted by the same rules. Merge a file sorted in the C locale with one sorted in a UTF-8 locale and the result is not in either order:
$ printf 'a-b\nab\nAb\n' | LC_ALL=C sort > p1.txt # byte order
$ printf 'a-c\nac\nAc\n' | sort > p2.txt # locale order
$ LC_ALL=C sort -m p1.txt p2.txt
Ab
a-b
a-c
ab
ac
Ac
$ LC_ALL=C sort -m p1.txt p2.txt | LC_ALL=C sort -c
sort: -:6: disorder: Ac # the merge output is not sorted at all
The same applies to the key. sort -m -k2,2 over files that were sorted on field 1 will interleave them by field 2 and produce garbage, without a word of complaint. A merge is only as good as the promise you make about its inputs, and sort takes that promise entirely on trust.
So -m is the right tool for rotated logs, daily exports and anything else that arrives in pre-sorted chunks, as long as every chunk was produced by the same command in the same environment. It is the wrong tool for anything you have not verified, so pair it with -C and set the locale explicitly on both sides:
$ export LC_ALL=C
$ for f in chunk-*.txt; do sort -C "$f" || { echo "$f is not sorted"; exit 1; }; done
$ sort -m chunk-*.txt > merged.txt
6.4 Character Positions Inside a Field
The .C part of a key selects characters within a field, which lets you sort on a substring without cutting the line apart first. Given fixed-width timestamps:
$ cat logs.txt
2026-03-01 12:00:00 ERROR disk
2026-01-15 08:30:00 WARN net
2026-01-15 22:00:00 INFO boot
2025-12-31 23:59:59 ERROR eol
$ sort -k2.1,2.2 logs.txt # by the hour only
2026-01-15 08:30:00 WARN net
2026-03-01 12:00:00 ERROR disk
2025-12-31 23:59:59 ERROR eol
2026-01-15 22:00:00 INFO boot
$ sort -k1.6,1.7 logs.txt # by the month inside the date
2026-01-15 08:30:00 WARN net
2026-01-15 22:00:00 INFO boot
2026-03-01 12:00:00 ERROR disk
2025-12-31 23:59:59 ERROR eol
Reading -k1.6,1.7 out loud helps: "field 1, characters 6 to 7". That is the month in an ISO date.
This only works on genuinely fixed-width data. If the field can shift by a character, use awk to build a proper column first. Which brings up the general principle: sorting by an ISO 8601 date needs no options at all, because YYYY-MM-DD was designed so that text order and date order are the same thing. If you control the format, that is the format to choose.
6.5 Months, Blanks, Case and Punctuation
Four smaller ordering options exist, and three of them are more useful than they look.
-M (short for Month) understands month abbreviations, with anything unrecognised sorting first:
$ printf 'Mar\nJan\nFeb\nDec\nfoo\nJul\n' | sort -M
foo
Jan
Feb
Mar
Jul
Dec
It is case-insensitive, and it accepts full month names as well as three-letter ones. The names it accepts come from your locale, which makes it useless on a server whose locale you do not control and a trap on one whose locale is not English.
-b (short for ignore leading blanks) removes leading whitespace from the key before comparing. In byte order a space sorts before every letter, so indented lines float to the top without it:
$ printf ' zulu\nbravo\n' > blank.txt
$ LC_ALL=C sort blank.txt
zulu # the two spaces win
bravo
$ LC_ALL=C sort -b blank.txt
bravo
zulu
-f (short for case fold) makes the comparison case-insensitive. In a UTF-8 locale this is close to the default behaviour anyway; in the C locale it changes everything:
$ printf 'Zebra\napple\nBanana\n' > f.txt
$ LC_ALL=C sort f.txt
Banana
Zebra
apple # all capitals first
$ LC_ALL=C sort -f f.txt
apple
Banana
Zebra
-d (short for dictionary order) considers only letters, digits and blanks, ignoring punctuation. -i (short for ignore non-printing) drops control characters such as tabs and escape sequences from the comparison, which is occasionally useful on coloured command output that has been captured to a file.
LC_ALL=C sort -df is a reasonable approximation of "sort this the way a phone book would", and unlike a UTF-8 locale it will still behave the same way after a glibc upgrade.
6.6 Filenames: -z and --files0-from
Filenames may contain newlines. That is legal on Linux, and it means any pipeline built out of lines can be broken by a filename, deliberately or by accident. -z (short for zero-terminated) switches both input and output to NUL bytes, which cannot appear in a filename:
$ find /var/www -type f -print0 | sort -z | xargs -0 ls -l
Every tool in that chain uses the NUL convention: find -print0, sort -z, xargs -0. Mixing them with a line-based tool defeats the point. grep -z, sed -z and du --files0-from=- are the other members of the family.
sort also has --files0-from=F, which reads its list of input files from a NUL-separated list rather than from the command line. That avoids the "argument list too long" limit when you have a very large number of files:
$ find /var/log -name '*.csv' -print0 > list0
$ sort --files0-from=list0 -t, -k2,2n > combined.csv
6.7 Shuffling: -R
-R (short for Random) is described in the manual as "shuffle, but group identical keys", and that second half is the part that matters. It hashes the key and sorts on the hash, so identical lines still end up next to each other:
$ printf 'a\na\nb\nb\nc\nc\n' | sort -R --random-source=seed
b
b
c
c
a
a # groups intact, group order random
$ printf 'a\na\nb\nb\nc\nc\n' | shuf --random-source=seed
b
c
b
a
c
a # genuinely shuffled
The exact permutation depends on the seed file, so your own run will look different. What does not change is the shape: sort -R keeps the pairs together and shuf does not.
If you want a real shuffle, use shuf. If you want to randomise the order of groups while keeping each group together, sort -R is the only thing that does it. Both accept --random-source, which makes the result reproducible: the same seed file gives the same permutation every time, which is what you want in a test.
6.8 When It Does Not Fit: the External Merge Sort
Section 1.2 said sort has to hold the whole input before it can output anything. That raises an obvious question: what happens when the whole input does not fit in memory? A lot of people assume it fails, or thrashes. It does neither. It switches strategy, and the strategy is worth understanding because it explains every tuning option in the next section.
The technique is called an external merge sort, and it works in two phases:
PHASE 1 make runs PHASE 2 merge runs
read as much as the buffer holds open 16 runs at once
sort it in memory read the smallest line from each
write it out as a sorted "run" write it to the output
repeat until the input is gone repeat until every run is empty
more than 16 runs left? merge again
Each run is sorted, so merging them needs only one line from each in memory at a time, however large the runs are. That is the whole trick: sorting needs everything at once, merging does not.
You can watch it happen. Give sort a small buffer, point -T at a directory you can see, and count the files in it while the sort runs:
$ sort -S 10M -T ./tmpdir big.txt -o /dev/null &
$ while kill -0 $! 2>/dev/null; do ls tmpdir | wc -l; done | uniq
0 1 2 3 4 5 6 7 8 ... 66 67 68 69 <- phase 1: one run at a time
69 63 57 54 53 46 39 35 29 24 23 <- phase 2: runs consumed and merged
16 10 8 3 2 1 0
The climb to 69 is phase 1: the 62 MB input divided by a 10 MB buffer. The decline is phase 2. It is not a neat staircase, because each merge deletes its sixteen inputs one at a time as it drains them while its own output file is already there, so the count wobbles down rather than dropping in blocks.
Sixteen is not a guess: --batch-size defaults to 16, and the coreutils manual says so. With 69 runs that means sort cannot do it in one pass. It merges groups of 16 into new temporary files, then merges those, so the data is read and rewritten twice on its way out. The next few paragraphs put a number on what that costs.
The size of the buffer decides how many runs there are, and that is the only thing -S really controls:
| Buffer | Runs created | Peak memory | Time |
|---|---|---|---|
-S 5M |
136 | 8 MB | 1.47 s |
-S 10M |
69 | - | 1.52 s |
-S 50M |
14 | 54 MB | 0.84 s |
-S 200M |
4 | 208 MB | 0.88 s |
-S 2G |
0 | 691 MB | 0.81 s |
no -S at all |
0 | 691 MB | 0.86 s |
Read the last three rows carefully, because they contain the most useful practical fact in this section. -S 50M was as fast as the default and used one thirteenth of the memory. Once the number of runs drops below the batch size, one merge pass is enough, and buying more memory after that buys nothing. Note also that -S 2G did not use 2 GB: -S is a ceiling, not a reservation.
The reason is easiest to see if you stop counting seconds and start counting bytes moved. For an external sort that is the unit that matters, because every merge pass reads and rewrites the entire data set. Measured on the same 62 MB file, as total bytes read plus written by the process:
in memory, no runs 61 MB read + 61 MB written = 123 MB ( 2x )
-S 50M, 14 runs, 1 pass 123 MB read + 123 MB written = 247 MB ( 4x )
-S 10M, 69 runs, 2 pass 182 MB read + 182 MB written = 365 MB ( 6x )
Each merge pass costs one more full read and one more full write. That is the whole cost model, and it turns --batch-size from a mystery into arithmetic. Forcing more passes on the same 136 runs:
$ sort -S 5M --batch-size=2 ... 1091 MB moved
$ sort -S 5M --batch-size=4 ... 611 MB moved
$ sort -S 5M --batch-size=16 ... 365 MB moved (the default)
$ sort -S 5M --batch-size=200 ... 247 MB moved (all 136 runs at once)
For a sort that fits in memory, think in seconds. For a sort that does not, think in bytes moved. Every extra merge pass rewrites your entire data set, and the buffer size is only interesting because it decides how many passes there are.
This also explains the No space left on device failure that catches nightly jobs. The runs all exist at once before the merging starts, so a sort needs temporary space of roughly the size of its input, in /tmp, on top of the space its output will take somewhere else.
6.9 Tuning a Big Sort
Five options control the machinery from the previous section. Only the first two are worth touching often.
| Option | What it controls | When to touch it |
|---|---|---|
-S SIZE |
Main memory buffer. Accepts K M G or a percentage of RAM, such as -S 50% |
To stop sort eating a server, or to give it more so it avoids temporary files entirely |
-T DIR |
Where temporary files go. Defaults to $TMPDIR, then /tmp |
When /tmp is small, in RAM (tmpfs), or on the same disk you are reading from |
--parallel=N |
How many threads sort concurrently | To keep sort from taking every core on a shared machine |
--compress-program=PROG |
Compress the temporary files with PROG |
Only when temporary space is the constraint. It costs CPU |
--batch-size=N |
How many temporary files are merged at once | Almost never. Lower it only if you hit an open-file limit |
All the numbers here come from the same 5,000,000-line, 62 MB file on one machine. Pushed to an extreme, a tiny buffer produces a lot of runs and a slow sort:
$ sort -S 1M -T ./tmpdir big.txt -o /dev/null 1.66 s, 717 temporary files
$ sort big.txt -o /dev/null 0.82 s, no temporary files
Two things follow for -T. First, if /tmp is small, a big sort fails with No space left on device even though the disk holding your data has plenty of room, because the temporary files land somewhere else entirely. Peak temporary usage in the run above was about 87 MB for a 62 MB input. Second, -T pointing at a tmpfs is not always a win: tmpfs lives in RAM, so you have moved the memory problem rather than solved it.
For -S, the practical advice is the opposite of what people expect. Do not reach for a big number. Reach for a number that gets the run count under the batch size and stop there: on this file -S 50M matched the default's speed at one thirteenth of its memory. A cap also protects the rest of the machine, because the default will take whatever it needs.
Compression is the option people reach for first and need least:
$ sort -S 1M -T ./tmpdir big.txt -o /dev/null 1.61 s
$ sort -S 1M -T ./tmpdir --compress-program=gzip big.txt -o /dev/null 7.37 s
On an NVMe SSD, compressing temporary files made the sort four and a half times slower. It pays off only when the temporary filesystem is genuinely too small or genuinely slow, such as a network mount. Measure before you add it.
Back to top7. Something Most Users Do Not Know
7.1 Your Cron Job Sorts Differently From Your Shell
This is the one that costs people real time, because the symptom is "it works when I run it and fails at night".
Your interactive shell has a locale, set from your login environment. cron and systemd units start with a nearly empty environment and no LANG at all, and sort with no locale falls back to byte comparison. So the same script produces two different orders depending on who starts it:
$ sort fruit.txt | tr '\n' ' '
apple Apple banana Banana
$ env -i /usr/bin/sort fruit.txt | tr '\n' ' ' # what cron sees
Apple Banana apple banana
--debug says so out loud, and this is the fastest possible diagnosis:
$ env -i /usr/bin/sort --debug /dev/null
/usr/bin/sort: text ordering performed using simple byte comparison
On its own, a different order is harmless. It stops being harmless the moment a second command has to agree about the order. comm and join both require their inputs to be sorted the same way, and both fail in a way that looks like a data problem rather than a locale problem:
$ printf 'a-b\nab\nAb\n' | sort > c1.txt # locale order
$ printf 'a-b\nab\nAb\n' | LC_ALL=C sort > c2.txt # byte order
$ comm c1.txt c2.txt
a-b
ab
Ab
comm: file 2 is not in sorted order
a-b
ab
comm: input is not in sorted order
comm here reports that three identical lines are partly in one file and partly in the other, which is nonsense, and only mentions the real cause afterwards. This is exactly the failure that shows up as "the backup comparison says files are missing that are clearly there".
The fix is one export at the top of any script that sorts:
#!/bin/bash
export LC_ALL=C
That gives you an order that is identical on every machine, in every locale, under cron, inside a container, and after a glibc upgrade. Use the locale order for output a person reads, and LC_ALL=C for everything a program reads. The correct set-operation idiom then looks like this:
$ comm -23 <(LC_ALL=C sort live.txt) <(LC_ALL=C sort backup.txt) # only in live
$ comm -13 <(LC_ALL=C sort live.txt) <(LC_ALL=C sort backup.txt) # only in backup
$ comm -12 <(LC_ALL=C sort live.txt) <(LC_ALL=C sort backup.txt) # in both
Read the flags as "suppress column N": -23 hides columns 2 and 3, leaving only what is unique to the first file.
7.2 LC_ALL=C Is Also a Speed Switch
Locale-aware comparison is not free. Every comparison in a UTF-8 locale has to decode characters and apply multi-level collation rules; in the C locale it is a memcmp. On the same 5,000,000-line file:
$ sort big.txt -o /dev/null 1.98 s
$ LC_ALL=C sort big.txt -o /dev/null 0.85 s
$ sort --parallel=1 big.txt -o /dev/null 6.22 s
$ LC_ALL=C sort --parallel=1 big.txt -o /dev/null 1.88 s
Twice as fast with all cores, more than three times as fast on a single core. The parallelism in modern sort hides a lot of the cost, which is why this is less famous than it used to be, but on a small VPS with one or two vCPUs the difference is very visible.
So LC_ALL=C buys correctness and speed. The only thing it costs is human-friendly ordering, which most pipelines do not need.
7.3 sort -u With a Key Throws Away a Line You May Want
This one can lose data quietly, and it deserves to be better known.
-u keeps one line out of every group that compares equal. When the comparison is the whole line, the survivors are all identical and nothing is lost. When you add a key, the lines in a group are not identical, and sort keeps exactly one of them:
$ cat u.txt
alice 10
alice 99
bob 5
$ sort -u -k1,1 u.txt
alice 10
bob 5 # "alice 99" is gone
Which one survives? Not the smallest, and not the largest. -u switches off the last-resort comparison (section 6.1), so within a group the winner is whichever line the sort happened to reach first, which in practice is input order:
$ printf 'zeta 1\nalpha 1\nmid 1\n' | sort -u -k2,2
zeta 1
$ printf 'alpha 1\nzeta 1\nmid 1\n' | sort -u -k2,2
alpha 1 # same data, different input order, different answer
If you want a specific representative, say it out loud: sort by the key and then by the tiebreaker you care about, and let -u take the first of each group. Note the -u must go with a key list that ends on the deduplication key, so this idiom uses two commands:
$ sort -k1,1 -k2,2nr u.txt | sort -u -k1,1 --stable
alice 99
bob 5 # the biggest number per name
Or avoid the whole question and use awk, which is clearer about what it keeps:
$ awk '!seen[$1]++' u.txt
alice 10
bob 5 # explicitly the first one seen
The same trap appears without any key at all, whenever an ordering option makes different lines compare equal:
$ printf 'Apple\napple\nAPPLE\n' | sort -uf
Apple # -f made all three equal
$ printf '1\n1.0\n01\n1.00\n2\n' | sort -un
1
2 # -n made four lines equal
Neither of those is a bug. -u means "unique according to the comparison in force", and the comparison in force is the one you asked for. But if your intention was "remove exact duplicate lines", the safe spelling is a plain sort -u with no other ordering options, or LC_ALL=C sort -u to be certain the comparison is byte-for-byte.
7.4 sort Has Been Multi-Threaded Since 2010, and Stops Helping at 8
Since coreutils 8.6, sort uses several CPU cores without being asked. Since 8.8 it caps itself at eight threads by default, and the release note gives the reason as "diminishing performance gains". You can measure the curve on your own machine in a minute:
$ LC_ALL=C sort --parallel=1 big.txt -o /dev/null 1.87 s
$ LC_ALL=C sort --parallel=2 big.txt -o /dev/null 1.21 s
$ LC_ALL=C sort --parallel=4 big.txt -o /dev/null 0.91 s
$ LC_ALL=C sort --parallel=8 big.txt -o /dev/null 0.79 s
$ LC_ALL=C sort --parallel=16 big.txt -o /dev/null 0.79 s (16 cores available)
Doubling from 1 to 2 saved 0.66 seconds. Doubling from 8 to 16 saved nothing at all, on a machine with 16 cores to give. The default is well chosen and there is rarely a reason to raise it.
There is a reason to lower it. On a shared web server, an unrestricted sort of a large log will take eight cores for as long as it runs, and everything else on the box notices. --parallel=2, or nice and ionice, is the considerate choice during business hours.
7.5 sort Uses About Eleven Times the File Size in Memory
Because sort must hold everything before it can output anything (section 1.2), its memory use scales with the input. It is not one-to-one. Every line becomes a record with pointers and key offsets, so the overhead is large:
sort 12 MB / 1,000,000 lines → 137 MB resident
sort 61 MB / 5,000,000 lines → 674 MB resident
Roughly eleven times the file size, and it scales with the number of lines rather than the number of bytes, so many short lines cost more than a few long ones. A one-gigabyte log on a two-gigabyte VPS will not sort in memory; it will spill to /tmp, and if /tmp is also small it will fail outright with No space left on device while df shows the data disk half empty.
Two defences, and they are the first things to reach for when a nightly job starts dying:
$ sort -S 200M -T /var/tmp big.log -o sorted.log # cap RAM, spill somewhere roomy
And before that, ask whether you need to sort at all. sort -C checks without buffering. awk '!seen[$0]++' deduplicates without sorting. grep, cut and awk all stream. Filtering the file down before the sort, rather than after, is usually the whole fix:
$ grep ' 500 ' access.log | awk '{print $1}' | sort | uniq -c | sort -rn
7.6 tsort Is the Other Sort
Coreutils ships a second sorting command that almost nobody has used. tsort does a topological sort: you give it pairs meaning "this must come before that", and it produces an order that satisfies all of them.
$ printf 'b c\na b\nc d\n' | tsort
a
b
c
d
That is dependency resolution in one command: package A before package B, migration 3 before migration 7, this service before that one. It also detects circular dependencies and reports them, which is often the actual question you had.
Its origin story is a good one. Very early Unix linkers read an archive file exactly once, in order, deciding as they went whether each object was needed. If scanf.o called something in read.o but appeared after it, the link failed. So a shell script called lorder produced the dependency pairs, tsort put them in a workable order, and that order decided how objects were added to the archive. The coreutils manual notes that the whole procedure has been obsolete since about 1980, because archives have carried a symbol table ever since. The command stayed, and it is still the shortest way to answer "in what order can I do these things".
7.7 Knowing Where sort Stops
| When you need | Reach for |
|---|---|
| A genuine shuffle | shuf. sort -R groups identical lines (section 6.7) |
| Counting, not just ordering | uniq -c, and uniq -d / uniq -u |
| Set operations between two files | comm, and join for a relational join. Both need matching sort orders |
| Dependency order | tsort |
| CSV with quoted fields containing the separator | csvsort (csvkit), mlr (Miller), qsv. sort -t, cannot do it |
| JSON | jq 'sort_by(.field)', or jq -S to sort object keys |
| Dedup that keeps the first-seen order | awk '!seen[$0]++' |
| Sorting by file size or timestamp | ls -S, ls -t, or find -printf '%T@ %p\n' | sort -k1,1nr |
| Sorting processes by memory or CPU | ps -eo pid,rss,comm --sort=-rss. ps sorts itself |
| Grouped statistics: sums, averages, per-group counts | datamash, or awk |
| Anything relational and repeated | SQLite. sqlite3 :memory: '.import file.csv t' 'select ...' |
| An order that must survive an OS upgrade | LC_ALL=C, always (section 3.1) |
The CSV row is the one that catches people, because sort -t, looks like it should work and produces output rather than an error. Given a quoted field containing a comma, the field numbers all shift:
$ cat bad.csv
"Smith, John",Utrecht,300
"Doe, Jane",Amsterdam,100
$ sort -t, -k3,3n --debug bad.csv
"Doe, Jane",Amsterdam,100
^ no match for key
_________________________
"Smith, John",Utrecht,300
^ no match for key
_________________________
Field 3 is Amsterdam, not 100, because the quoted comma was counted as a separator. --debug caught it. Without --debug you would have got two lines in an order that means nothing.
8. Best Practices
- Put
export LC_ALL=Cat the top of any script that sorts. It makes the order identical on every machine, under cron, inside a container and after an OS upgrade, and it is roughly twice as fast. Leave the locale alone only for output a person will read. - Always write the end of the key.
-k2,2, not-k2. The comma is the difference between "field 2" and "field 2 to the end of the line", and the wrong one is usually right by accident until it is not. - Put the ordering letter on the key.
-k3,3n, not-n -k3,3. A global-napplies to every key, and turns your text key into zero. - Reach for
--debugthe moment something looks wrong. It underlines the exact bytes being compared and names the locale. It answers in one second what guessing answers in ten minutes. - Learn the frequency counter by heart:
sort | uniq -c | sort -rn | head. It answers most log questions. - Use
-o file, never> file, when the output goes back to the input. The shell truncates the file beforesortopens it. - Match the numeric sort to the data.
-hfor suffixed sizes,-nfor plain numbers,-Vfor version and release numbers,-gonly for scientific notation. - Use ISO dates and you will not need a date sort.
YYYY-MM-DD HH:MM:SSsorts correctly as plain text, in every locale, with no options. - Be careful with
-uwhen you have also given a key or an ordering option. It keeps one arbitrary line per group. If you mean "remove exact duplicates", use plainLC_ALL=C sort -u. - Verify before you merge.
sort -mtrusts its input completely and produces silent garbage if that input is not sorted.sort -Cis a cheap guard. - Cap the memory of big sorts, and cap it low:
-S 50M -T /var/tmp. The default takes whatever it needs (691 MB for a 62 MB file here) and a small buffer matched it exactly, because all that matters is getting the run count under the batch size. The temporary files go to/tmp, which is often the smallest filesystem you have. - For a sort that spills to disk, count bytes moved, not seconds. Every merge pass reads and rewrites the whole data set. That is why
-Shelps and why lowering--batch-sizehurts (section 6.8). - Never merge chunks that were sorted by different rules.
-massumes one order across every input. SetLC_ALL=Cand the same key on the side that writes the chunks and on the side that merges them. - Filter before you sort, not after.
grep,cutandawkstream and cost nothing;sortbuffers everything. - Use
-zwithfind -print0andxargs -0whenever filenames come from somewhere you do not control. - Be considerate on shared servers. A large sort takes eight cores by default.
--parallel=2andnicekeep the rest of the machine responsive. - Do not add
--compress-programwithout measuring. On fast local storage it makes sorting several times slower. - Read the manual once. It is short, and the shouted warning at the bottom of it is the most important sentence about this command.
$ man 1 sort
$ sort --help # the KEYDEF paragraph at the bottom is the useful part
$ info coreutils 'sort invocation'
$ man 1 uniq # sort's constant companion
$ man 1 comm ; man 1 join # both need matching sort orders
$ man 1 shuf ; man 1 tsort
Back to top9. Common Mistakes
9.1 Myth Versus Reality
| Myth | Reality |
|---|---|
"sort gives the same result everywhere." |
It gives the result your locale asks for. Your shell, your cron job and your container can each produce a different order from the same file (section 7.1). |
"-k2 sorts by the second column." |
It sorts from the start of field 2 to the end of the line. -k2,2 sorts by the second column (section 5.2). |
"sort -n -k3,3 and sort -k3,3n are the same." |
Only when there is one key. The global -n applies to every key, so a text key becomes zero and stops sorting anything (section 5.4). |
"sort -u removes duplicate lines." |
It removes lines that compare equal. With -f, -n or a key, that is a much larger set than "identical" (section 7.3). |
"sort -u -k1,1 keeps the smallest line in each group." |
It keeps whichever line came first in the input. -u switches off the last-resort comparison, so the survivor is arbitrary (section 7.3). |
| "Lines that tie keep their original order." | Not without -s. GNU sort falls back on comparing the whole line, so it sorts columns you never mentioned (section 6.1). |
"sort file > file sorts the file in place." |
It empties the file. The shell truncates the target before sort starts. Use sort -o file file (section 4.6). |
"-t' ' is the safe way to split on spaces." |
It is usually the opposite. With -t, every single space separates, so aligned columns become empty fields. The default splitting handles runs of blanks correctly (section 5.3). |
"sort -n handles any number." |
Not scientific notation: 1e3 reads as 1. Use -g for that, and accept that -g then loses thousands separators (section 4.2). |
"sort -n works the same in every locale." |
It uses the locale's decimal point and thousands separator. In en_US, 1,5 parses as fifteen. A European CSV with decimal commas sorts into nonsense. |
"du -h | sort -n shows the biggest directories." |
-n ignores the suffix, so 1.1T sorts below 1023. -h reads suffixes; use du -h | sort -h or du -k | sort -n. |
"sort -V is only for version numbers." |
It is the right tool for any digits embedded in text: log.2 before log.10, eth1 before eth10, and IPv4 addresses too (section 4.3). |
"sort -m checks that its input is sorted." |
It checks nothing. Unsorted input gives unsorted output with no warning (section 6.3). BusyBox makes it worse by accepting -m and doing a full sort instead. |
"sort -R shuffles." |
It randomises the order of groups but keeps identical lines together. shuf shuffles (section 6.7). |
"sort is a streaming filter like grep." |
It buffers the whole input first. tail -f log | sort prints nothing, ever (section 1.2). |
| "Sorting a big file only costs disk I/O." | It costs about eleven times the file size in RAM, and spills to /tmp when that runs out (section 7.5). |
"An exit code of 1 from sort means it failed." |
From -c or -C it means "not sorted", which is an answer, not an error. Real errors return 2 (section 6.2). |
| "A sorted list stays sorted." | Not across a glibc upgrade. glibc 2.28 in 2018 changed collation for ordinary ASCII strings and broke stored orders, including database indexes (section 3.1). |
"A bigger -S is always faster." |
Only until the runs fit in one merge pass. -S 50M matched the default here while using 54 MB against 691 MB (section 6.8). |
"-S 2G makes sort allocate 2 GB." |
It is a ceiling, not a reservation. On a 62 MB file -S 2G peaked at the same 691 MB as no -S at all. |
"sort -m only needs its inputs to be sorted." |
It needs them sorted by the same rules. Merging a C-sorted file with a locale-sorted one gives output that sort -c rejects (section 6.3). |
"sort -t, handles CSV." |
Only CSV with no quoted separators. One "Smith, John" shifts every field number after it (section 7.7). |
9.2 Other Traps to Avoid
- Piping into
uniqwithout sorting first.uniqonly ever compares neighbours, so it removes nothing from unsorted input and gives you a false clean bill of health. - Replacing
sort | uniq -cwithsort -u. Shorter, and it deletes the counts you were about to look at. - Sorting a file that is being written to.
sortis not atomic. On an active log you get a snapshot smeared across the time the read took. - Assuming
sort | head -10is cheap. The whole input is still read and sorted; only the writing stops early. - Forgetting that
-rreverses every key. To reverse only one, attachrto that key:-k1,1 -k2,2nr. - Sorting
ls -loutput by column number. The column count changes for symlinks and for files with unusual dates. Usels -S,ls -t, orfind -printfwith a format you chose. - Leaving
/tmpas the temporary directory for a nightly job. On many servers/tmpis small, or is atmpfsliving in RAM.-T /var/tmpis usually the safer choice. - Sorting with a locale in one place and comparing in another.
commandjoinreport a data problem when the real problem is that the two lists were sorted by different rules. - Merging chunks produced by different machines, jobs or locales.
sort -mpromises nothing about inputs that disagree about the order, and the result passes no check you have not written. - Lowering
--batch-sizeto "save memory". It costs I/O instead, and a lot of it: on the same file,--batch-size=2moved 1091 MB where the default moved 365 MB. - Trusting a sort inside a minimal container. BusyBox behaves like
LC_ALL=Cwhatever you set, silently ignores-mand-S, and does not have--debugto tell you so. - Using
-band expecting it to affect every key. It applies to keys, and--debugwill tell you when a key has picked up leading whitespace you did not want. - Sorting user-supplied filenames as lines. A newline in a filename is legal and breaks the pipeline. Use
-zwith-print0andxargs -0. - Running an unrestricted sort on a shared web server at midday. Eight cores and gigabytes of RAM, for as long as it takes.
nice --parallel=2 -S 200Mis the polite version.
10. Summary
sort looks like the simplest command in the toolbox and is one of the subtlest. Everything awkward about it comes from three facts, and once you hold those three in your head the rest stops being surprising.
It borrows its order from the environment. There is no built-in alphabetical order; there is LC_COLLATE, which differs between your shell, your cron job, your container and your colleague's laptop, and which changed for everyone in 2018.
It compares a key, not a line. The default key happens to be the whole line, which hides the distinction until you write your first -k and it goes wrong.
It has to read everything before it can print anything. That single constraint produces the memory use, the temporary files, the missing streaming behaviour, and the reason it is worth filtering before you sort.
export LC_ALL=Cin every script that sorts. Identical output everywhere, immune to OS upgrades, and about twice as fast.- Write
-k2,2, not-k2. Without the end field the key runs to the end of the line, which is almost never what you meant. - Put the ordering letter on the key, as in
-k3,3n. A global-nflattens your text keys to zero. --debugends every argument about this command. It underlines the bytes being compared, names the locale, and saysno match for keywhen your key is empty.- Pick the right numeric sort:
-hfor1.4G,-nfor plain numbers,-Vforlog.2beforelog.10,-gonly for1e3. sort | uniq -c | sort -rn | headanswers most log questions. The first sort is there souniqcan work at all.sort file > fileempties the file. Usesort -o file file, which has been safe since Version 4 Unix.-ukeeps one arbitrary line per equal group, not the smallest one, and with-f,-nor a key "equal" is much broader than "identical".- Ties are not left alone:
sortcompares the whole line as a last resort.-sturns that off and keeps input order. -mmerges without checking. Guard it with-C, which returns 1 for "not sorted" and 2 for a real error.- Big sorts cost about eleven times the file size in RAM and spill into
/tmp.-S 50M -T /var/tmpis the fix, and it is not a compromise: a small buffer matched the default's speed at one thirteenth of the memory. Filtering first is better still. - When it spills, it becomes an external merge sort: sorted runs on disk, merged 16 at a time. Count bytes moved, not seconds, because every extra merge pass rewrites the whole data set (123 MB in memory, 247 MB with one pass, 365 MB with two).
- It has been multi-threaded since 2010 and stops gaining after 8 threads: 1.87 s at
--parallel=1, 0.79 s at 8, and 0.79 s at 16. -Ris notshuf,sort -t,is not a CSV parser, andtsortis a completely different command that resolves dependencies.
This is the quick reference worth keeping:
THE ONES TO REMEMBER
sort | uniq -c | sort -rn | head what appears most often
LC_ALL=C sort same order everywhere, and faster
sort -o file file sort in place ( > file EMPTIES it )
sort --debug ... show me the key you are actually using
ORDERING
-n plain numbers -h 1023 < 2K < 900M < 1.4G
-g also 1e3, inf, nan -V log.2 before log.10, and IPv4
-r reverse EVERY key -M Jan Feb Mar (locale month names)
-f ignore case -b ignore leading blanks
-d letters/digits only -i ignore control characters
KEYS -k F[.C][OPTS][,F[.C][OPTS]]
-k2 field 2 TO END OF LINE <- almost never what you want
-k2,2 field 2 only <- write this
-k3,3n field 3, numeric <- letter ON the key, not global
-k1,1 -k3,3nr name up, number down
-k1.6,1.7 field 1, characters 6-7
-t: one literal separator; repeats make EMPTY fields
default a RUN of blanks is one separator
MODES
-u keep one line per equal group (arbitrary; disables last resort)
-s stable: equal lines keep input order
-c is it sorted? -C same, quietly. exit 1 = no, 2 = error
-cu is it sorted AND free of duplicates?
-m merge pre-sorted files; checks NOTHING, and assumes every
input used the SAME locale and the SAME key
BIG FILES ( when it does not fit: sorted runs on disk, merged 16 at a time )
-S 50M cap the buffer LOW; matched the default at 1/13th the RAM
-T /var/tmp spill here, not /tmp ( needs ~1x the input, plus the output )
--parallel=2 be polite on a shared box ( default caps at 8 )
--compress-program=gzip only if temp SPACE is the problem; costs speed
--batch-size leave it alone. lowering it multiplies the bytes moved
every merge pass re-reads AND re-writes everything: 2x, 4x, 6x the file
SAFE FILENAMES
find . -print0 | sort -z | xargs -0 ...
TEAMMATES
uniq -c/-d/-u comm -12/-23/-13 join shuf tsort
LC_ALL=C for programs. The locale for people. --debug when in doubt.
Most of the time sort is a five-second command that does exactly what you expect. The trouble starts when its output feeds something else: a comparison against yesterday's list, a comm against a backup, a deduplicated export, a nightly job that has run fine for two years and then fails after an upgrade. Those are the moments when the order matters more than the sorting, and they are usually cheaper to prevent than to debug.


Peter is a Joomla specialist and a Linux admin for fast, secure and scalable websites.












