Linux concept: DNS resolver

Two people run a lookup for the same name on the same laptop, in the same minute, and get two different answers. Neither of them is wrong and neither machine is broken. On Linux, "resolving a name" is not one thing: it is a stack of four layers, each with its own configuration file, its own cache and its own opinion, and a command-line DNS tool skips most of them. Until you know which layer answered, you cannot tell whether the problem is your domain, your network, or a single line in a file you have never opened.

This article is about the client side: the machine in front of you. What /etc/resolv.conf really is on a modern system, why /etc/nsswitch.conf decides more than DNS does, what resolvectl can tell you that dig cannot, how containers resolve names differently from their host, and how to run your own resolver without becoming somebody else's attack tool.

How a Linux machine turns a name into an address, layer by layer.

Goal: after reading this you can say exactly which layer answered a lookup, and change the setting in the place where it is actually generated.

2. Where the Names Come From

Three filenames carry most of this subject, and each of them is an abbreviation whose expansion explains what the file is for.

The word stub is worth pinning down too, because it appears in every diagnostic message on a modern system. A stub resolver is a client that knows almost nothing: it asks one server and believes the answer. It does not chase referrals from the root, it does not know what an authoritative server is, and it cannot resolve anything on its own. That work belongs to a recursive resolver, which is what your ISP or 1.1.1.1 runs.

The confusing part on Linux is that systemd-resolved calls itself a "stub listener" and runs on 127.0.0.53. It is a stub from your applications' point of view, and a client from the network's point of view, and it is not resolving anything from the root. It is a forwarder with a cache.

And resolvectl is simply resolve control, following the same pattern as systemctl, journalctl and hostnamectl: a control program for one systemd service.

4. Simple Use Cases

Five commands answer almost every everyday question. Learn these first, because most of the time the answer is in one of them and no editing is needed at all.

4.1 What Is This Machine Actually Using?

The obvious first move is to read /etc/resolv.conf. Do it, but read what it says:

$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
...
nameserver 127.0.0.53
options edns0 trust-ad
search localdomain

That 127.0.0.53 is not a DNS server on the internet. It is the local stub, and the real upstream servers sit behind it. The file is telling you, in a comment, that it cannot answer your question. The command that can is resolvectl status:

$ resolvectl status
Global
           Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 9.9.9.9

Link 3 (wlp0s20f3)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: localdomain

Link 6 (docker0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Read that output as a per-interface table, because that is what it is. The Wi-Fi interface has a DNS server; docker0 has none. The minus signs are features that are switched off: no DNS over TLS, no DNSSEC validation, no multicast DNS. resolv.conf mode: stub confirms that the file you just read is the generated stub version rather than a static one you wrote.

For just the servers, without the rest:

$ resolvectl dns
Global:
Link 3 (wlp0s20f3): 192.168.1.1
Link 6 (docker0):

4.2 What Does This Machine Resolve a Name To?

resolvectl query asks the same path an application would, and then tells you things no other tool reports:

$ resolvectl query petermartin.nl
petermartin.nl: 23.88.98.40                                 -- link: wlp0s20f3
                2a01:4f8:c0c:6848::1                        -- link: wlp0s20f3

-- Information acquired via protocol DNS in 22.3ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache network

The last three lines are the reason to use this command. It reports which interface answered, whether the answer was DNSSEC validated, whether it travelled encrypted, and whether it came from the cache or the network. Those are four questions people normally guess at. Here the answer came partly from cache, was not validated, and travelled in clear text, which is the ordinary state of a laptop talking to a home router.

4.3 getent hosts Versus getent ahosts

getent queries the NSS databases exactly as an application would. For names it has two forms, and they are not the same:

$ getent hosts petermartin.nl
2a01:4f8:c0c:6848::1 petermartin.nl

$ getent ahosts petermartin.nl
23.88.98.40     STREAM petermartin.nl
23.88.98.40     DGRAM
23.88.98.40     RAW
2a01:4f8:c0c:6848::1 STREAM
2a01:4f8:c0c:6848::1 DGRAM
2a01:4f8:c0c:6848::1 RAW

Look at the order. hosts returned the IPv6 address; ahosts put IPv4 first. Same machine, same name, same second. Section 7.1 explains why, and it is the single most useful surprise in this article.

The short version for now: ahosts is the one that tells you what an application will actually do. Use it, and ignore hosts unless you specifically want the legacy behaviour.

4.4 Where Did That Answer Come From?

Three tools, three different layers, and knowing which is which turns a confusing bug into a one-line diagnosis:

When those three disagree, the disagreement is the answer. getent differing from dig means something local is intercepting the name: an /etc/hosts entry, an mDNS responder, or a search domain. resolvectl differing from dig usually means a cache, and section 4.5 clears it.

4.5 Flushing the Cache

After a record changes, the local daemon may still be holding the old answer for the rest of its TTL:

$ resolvectl flush-caches

It prints nothing and returns immediately. That is success. Note what it does not do: it clears the cache on this machine only. Your router, your ISP's resolver and your browser all have their own, and none of them is affected. If a name still resolves to the old address after this command, the stale copy is not on your machine.

There is no cache to flush on a system without a local caching daemon. Plenty of servers run that way, and the absence of a cache is why a change appears there immediately and on your laptop five minutes later.

6. Advanced Use Cases

6.1 Turning On DNS over TLS and DNSSEC Validation

Classic DNS travels in plain text, so anyone on the path can read what you look up and change the answer. systemd-resolved can encrypt the hop to your upstream server and validate signatures, and on most systems both are off by default. The resolvectl status output in section 4.1 says so: -DNSOverTLS and DNSSEC=no/unsupported.

Both are set in /etc/systemd/resolved.conf. The shipped file is entirely commented out, which is a useful reference for the defaults:

$ grep -E '^#(DNS|FallbackDNS|DNSSEC|DNSOverTLS|Cache)' /etc/systemd/resolved.conf
#DNS=
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative

To encrypt, name a server that supports it and pin the hostname its certificate must match:

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com
DNSOverTLS=yes
DNSSEC=allow-downgrade

Then systemctl restart systemd-resolved and confirm with resolvectl status that the minus signs turned into plus signs.

Two warnings worth taking seriously. DNSOverTLS=yes is strict: if the server does not support it, resolution fails rather than falling back, which is the correct behaviour and also a good way to lose all name resolution on a captive-portal Wi-Fi. And DNSSEC=yes makes a broken signature anywhere in a chain fatal, including on domains whose owners have made a mistake you cannot fix. allow-downgrade validates where it can and does not break where it cannot, which is the setting most people actually want.

6.2 Split DNS, and Why a VPN Should Not Take Over Everything

A VPN typically needs two things: internal names must resolve through the tunnel, and everything else should keep using the local network. The old approach - rewrite /etc/resolv.conf and put the VPN's server first - sends every lookup you make through the company's resolver, including your personal browsing, and breaks whatever the local network was resolving.

Routing domains are the correct mechanism. A domain prefixed with ~ is not appended to short names; it only routes queries for that suffix to that link:

# route only corp.example.com through the VPN link
$ sudo resolvectl domain tun0 '~corp.example.com'
$ sudo resolvectl dns tun0 10.0.0.53

$ resolvectl domain
Link 3 (wlp0s20f3): localdomain
Link 8 (tun0): ~corp.example.com

Now db.corp.example.com goes to 10.0.0.53 and everything else goes to the Wi-Fi's server. To hand every query to a link, the routing domain is ~., a tilde and a dot: the root, meaning "everything". Seeing ~. on a VPN link tells you it has claimed all your DNS, which is sometimes intended and is worth knowing about either way.

resolvectl revert tun0 undoes per-link settings made this way.

6.3 Containers Resolve Names Differently From Their Host

A container does not inherit your machine's resolver settings, and the difference bites during exactly the debugging session where you can least afford a surprise. What it gets depends on which Docker network it joined, and the two cases behave completely differently.

On the default bridge, the container is handed the host's upstream server and nothing else:

$ docker run --rm alpine cat /etc/resolv.conf
# Generated by Docker Engine.
...
nameserver 192.168.1.1

$ docker run -d --name dnsdb2 alpine sleep 60
$ docker run --rm alpine getent hosts dnsdb2
$ echo $?
2                             # nothing: container names do not resolve here

On a user-defined network, Docker inserts its own embedded resolver and container names start working:

$ docker network create dnstest
$ docker run --rm --network dnstest alpine cat /etc/resolv.conf
# Generated by Docker Engine.
...
nameserver 127.0.0.11

$ docker run -d --network dnstest --name dnsdb alpine sleep 60
$ docker run --rm --network dnstest alpine getent hosts dnsdb
172.18.0.2        dnsdb  dnsdb

That address, 127.0.0.11, is Docker's built-in DNS server, and it is what resolves one container name to another. This is why service names work in a Docker Compose stack without anyone configuring DNS: Compose creates a user-defined network for the project automatically.

Notice also that the host was using 127.0.0.53 and the container on the default bridge was given 192.168.1.1. Docker deliberately does not pass a loopback address into a container, because inside the container's network namespace it would point at the container itself.

Two practical consequences. First, when a containerised application cannot reach its database, test resolution inside the container, not on the host, because they are answering from different resolvers. Second, when names do not resolve in a stack, check which network the containers are on before you look at anything else:

$ docker exec -it mycontainer cat /etc/resolv.conf
$ docker exec -it mycontainer getent hosts database

The same idea scales up. Kubernetes runs a cluster DNS service, usually CoreDNS, and gives every service a name such as database.default.svc.cluster.local so applications can find each other without hard-coded addresses. Inside modern platforms, DNS is the service-discovery layer rather than an internet directory.

6.4 Running Your Own Resolver, Without Becoming an Open One

There are good reasons to run a real recursive resolver on Linux: caching for a whole office, your own DNSSEC validation, independence from a third party, and logs you control. unbound is the usual choice; BIND can do it too, and dnsmasq is a lighter forwarder-with-a-cache that many home routers already run.

There is one way to get this badly wrong, and it is worth stating plainly. A recursive resolver that answers anyone on the internet is an open resolver, and it will be found and abused within days.

The abuse is amplification. DNS traditionally runs over UDP, which does not verify who sent a packet, so an attacker sends small queries with the victim's address forged as the source. Your server dutifully sends much larger answers to the victim. A modest request becomes a flood, the attacker spends almost nothing, and the traffic arrives from your address rather than theirs.

Preventing it is not complicated:

• Restrict recursion to clients you trust, meaning your own networks. This one setting removes the entire problem.
• Keep the two roles separate. A server that publishes your zones to the world is authoritative and needs no recursion at all. A server that resolves for your staff is recursive and should not be reachable from outside. Combining them on one public address is how this usually happens by accident.
• Turn on response rate limiting on anything that does face the internet.
• Bind to the right interface. A resolver listening on 0.0.0.0 on a machine with a public address is exposed even if you believed it was internal.

The general principle carries beyond DNS: before exposing any server, know which role it is meant to play, because the correct configuration differs per role and the failure mode here is that somebody else uses your bandwidth to attack a stranger.

6.5 Bypassing a Layer on Purpose

When you suspect one layer, take it out of the picture rather than reasoning about it. Each of these skips something specific:

$ resolvectl query --cache=no petermartin.nl    # skip the local cache
$ resolvectl query --protocol=dns NAME          # skip mDNS and LLMNR
$ dig @192.168.1.1 petermartin.nl               # skip the local daemon entirely
$ dig @1.1.1.1 petermartin.nl                   # skip the local network's server too
$ getent ahosts NAME                            # include every layer, like an application

Work down that list and the first command whose answer changes tells you which layer was responsible. It is faster than reading configuration files, and it works on a machine whose configuration you have never seen.

8. Best Practices

• Never edit /etc/resolv.conf. On most systems it is a generated symlink. Change the setting where it is produced: the NetworkManager connection profile, /etc/systemd/resolved.conf, or the DHCP server that handed it out.
• Use resolvectl status to find the real servers. The nameserver line in resolv.conf is almost always 127.0.0.53, which tells you nothing about the network.
• Prefer getent ahosts over getent hosts. It calls getaddrinfo() and returns addresses in the order applications will actually try them.
• Use the right tool for the question. getent for "what does this machine do", resolvectl query for "what does the local resolver have and from where", dig for "what does DNS itself say". When they disagree, that disagreement is the diagnosis.
• Check /etc/hosts first when one machine disagrees with all the others. It costs five seconds and it is the answer more often than anything else.
• Remove hosts-file entries when the migration is over. An override pointing at a decommissioned server produces a failure nobody can explain because nothing is wrong with the domain.
• Test container DNS inside the container. A container answers from a different resolver than its host, and container names only resolve on a user-defined network, not on the default bridge.
• Use routing domains for a VPN, not a global override. ~corp.example.com on the tunnel link routes only what belongs there. A link holding ~. has taken every lookup you make.
• Prefer DNSSEC=allow-downgrade to DNSSEC=yes unless you are prepared for a stranger's broken signature to break your resolution.
• Test DNSOverTLS=yes before you rely on it. It does not fall back, which is correct and will also cut you off on a captive-portal network.
• Never expose a recursive resolver to the internet. Restrict recursion to your own networks, keep authoritative and recursive roles on separate servers, and check what your resolver is bound to rather than what you assume.
• Remember that flushing the local cache flushes only the local cache. Your router, your ISP and your browser hold their own copies and none of them is affected.
• Read the documentation on the machine. man resolved.conf lists every setting with its default, man nsswitch.conf explains the action syntax in brackets, man gai.conf covers address ordering, and resolvectl --help lists twenty subcommands most people never discover.

10. Summary

On Linux, resolving a name is a stack, not a lookup. Almost everything confusing about it comes from asking one layer a question that another layer answered.

• Four layers sit between an application and the network: the application itself, NSS (/etc/nsswitch.conf), the stub resolver (/etc/resolv.conf), and a local caching daemon such as systemd-resolved.
• DNS is nearly the last source consulted, not the first. files comes before dns in nsswitch.conf, so /etc/hosts wins for every application on the machine.
• /etc/resolv.conf is a generated symlink containing 127.0.0.53, the local stub. It cannot tell you which servers the machine really uses; resolvectl status can, per interface.
• Three tools ask three different layers. getent ahosts behaves like an application, resolvectl query asks the local daemon and reports where the answer came from, and dig asks a name server directly. Their disagreement is the diagnosis.
• getent hosts calls gethostbyname2(); getent ahosts calls getaddrinfo(). Only the second returns addresses in the order software will actually try them.
• /etc/gai.conf decides IPv4 versus IPv6 preference for every program on the machine. It ships entirely commented out, and one uncommented line fixes the slow-first-connection problem caused by broken IPv6.
• Per-interface DNS is why the daemon exists. A routing domain written ~corp.example.com routes only that suffix through a link; ~. routes everything, which is what a VPN that "broke DNS" has usually done.
• Containers resolve differently. The default bridge gives the host's upstream server and no container names; a user-defined network gives Docker's own resolver on 127.0.0.11, and names work.
• systemd-resolved caches positive answers only (Cache=no-negative), and its cache is readable only as root.
• A recursive resolver exposed to the internet is an open resolver and becomes an amplification weapon within days. Restrict recursion, and keep authoritative and recursive roles apart.
• A browser with DNS over HTTPS ignores all of this, which is why it can disagree with every command-line tool on the same machine.

The commands worth keeping within reach:

WHAT IS THIS MACHINE USING
  resolvectl status              per-link servers, DNSSEC and DoT state
  resolvectl dns                 just the servers
  resolvectl domain              search domains, and ~routing domains
  nmcli dev show | grep DNS      what NetworkManager handed over
  grep '^hosts:' /etc/nsswitch.conf    the order that decides everything

WHAT DOES A NAME RESOLVE TO
  getent ahosts NAME             like an application, correct ordering
  resolvectl query NAME          + which link, cached or not, validated or not
  dig NAME                       DNS itself, ignoring every local layer
  dig +search NAME               with the search list, like an application

NARROWING IT DOWN
  resolvectl query --cache=no NAME     skip the local cache
  dig @192.168.1.1 NAME                skip the local daemon
  dig @1.1.1.1 NAME                    skip the local network
  sudo resolvectl monitor              watch queries as they happen

CHANGING IT, WHERE IT IS GENERATED
  nmcli connection modify ...          per network profile
  /etc/systemd/resolved.conf           DNS=, DNSSEC=, DNSOverTLS=, Cache=
  /etc/hosts                           override one name for everything
  resolvectl dns LINK SERVER           temporary; resolvectl revert LINK

DOCUMENTATION
  man resolved.conf   man nsswitch.conf   man gai.conf   resolvectl --help

And when a name resolves correctly on every machine but one, the cause is almost never the domain. It is a line somebody added to /etc/hosts during a migration that finished a long time ago.