HTTP Status Codes Explained: 301 vs 302, 404 vs 410

Two status codes are famous. 404 means the page is not there, 500 means the server fell over, and most people stop learning at that point. The trouble is that neither of those is the code that quietly costs you anything. The expensive ones are the codes nobody looks at: a 302 where you meant a 301, a 200 on a page whose text says "sorry, nothing here", a 503 on your robots.txt that stops Google crawling the entire site for twelve hours. All three look fine in a browser. All three change what search engines, caches and other machines do with your site.

This article explains what a status code actually is, where the three-digit shape came from, and what each code makes the other side do. It covers the five redirects and why they are not interchangeable, the difference between 404, 410 and a soft 404, the conditional and partial-content codes that make caching work, and it ends with the complete IANA register of every assigned code so you never have to guess what a number in a log file means.

The three digits at the top of every HTTP response, and what each one asks the other side to do.

Goal: after reading this you can look at any status code, say what it makes a browser, a cache and a crawler do, and choose the right one when you are the server.

2. Where the Name Comes From

The term is descriptive rather than clever. The first line of an HTTP response is called the status line, the number on it is the status code, and the words after it are the reason phrase. HTTP/2 and HTTP/3 kept the code, dropped the line and the phrase, and kept the name.

What is worth knowing is that HTTP did not invent the shape. A three-digit reply whose first digit gives the class was already standard practice on the internet when the web was built. RFC 959 (October 1985) specified it for FTP in almost the same words HTTP uses today:

Two differences are worth noticing, because both still catch people out.

First, FTP splits 4yz from 5yz on temporary versus permanent. HTTP splits 4xx from 5xx on whose fault it is. That is why 429 Too Many Requests feels wrong in the 4xx block: by FTP's rule it would be a 4yz, but by HTTP's rule the client did send too many requests, so it is a client error even though the sensible response is to wait rather than to change anything. Google's crawlers quietly disagree and count it as a server error (section 7.8).

Second, FTP gave the second digit a meaning too: x0z syntax, x1z information, x2z connections, x3z authentication, x5z file system. HTTP deliberately did not. In HTTP the last two digits are just a serial number inside the class, which is why 404 and 451 have nothing in common beyond both being refusals, and why codes could be added later without needing a free slot in the right sub-range.

4. Simple Use Cases: Reading the Code

Before choosing codes, you need to be able to see them. Everything in this section is one command, and none of it downloads the page.

4.1 The One Command Worth Memorising

curl -I sends a HEAD request: headers only, no body. Add -s to silence the progress meter.

$ curl -sI https://petermartin.nl/en/ | head -1
HTTP/2 200

One caveat that costs people an afternoon: a few servers and applications handle HEAD differently from GET, so a HEAD can report a code the real page never returns. When the answer surprises you, confirm it with a real GET that throws the body away:

$ curl -s -o /dev/null -w '%{http_code}\n' https://petermartin.nl/en/
200

The -w (write-out) form is the more useful of the two, because you can ask for several facts at once and get one line back:

$ curl -s -o /dev/null -L -w 'code=%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' \
    https://developers.google.com/search/docs/crawling-indexing/http-network-errors
code=200 redirects=1 final=https://developers.google.com/crawling/docs/troubleshooting/http-status-codes

Read that carefully, because it contains two different answers. The final code is 200, and a monitoring tool that only records the final code would call this URL healthy. It also took a redirect to get there: Google moved its own status-code documentation to a new path and left a 301 behind. Both facts matter, and only one of them is the "status code" people usually quote.

4.2 The Four Codes That Describe Most Sites

On an ordinary content site, four codes cover almost every response. If you understand these, you can read a log file.

Everything else in this article is a refinement of those four: a redirect that behaves differently, a "not found" that means something more specific, or a failure that points at a different machine.

4.3 Following a Redirect, Hop by Hop

The interesting part of a redirect is never the destination, it is the route. -L follows redirects and, combined with -I, prints the head of every response in the chain:

$ curl -sIL https://developers.google.com/search/docs/crawling-indexing/http-network-errors \
    | grep -Ei '^HTTP/|^location'
HTTP/2 301
location: /crawling/docs/troubleshooting/http-status-codes
HTTP/2 200

That is a healthy redirect: one hop, permanent, straight to a 200. What you are looking for is the opposite shape - three or four hops, a mix of 301 and 302, a hop through http://, or a chain that ends in a 404. Each of those is a real cost, and section 5.4 covers what breaks.

4.4 Checking a List of URLs

After a migration you do not want one answer, you want a few hundred. A three-line shell loop is enough, and the output sorts and counts:

$ while read -r u; do
    printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 "$u")" "$u"
  done < urls.txt
301 https://petermartin.nl/
200 https://petermartin.nl/en/
200 https://petermartin.nl/en/focus-on/web/dns
404 https://petermartin.nl/this-page-is-gone

Note that the site's own home page answers 301, redirecting to the language folder. That is normal and intended, but it is exactly the kind of thing you want to see written down before you change a template. Deliberate redirects and accidental ones look identical in a log.

4.5 Reading Codes You Did Not Request

Two other places show you status codes for traffic you never sent yourself.

• Your access log. Every request your server answered, with the code it answered with. Grouping a day of log lines by code tells you more about a site's health than any dashboard: a sudden bloom of 404s means a bad deploy or a link rot problem, a bloom of 499s or 5xxs means the application is falling behind.
• Browser developer tools. The Network tab lists every request the page made, with its status. This is where you find the failing background request that the page itself hides: a 200 page that pulls in a 404 script still looks fine until something needs that script.

For the crawler's view of the same thing - which codes Googlebot met, on which of your URLs, and what it did about them - see how to read the Search Console reports correctly.

6. Advanced Use Cases: The Codes the Protocol Uses

The codes in section 5 are the ones you configure. The ones below are mostly emitted by the protocol on your behalf, by caches, by proxies, and by APIs. You will not choose them often, but you will meet them, and each one exists because something specific goes wrong without it.

6.1 Conditional Requests: 304 and Friends

Caching is not "the browser stores the file and stops asking". Most of the time the browser does ask, and the server answers with a 304 Not Modified: a full request and response that carries no content at all.

First, a normal request. Notice the etag, which is the server's opaque fingerprint for this exact version of the file:

$ curl -sI https://petermartin.nl/media/vendor/joomla-custom-elements/css/joomla-alert.min.css
HTTP/2 200
content-type: text/css
content-length: 3654
last-modified: Tue, 18 Aug 2026 15:09:56 GMT
etag: "6a8475c4-e46"
cache-control: max-age=2592000

Now ask again, but say which version you already hold:

$ curl -sI -H 'If-None-Match: "6a8475c4-e46"' \
    https://petermartin.nl/media/vendor/joomla-custom-elements/css/joomla-alert.min.css
HTTP/2 304
last-modified: Tue, 18 Aug 2026 15:09:56 GMT
etag: "6a8475c4-e46"
cache-control: max-age=2592000
                          (and no content-length, because there is no content)

3654 bytes became zero. The round trip still happened, so a 304 is not free - on a slow connection the latency is most of the cost anyway - but nothing was transferred. There are two ways to trigger it: If-None-Match with an ETag, which is exact, and If-Modified-Since with a date, which is cruder and only has one-second resolution.

Three more codes belong to this family:

One detail catches people writing their own caching layer: RFC 9110 requires a 304 to repeat the header fields a 200 would have sent for Content-Location, Date, ETag, Vary, Cache-Control and Expires. A 304 that drops the ETag forces the client to re-download next time, which defeats the purpose.

6.2 Partial Content: 206 and 416

When a client asks for part of a file - seeking in a video, resuming an interrupted download, or a download manager fetching four chunks at once - it sends a Range header, and a server that supports it answers 206 Partial Content:

$ curl -sI -r 0-49 https://petermartin.nl/media/vendor/joomla-custom-elements/css/joomla-alert.min.css
HTTP/2 206
content-length: 50
etag: "6a8475c4-e46"

Ask for bytes that do not exist and you get the matching refusal:

$ curl -sI -r 999999999-1000000000 \
    https://petermartin.nl/media/vendor/joomla-custom-elements/css/joomla-alert.min.css
HTTP/2 416

416 Range Not Satisfiable is worth recognising because of where it shows up: video that will not scrub, and large downloads that fail only when resumed. Both are range problems, and both are invisible in a normal page load.

6.3 The Interim Responses: 100, 101 and 103

A single request can have several responses: any number of 1xx interim ones, then exactly one final response. They carry no body and are usually invisible.

• 100 Continue answers a client that sent Expect: 100-continue and is asking permission before uploading a large body. It saves you from streaming a 2 GB file to a server that was going to reject it on the first header.
• 101 Switching Protocols is how a connection stops being HTTP. It is the second half of the WebSocket handshake, and the server MUST name the new protocol in an Upgrade header.
• 103 Early Hints (RFC 8297, December 2017) is the interesting one. The server sends Link headers for resources it knows the page will need while it is still building the page, so the browser can start fetching the stylesheet during the seconds the application spends on the database. It is still an Experimental RFC, and browsers implement it with caveats: most accept early hints only over HTTP/2 or later, and process only the first one they receive.

One rule limits all three: a 1xx must never be sent to an HTTP/1.0 client, because HTTP/1.0 defined none and would not know what to do with it.

6.4 The Codes an API Owes Its Callers

A web page can survive on four codes. An API cannot, because its clients are programs that have to branch on the answer.

The 400 versus 422 line is the one worth holding to: 400 means "I could not understand this", 422 means "I understood it perfectly and the content is wrong". A client can retry the second one after fixing a field; retrying the first without changing the shape of the request is pointless.

Even so, a status code is deliberately coarse, and an API usually has more to say than one of 64 numbers can carry. The standard place for the rest is RFC 9457 (July 2023), Problem Details for HTTP APIs, which replaced RFC 7807. Rather than every API inventing its own error shape, it defines one JSON document, sent under the media type application/problem+json:

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json

{
  "type":     "https://example.com/problems/invalid-email",
  "title":    "Invalid email address",
  "status":   422,
  "detail":   "The supplied address has no domain part.",
  "instance": "/subscriptions/8814"
}

Each member has one job. type is a URI naming the kind of problem, and RFC 9457 says consumers "MUST use the type URI... as the problem type's primary identifier" - so that, not the text, is what a client branches on. title is a short human-readable summary of that kind and should not change between occurrences. detail explains this particular occurrence and should help the caller fix it rather than carry debugging output. instance identifies the occurrence itself. You may add your own members beside them.

status repeats the HTTP code, and the RFC is careful about why: it is "only advisory", and generators "MUST use the same status code in the actual HTTP response, to assure that generic HTTP software that does not understand this format still behaves correctly". It is there so the original code survives when an intermediary rewrites it or when the body is logged without its response.

That division of labour settles most arguments about which code an API should return: the status code carries the protocol-level meaning that generic infrastructure acts on, and the body carries the application-specific detail. A cache, a proxy and a retry loop can all behave correctly on the 422 without understanding a word about email addresses.

6.5 The Codes That Come From the Middle

Some codes are never sent by your application, because they are answers about the connection rather than about the resource.

• 421 Misdirected Request means the request arrived at a server that cannot answer authoritatively for that hostname. With HTTP/2 connection reuse, a browser may send a request for one domain over a connection it opened for another, and this is the correct way to say "not here, open a new connection". RFC 9110 forbids a proxy from generating it.
• 425 Too Early (RFC 8470, September 2018) exists because of TLS 1.3. Its 0-RTT feature lets a client send data with the very first packet, which is fast and replayable by an attacker. A server that will not take that risk for this request answers 425 and the client retries once the handshake is complete.
• 426 Upgrade Required refuses the request until the client speaks a different protocol - in practice, a plaintext client being told to come back over TLS.
• 451 Unavailable For Legal Reasons (RFC 7725, February 2016) says the refusal is a legal demand, not a technical one. The RFC asks for a Link header with rel="blocked-by" identifying the entity doing the blocking, and notes that the code "implies neither the existence nor nonexistence of the resource". The number is a reference to Ray Bradbury's Fahrenheit 451.
• 511 Network Authentication Required is the captive portal code: the hotel wifi, not the website. RFC 6585 says origin servers SHOULD NOT generate it, because it is for intercepting proxies.

6.6 Which Codes Are Safe to Retry

For a failed request, the status code is usually an instruction about trying again: come back later, or do not bother. Automated clients act on that instruction thousands of times an hour, and getting it wrong is how one slow minute becomes an outage - every client that retries immediately adds load to a server that already said it had too much.

502 and 504 are the dangerous pair, and the danger is not obvious. Both are reported by a gateway that did not get a usable answer from upstream, which tells you nothing about whether upstream did the work. A 504 can mean the application never saw the request, or that it processed the order, charged the card and took too long to say so. Retry that automatically and you have charged the card twice.

A status code tells you whether the server would like another attempt. It does not tell you whether another attempt is safe. That is a property of the request method, not of the response.

RFC 9110 draws the line at the method: a client "SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent... or some means to detect that the original request was never applied". Which methods carry that promise, and how to give a POST one with an idempotency key, is the subject of methods rather than status codes, so this article stops at the boundary. Two hard rules from the same section are worth carrying across it: "A proxy MUST NOT automatically retry non-idempotent requests", and "A client SHOULD NOT automatically retry a failed automatic retry". The second is the one that prevents a retry storm.

When you do retry, the order of precedence is simple, and only the first line of it comes from the specification:

RETRY LADDER
  Retry-After present   obey it. The server has told you the answer
  no Retry-After        back off exponentially: 1s, 2s, 4s, 8s, 16s
  every attempt         add random jitter, so clients do not resynchronise
  always                cap the attempts AND set a total deadline
  after a failed retry  stop. Escalate to a human or a queue

  Retry-After is defined by RFC 9110. Backoff, jitter and caps are
  engineering practice, not protocol.

Jitter is the part people leave out. If a thousand clients all receive Retry-After: 120 at the same moment, obeying it perfectly means a thousand requests arrive in the same second, two minutes later - which is how a server that was recovering gets knocked over by the clients that were being polite.

6.7 Which Codes a Cache Will Store Without Being Asked

This surprises almost everyone. RFC 9110 lists the status codes that are heuristically cacheable, meaning a cache may store and reuse them even when you sent no caching headers at all:

200  203  204  206  300  301  308  404  405  410  414  501

everything else is NOT heuristically cacheable
       ... including 302, 307, 429 and every 5xx

Read that list twice. 404, 405, 410, 414 and 501 are in it. A 404 served with no cache headers may be remembered by a proxy or a CDN, which is why a page can keep returning "not found" for some visitors minutes after you published it. If your 404 handler is dynamic, say so explicitly with Cache-Control: no-store.

When you suspect that is what happened, the response says so. RFC 9111 defines Age as a cache's estimate of how many seconds ago the response was generated, and states the consequence plainly: "the presence of an Age header field implies that the response was not generated or validated by the origin server for this request".

$ curl -sI https://www.iana.org/ | grep -Ei '^HTTP/|^cache-control|^age:'
HTTP/2 200
cache-control: public, max-age=3600
age: 3023        ── a cache answered. This copy was made 3023 seconds ago,
                    so it has about 10 minutes of freshness left

That single header separates "my server is still wrong" from "my server is fine and something in front of it is still holding the old answer". Check it before you change anything, because the second case is fixed by waiting or purging, not by editing.

The flip side is just as useful: 302 and 307 are not on the list, which is the real difference between them and 301/308, and every 5xx is excluded, so a cache will not enshrine your outage.

6.8 The Codes That Are Not HTTP

Some numbers you will meet in logs and error pages are not in the standard at all. They are vendor extensions, and knowing which is which saves you searching the RFCs for something that was never there.

The distinction matters when you are debugging: a Cloudflare 52x means the edge could not get a usable answer from your server, so nothing in your application log will explain it. Look at the connection, the certificate and the firewall instead.

Do not invent codes of your own. RFC 959 gave that advice for FTP in 1985 - implementations "should not invent new codes for situations that are only slightly different from the ones described here" - and it has aged well. A client that meets your 499 will treat it as a 400, and you will have gained nothing but a number nobody can look up.

6.9 The Complete Register

Every status code lives in one place: the IANA HTTP Status Code Registry. RFC 9110 requires a registration to carry the code, a short description and a pointer to specification text, and adding one needs IETF Review. That is why the list grows by roughly one code every two years rather than one a week.

Below is the whole register as it stood on 7 September 2026: 64 named code points out of the 500 the range allows. Four of them cannot really be used - 305 is deprecated, 306 and 418 are reserved, 510 is obsoleted - and one, 104, holds a temporary registration that expires in November 2026. The rest of the range is genuinely empty, which is why a number you cannot find here is a vendor invention.

The shape of that table is the story of the web. There are 29 client-error codes and 11 server-error codes, because most of the ways an HTTP conversation can fail are ways the request can be wrong. And there are nine redirects for what is essentially one idea, because the first two were specified before anyone knew what browsers would actually do with them.

8. Best Practices

• Make the status code and the page agree. If the page says "not found", the status line must say 404. Every soft 404 on a site is a page that lies to machines and confuses nobody else.
• Use 308 and 307 for anything that can be posted to, and 301 or 302 only for pages that are always fetched. The method-rewriting behaviour of 301 and 302 is permitted by the standard, so it is not a bug you can report.
• Test with 302, publish with 301. A permanent redirect is cached by browsers by default and is genuinely hard to take back.
• Redirect once, to the final URL. Before adding a rule, check whether an existing rule already points at the URL you are about to leave. Chains cost round trips and rot silently.
• Return 503 with Retry-After for maintenance, and exempt /robots.txt from it. A 200 maintenance page gets indexed; a 503 on robots.txt stops crawling of the whole site.
• Send the headers the code requires. 401 needs WWW-Authenticate, 405 needs Allow, 429 and 503 want Retry-After, 201 wants Location. The code alone tells a client what happened; the header tells it what to do.
• Set Cache-Control explicitly on error pages. 404, 405, 410, 414 and 501 are heuristically cacheable, so silence is a decision you did not know you were making.
• Rate limit with 429, never with 403. One asks the caller to slow down; the other tells search engines the content is off limits.
• Do not let your codes enumerate your data. Answering 404 for what does not exist and 403 for what exists but is private tells an outsider which is which. Where that matters, return the same code for both.
• Retry on the codes that invite it, and never blindly on 502 or 504. Obey Retry-After where it is sent, back off with jitter where it is not, cap the attempts, and never retry a failed retry.
• Distinguish 400 from 422 in APIs. Unparseable is not the same as invalid, and clients can only act sensibly on the difference.
• Never branch on the reason phrase. It is advisory in HTTP/1.1 and absent in HTTP/2 and HTTP/3.
• Check codes after every deploy and every migration, in bulk, from outside the network. A three-line loop over a URL list catches more real problems than any dashboard.
• Monitor for the absence of a code too. DNS failures, refused connections and expired certificates produce no status code at all, and they are the outages that take a whole site down.
• Read the primary sources. RFC 9110 section 15 defines every code in one place and is far more readable than its reputation; the IANA registry is the only complete list and tells you which document defines each code. For what search engines do with them, Google's own page on how status codes affect its crawlers is short, current, and more reliable than any summary of it.

10. Summary

A status code is three digits that tell the other machine what to do. Almost every expensive mistake with them comes from treating them as a description of the page instead.

• The first digit is the class, and it is the only part a client is required to understand. An unrecognised code is handled as the x00 of its class, which is why new codes can be deployed safely. Valid codes run from 100 to 599; anything outside is treated as a 5xx.
• The 4xx/5xx split decides who fixes it: the request was wrong, or the server failed a reasonable request. The 1992 draft already admitted the two cannot always be told apart.
• The reason phrase is decoration. It is advisory in HTTP/1.1 and does not exist in HTTP/2 or HTTP/3, which carry a bare :status. Never match on the text.
• HTTP/0.9 had no status codes, and the 1991 specification says why that failed: "there is no way to distinguish an error response from a satisfactory response except for the content of the text". Every soft 404 recreates that problem deliberately.
• The five redirects differ on three things: whether the method survives, whether a cache may store the redirect without being told, and how strong a signal a search engine takes from it. 301 and 308 are cacheable and strong; 302, 303 and 307 are neither.
• 301 and 302 may turn a POST into a GET, by permission of the standard and by the behaviour of curl and every browser. Use 308 or 307 for anything postable, or watch form data disappear without an error.
• 404 is "nothing here", 410 is "deliberately gone", and a soft 404 is a 200 that lies. Google treats all 4xx except 429 the same, so 410 is about honesty rather than speed.
• 401 means unauthenticated, 403 means refused. Retrying with credentials can fix the first and not the second, and a server may answer 404 to hide that a forbidden resource exists at all.
• 429 is the one 4xx that means "come back later", and Google's crawlers count it as a server error rather than a client error. It is the correct way to shed crawl load; 403 is not.
• 503 with Retry-After is the only honest answer for planned downtime - and /robots.txt must be exempt from it, because a 5xx on that one file stops crawling of the whole site for 12 hours and starts a 30-day clock.
• 500, 502 and 504 name three different machines: the application failed, the gateway got no valid answer, the gateway gave up waiting. Only the first leaves a stack trace.
• Conditional requests are why caching works. An ETag plus If-None-Match turns a download into a 304 with no body - still a round trip, but no content.
• An API needs more than a code. RFC 9457 Problem Details puts the protocol meaning in the status line and the application detail in an application/problem+json body, so infrastructure can act on the code while the caller reads the reason.
• Some error codes are cached by default. 404, 405, 410, 414 and 501 are heuristically cacheable; 302, 307, 429 and every 5xx are not.
• The code says whether to retry; the method says whether it is safe. 408, 421, 425, 429 and 503 invite another attempt, and 502 and 504 hide the real question, because the work may have completed after the gateway stopped waiting.
• IANA has assigned 64 code points out of 500. Four are unusable - 305 deprecated, 306 and 418 reserved, 510 obsoleted - and 418 is blocked permanently by an April Fools RFC from 1998. Numbers outside the register, such as nginx's 444 and 499 or Cloudflare's 52x, are vendor inventions.
• The worst failures have no status code. An expired domain, a broken name server, a closed port and an expired certificate all fail before HTTP starts, which is why status-code monitoring alone reports nothing during the largest outages.

The reference worth keeping next to a terminal:

THE CLASSES
  1xx  interim, more is coming      4xx  your request was wrong
  2xx  it worked                    5xx  the server failed
  3xx  go somewhere else            unknown code → treat as x00 of its class

THE FIVE REDIRECTS
              method kept?   cacheable by default?   search signal
  301           no                 YES                 strong
  302           no                 no                  weak
  303           no (always GET)    no                  weak
  307           YES                no                  weak
  308           YES                YES                 strong
  postable URL → 308 or 307.   testing → 302, then switch to 301.

CHOOSING THE ERROR
  gone, might return         404
  gone on purpose, forever   410
  not logged in              401  + WWW-Authenticate
  logged in, still no        403
  too many requests          429  + Retry-After
  down right now             503  + Retry-After   (but NOT on /robots.txt)
  app crashed                500       gateway got nothing valid   502
  gateway timed out          504       validation failed           422

READING A CODE
  curl -sI URL | head -1                     the code, no body
  curl -s -o /dev/null -w '%{http_code}\n'   the code from a real GET
  curl -sIL URL | grep -Ei '^HTTP/|^loc'     every hop in the chain
  curl -s -o /dev/null -L \
    -w 'code=%{http_code} hops=%{num_redirects}\n'    both at once

HEADERS THE CODE REQUIRES
  401 → WWW-Authenticate     405 → Allow        201 → Location
  429 → Retry-After          503 → Retry-After  304 → ETag, Vary, Date

CACHED WITHOUT BEING ASKED  (heuristically cacheable)
  200 203 204 206 300 301 308 404 405 410 414 501
  NOT: 302 307 429 and every 5xx
  Age: header present → a cache answered, not your origin server

RETRYING
  invited        408  421  425  429  503      (429 and 503: obey Retry-After)
  careful        502  504   the work may have completed anyway
  pointless      400 404 405 409 410 414 415 422  until you change the request
  no Retry-After back off 1s 2s 4s 8s, add jitter, cap it
  never          a retry of a failed automatic retry

HOP LIMITS
  browsers 20    curl 50    Googlebot 10    Googlebot on robots.txt 5
  Google Inspection Tools: does not follow redirects at all

NO STATUS CODE AT ALL
  NXDOMAIN, connection refused, TLS handshake failure, timeout
  check name → connection → certificate → only then the code

Verified 7 September 2026 against RFC 9110 (HTTP semantics), RFC 9113 (HTTP/2),
the IANA HTTP Status Code Registry, and Google's crawler documentation.

Status codes are the smallest part of a response and the only part everything else obeys. Once you can read them, most "the site is broken" reports resolve into a single number and a single machine. And when pages quietly disappear from search while every human check says the site is fine, the answer is often one file nobody ever visits: a robots.txt that answered 503 for an afternoon.