robots.txt in Joomla

2. The Joomla robots.txt File

Joomla ships a sensible default file. Here is the body that does the work, after the comment block at the top:

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

2.1 What Each Line Means

• User-agent: * means the rules below apply to every crawler.
• Each Disallow line names a folder that crawlers should skip.

The list is made of Joomla's internal folders. None of them contain pages a visitor should land on from a search result. They hold the admin panel (/administrator/), the headless API (/api/), core code (/components/, /libraries/, /modules/, /plugins/), and working folders (/cache/, /tmp/, /logs/). Keeping crawlers out of them saves "crawl budget" for your real content and avoids indexing files that should never be public.

2.2 What the File Does Not Block

Notice what is missing: there is no Disallow: /images/ and no Disallow: /media/. That is on purpose. Search engines need to read your CSS, JavaScript, and images to render and rank a page correctly. Blocking those folders is a classic SEO mistake (see section 12). The default file blocks code folders, not asset folders.

2.3 The Comment Block at the Top

Above the rules, the shipped file carries an important note about subfolder installs:

# If the Joomla site is installed within a folder
# eg www.example.com/joomla/ then the robots.txt file
# MUST be moved to the site root
# eg www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to all of the
# paths.

This matters because crawlers only ever read robots.txt from the domain root. Section 6 explains exactly what to change for a subfolder install.

4. robots.txt Versus Meta Robots

Joomla gives you two separate ways to influence crawlers, and confusing them causes a lot of indexing problems. They work at different layers and have different effects.

4.1 The Trap: Disallow Plus Noindex

There is one combination that surprises people. If you Disallow a page in robots.txt and set a noindex meta tag on it, the noindex never works. The reason is logical: a crawler that is told not to fetch the page never reads its HTML, so it never sees the noindex tag.

To remove a page from search results, let crawlers fetch it (do not Disallow it) and add a noindex meta tag. The crawler reads the page, sees the tag, and drops it from the index.

So the rule of thumb is: use robots.txt to manage crawling of whole folders, and use the meta robots tag to manage indexing of individual pages.

6. Subfolder Installs

Crawlers only read robots.txt from the domain root, never from a subfolder. If your Joomla site lives at https://example.test/blog/, a crawler still asks for https://example.test/robots.txt, not https://example.test/blog/robots.txt.

The shipped file's own comment block explains the two steps you must take:

1. Move the file to the real domain root. Copy robots.txt from /blog/ up to the top-level folder so the crawler can find it.
2. Prefix every path with the folder name. Each Disallow path must include the subfolder, or it points at the wrong place.

For a site in /blog/, the rules change like this:

User-agent: *
Disallow: /blog/administrator/
Disallow: /blog/api/
Disallow: /blog/cache/
Disallow: /blog/tmp/
# ...and so on for every folder...

If you forget the prefix, the rules silently protect nothing, because /administrator/ does not exist at the domain root in a subfolder install.

8. Under the Hood (Developer View)

8.1 How the File Is Served

There is no PHP behind robots.txt. It is a static file, and the web server delivers it directly. In Joomla's .htaccess, the front-controller rewrite only fires when the request does not match a real file (!-f). Because robots.txt is a real file on disk, Apache serves it straight away and the request never reaches index.php. The same is true on Nginx with the usual try_files rule.

This has a practical consequence: you cannot change robots.txt from the Joomla admin panel. You edit it with FTP, the hosting file manager, or a code editor.

8.2 Where the Meta Robots Tag Is Built

The meta robots cascade from section 5 is real code, not magic. The global value is read and stored as a page parameter in SiteApplication:

// libraries/src/Application/SiteApplication.php (simplified)
$robots = $this->get('robots');
// ...later, after merging menu item params...
$params[$hash]->def('robots', $robots);

The component view then applies it, and lets the item's own metadata override it. In the com_content article view:

// components/com_content/src/View/Article/HtmlView.php (simplified)
if ($this->params->get('robots')) {
    $this->getDocument()->setMetaData('robots', $this->params->get('robots'));
}

// ...then the article's own metadata is applied last, so it wins...
foreach ($this->item->metadata->toArray() as $k => $v) {
    if ($v) {
        $this->getDocument()->setMetaData($k, $v);
    }
}

Because the article's own metadata is written last, the article-level Robots setting overrides the menu and global values. That is exactly the "most specific wins" behaviour you see in the backend.

8.3 The Print View Is Always Noindex

One detail catches developers out. When a page is rendered in Joomla's print view, the article view forces a noindex, nofollow tag regardless of your settings:

if ($this->print) {
    $this->getDocument()->setMetaData('robots', 'noindex, nofollow');
}

This stops the stripped-down print version of a page from competing with the real page in search results. You do not need to configure it; Joomla does it for you.

8.4 The Disallowed /api/ Folder

The default file disallows /api/, the entry point for Joomla's Web Services API. There is no reason for a search engine to crawl your REST endpoints, and the API requires a token anyway. Keeping it out of robots.txt is good hygiene. If you build a public, link-shareable view on top of the API, expose it through a normal front-end menu item, not through the raw /api/ path.

8.5 The HTTP Status Code Matters

Because robots.txt is fetched over HTTP, the status code your server returns changes how crawlers behave. This is easy to overlook and occasionally causes a whole site to stop being crawled.

The lesson: a broken or firewalled robots.txt is worse than a missing one. If maintenance mode or a security plugin starts returning a 503 for every request, including robots.txt, Google reads that as "stop crawling" and visits slow down. Confirm the file returns a clean 200 with a quick request:

curl -I https://example.test/robots.txt

8.6 The X-Robots-Tag Header

The meta robots tag only works for HTML pages, because it lives in the <head>. Files such as PDFs, images, and other downloads have no <head>, so you cannot add a noindex tag to them. For those, the server sends an HTTP header instead:

X-Robots-Tag: noindex

It accepts the same values as the meta tag (noindex, nofollow, and so on) but applies them to any file type. A common use is to keep generated PDFs out of search results. In Apache you scope it by file type:

<FilesMatch "\.pdf$">
    <IfModule mod_headers.c>
        Header set X-Robots-Tag "noindex"
    </IfModule>
</FilesMatch>

Like the meta tag, X-Robots-Tag controls indexing, not crawling. The crawler must be allowed to fetch the file to see the header, so do not also Disallow the file in robots.txt (the same trap as section 4.1). Joomla core does not set this header for you; it is a server-level addition.

10. robots.txt Behind a CDN

Many Joomla sites sit behind a content delivery network such as Cloudflare, Fastly, or Bunny. The CDN sits between the crawler and your origin server, and it changes how robots.txt is delivered.

10.1 The File Is Cached at the Edge

A CDN caches static files, and robots.txt is static. After you edit the file on your server, the CDN may keep serving the old copy for minutes or hours until its cache expires. On top of that, crawlers cache robots.txt themselves, often for about a day. So a change is rarely instant.

After an important change, purge the CDN cache for robots.txt so the new rules go live at once. In Cloudflare this is a single "Purge by URL" action.

10.2 Watch for Two Sources of Truth

Some CDNs can generate or inject their own robots.txt, or rewrite the one you serve. Cloudflare, for example, can add a managed robots.txt for its bot-management features. If your live file does not match what you uploaded, check the CDN dashboard before you blame Joomla. There should be exactly one place that owns the file.

10.3 Confirm It Still Returns 200

A firewall rule or bot-fight setting at the CDN can accidentally block or challenge the request for robots.txt. As section 8.5 explained, a 403 or 5xx on this file harms crawling. Test the public URL through the CDN, not just on the origin:

curl -I https://example.test/robots.txt

A clean 200 OK with Content-Type: text/plain confirms crawlers can read it.

12. Common Mistakes and Pitfalls

12.1 The Accidental Site-Wide Block

Symptom: your whole site vanishes from Google after a redesign or a move from a staging server.

Fix: check for Disallow: / in robots.txt. Staging sites often block everything to stay private, and that file gets copied to production by mistake. Remove the line so crawlers can reach your pages again.

12.2 Blocking CSS, JavaScript, or Images

Symptom: Google Search Console reports that pages cannot be rendered, or rankings drop after you "tidied up" the file.

Fix: remove any Disallow for /images/, /media/, or template asset folders. Crawlers need those files to see the page as a visitor does.

12.3 Expecting robots.txt to Hide a Page

Symptom: a page you disallowed still appears in Google, often as a bare link with no description.

Fix: Disallow stops crawling, not indexing. Remove the Disallow, set the page's Robots to noindex in Joomla, and let the crawler fetch it and see the tag.

12.4 Disallow Plus Noindex Cancel Out

Symptom: you set both a Disallow and a noindex, yet the page refuses to leave the index.

Fix: the crawler never reads the noindex because the Disallow keeps it from fetching the page. Drop the Disallow and keep only the noindex.

12.5 The Subfolder Path Mistake

Symptom: on a subfolder install, the rules seem to protect nothing.

Fix: move the file to the domain root and prefix every path with the folder name, for example Disallow: /blog/administrator/ (see section 6).

12.6 Treating robots.txt as Security

Symptom: you list a secret folder in robots.txt to "hide" it, and then find it being probed.

Fix: never name sensitive paths in a public file; you are advertising them. Protect them with a login or server access rules instead. robots.txt is advice for polite crawlers, not a wall.

12.7 Blocking a Page You Marked Canonical

Symptom: you set a rel="canonical" tag on a duplicate URL, yet Google keeps indexing the duplicate instead of the canonical version.

Fix: the duplicate is blocked in robots.txt, so Google never crawls it and never reads its canonical tag. Remove the Disallow for that URL and let Google fetch it; only then can it honour the canonical.

12.8 A Broken robots.txt Returning an Error

Symptom: crawl rate drops to almost nothing after enabling maintenance mode, a firewall, or a CDN bot rule.

Fix: the request for robots.txt returns a 403 or 5xx, which Google reads as "stop crawling". Make sure robots.txt always returns a clean 200, even during maintenance (see section 8.5).

14. Quick Reference

LOCATION     public_html/robots.txt  (served live, no rename needed)
VIEW IT      https://your-site.test/robots.txt
ALL CRAWLERS User-agent: *
BLOCK FOLDER Disallow: /folder/
ALLOW ALL    Disallow:            (empty value)
BLOCK SITE   Disallow: /          (staging only, never live)
EXCEPTION    Allow: /folder/file.pdf
WILDCARDS    * = any chars, $ = end of URL  (Disallow: /*?print=)
SITEMAP      Sitemap: https://your-site.test/sitemap.xml
META ROBOTS  Global Config → Site → Metadata → Robots
CASCADE      Article > Menu item > Global  (most specific wins)
VALUES       index,follow / noindex,follow / index,nofollow / noindex,nofollow
EMPTY DEFAULT No robots meta tag written = index, follow
HIDE A PAGE  noindex meta tag, do NOT also Disallow it
NON-HTML     X-Robots-Tag: noindex  (header, for PDFs/images)
CANONICAL    Keep duplicate crawlable or its canonical is never read
SUBFOLDER    Move to root + prefix paths: /blog/administrator/
PRINT VIEW   Always noindex, nofollow (forced by Joomla)
HTTP STATUS  Must return 200; 403/5xx stalls crawling
AI CRAWLERS  GPTBot, ClaudeBot, Google-Extended, CCBot (block per UA)
CDN          robots.txt is edge-cached; purge after edits
NOT SECURITY robots.txt is public advice, not an access lock