Another silent killer is the wildcard and its interaction with Google’s interpretation of character limits. Robots.txt supports limited pattern matching, but many webmasters try to be clever with overly broad `Allow` and `Disallow` rules. For instance, `Disallow: /?utm_` is intended to block all query strings containing “utm_” to prevent parameter-based duplicates. That works—until you inadvertently block a query parameter that your CMS uses for pagination or filtering on product listing pages. Suddenly, your category pages with sort or page numbers get a `Disallow` hit, and those pages vanish from the crawl queue. The index only retains the canonical version, if one exists, but you lose deeper crawl paths. The fix is not to avoid robots.txt block rules entirely but to test each pattern using the Google-supported tester or a local crawler (like Screaming Frog with a custom robots.txt) before deployment. Remember that Google treats `Disallow` as an absolute block for the designated URL path, and overlapping rules can create a cascade that locks out entire sections of your site.

Then there’s the “sitemap” directive within robots.txt. This is meant to point crawlers to your primary XML sitemap, but it’s often misconfigured or placed at the wrong depth. If your robots.txt lives at `https://example.com/robots.txt` and your sitemap is at `https://example.com/sitemaps/prod/sitemap.xml`, you need to include the full URL. A relative path like `Sitemap: /sitemaps/prod/sitemap.xml` is technically valid, but if you ever move the robots.txt to a subdomain or subdirectory, the relative reference breaks. Worse, some CMS platforms generate a robots.txt dynamically and inject the sitemap pointer only when a certain plugin is active. If the plugin deactivates during a deployment, the sitemap directive disappears, and Google may take weeks to discover new content. Always validate that the `Sitemap` field exists and resolves to a 200 response from the crawler’s user-agent.

A more advanced trap involves user-agent-specific rules for Googlebot versus Googlebot-Image versus Googlebot-News. Many sites disallow “ia_archiver” or “Baiduspider” but leave the `User-agent: ` section open. That’s fine for resource allocation, but if you unintentionally place a `Disallow: /` under `User-agent: Googlebot-Image` while keeping `Allow: /` for `User-agent: `, Google’s image crawler will completely ignore your image assets. Since images are often indexed separately and can drive substantial traffic via image search, this oversight silently devalues a major acquisition channel. The fix is to audit each user-agent directive with a dedicated rule set, not just the generic wildcard. Use Google’s robots.txt testing tool (still available in Search Console legacy tools) to simulate each agent’s perspective.

Finally, do not underestimate the impact of time-to-live and caching on robots.txt. If you update your robots.txt to unblock a path but your CDN or server caches the old version for 24 hours, Googlebot will continue to receive the stale directive. Indexation delays compound because Google usually fetches robots.txt at the start of each crawl session and retains it for up to five days. That means a five-day cache-to-uncache cycle can waste an entire week of crawl opportunities. Implement a low `max-age` header (e.g., 3600 seconds) for robots.txt, or use a `Cache-Control: no-cache` directive. Check your server’s response headers for the file to ensure it reflects the latest rule set.

Robots.txt is not set-and-forget. It is a living file that must be revisited after every site migration, CMS update, or content restructuring. The most dangerous misconfigurations are the ones that don’t throw errors—they just quietly starve your indexation pipeline. If you’re performing a technical SEO health check, make robots.txt the first file you manually inspect, not the last. Your crawl budget and index coverage depend on it.