Problem
A page is blocked from crawling while the site expects its noindex directive to be seen.
Why it happens
robots.txt controls crawler access, while noindex is a page-level indexing instruction. A crawler blocked before fetching a page may not read its noindex.
How to diagnose
Check the robots response, the exact path rules and the page’s meta robots or X-Robots-Tag. Test with the relevant crawler user agent and confirm the final URL after redirects.
How to fix
If the goal is to prevent indexing, allow the crawler to fetch the page and serve noindex, or protect the page behind authentication. Use robots.txt to manage crawl access, not as a privacy barrier.
How to verify
Fetch robots.txt and the page again. Confirm the crawler can see the intended directive, then inspect index status in the property you own.