Both robots.txt and a noindex tag are used to keep pages out of Google, but they work in different ways and solve different problems. Using the wrong one is a common reason a page you tried to hide keeps showing up in search results anyway.
What robots.txt does
A robots.txt file sits at the root of your website (for example, yourdomain.com/robots.txt) and tells search engine crawlers which parts of your site they're allowed to visit. It's an instruction to stay away, not an instruction to hide something that's already been seen.
- It blocks crawling: search engines won't fetch the content of a disallowed page.
- It does not remove a page that's already indexed.
- It's a request, not an enforcement mechanism. Well-behaved search engines respect it, but it can't physically stop a page from being fetched.
What a noindex tag does
A noindex tag is a meta tag placed in the <head> of a specific page's HTML (or sent as an HTTP header). It tells a search engine: "you can read this page, but don't list it in search results."
- It blocks indexing: the page stays out of search results even though crawlers can still visit it.
- It only works if the search engine is actually allowed to crawl the page in the first place, since it needs to read the tag.
- It's added per page, usually through your SEO plugin's page settings (on WordPress) or directly in your theme's template code.
Why blocking a page in robots.txt doesn't always keep it out of Google
This is the part that catches most website owners off guard. If a page is disallowed in robots.txt but other websites link to it, Google can still show that URL in search results, just without a proper title or description, because it was never allowed to crawl the page to read a noindex tag on it.
If your real goal is "don't let this page show up in Google search results," use a noindex tag, not robots.txt. If your goal is "don't let search engines waste time crawling this section of my site" (like an internal search results page or an admin folder), robots.txt is the right tool.
Which one should you use?
- To stop a page from appearing in Google search results (a thank-you page, an internal staging page, a duplicate page): use a noindex tag, and make sure the page is not blocked in robots.txt.
- To stop search engines from crawling low-value sections of your site (shopping cart pages, filtered search URLs, admin areas) purely to save crawl budget, not to hide them from search results: use robots.txt.
- To remove a page that's already indexed and showing up in search results right now: add a noindex tag first, wait for Google to recrawl and drop it, then you can block it in robots.txt afterward if you also want to stop future crawling.
If you're not sure whether a page is currently blocked, crawled, or indexed, check it directly in Google Search Console's URL Inspection tool before changing anything, so you're fixing the actual problem rather than guessing.
For the basics of each one on its own, see What Is a robots.txt File and How Do You Use It? and How to Add a noindex Tag to Keep a Page Out of Google Search Results.