A robots.txt file is a small plain-text file that sits in your website's root folder and gives instructions to search engine crawlers (like Googlebot) about which pages or folders they're allowed to visit. It doesn't control who can see your site in a browser. It only guides automated crawlers, and well-behaved ones like Google's generally respect it.
Most websites don't strictly need one, but it's useful when you want to keep crawlers away from admin areas, staging folders, or duplicate content, or when you want to point search engines straight to your sitemap.
What robots.txt actually controls
A robots.txt file uses simple rules to allow or disallow crawling of specific paths. It does not remove pages that are already indexed, and it doesn't stop humans from visiting a URL directly if they have the link. Think of it as a set of polite instructions for automated bots, not a security tool.
Never rely on robots.txt to hide sensitive information. Anyone can view the file directly by visiting yourdomain.com/robots.txt, and disallowing a folder there can actually draw attention to it. Use password protection for anything that genuinely needs to stay private.
Creating or editing your robots.txt file
The file must be named exactly robots.txt (all lowercase) and placed in the root of your website, usually the public_html folder. The process is the same whether your hosting account runs on cPanel or DirectAdmin: open your control panel's File Manager, navigate to public_html, and create or edit the file there as plain text.
- Log in to your hosting control panel.
- Open File Manager.
- Go into the public_html folder (or the specific folder for your domain, if you host multiple sites).
- If a robots.txt file already exists, select it and click Edit. If not, create a new file and name it robots.txt.
- Add your rules, then save the file.
Basic rules you can use
A robots.txt file is made up of simple blocks that target one or more crawlers. Here are a few common examples:
- Allow every crawler to access everything: User-agent: * followed by Disallow: (left empty)
- Block crawlers from a specific folder, such as an admin area: User-agent: * followed by Disallow: /admin/
- Point crawlers to your sitemap so they can find your pages faster: Sitemap: https://yourdomain.com/sitemap.xml
If you've already generated a sitemap, adding its address to your robots.txt file is a quick way to help search engines discover it, alongside submitting it directly through Google Search Console.
Checking your file is working
- Visit yourdomain.com/robots.txt directly in your browser, replacing yourdomain.com with your actual domain.
- Confirm the rules you saved appear exactly as you wrote them.
- Check for typos in folder paths. A single missing slash can mean a rule doesn't match the folder you intended.
Be careful with Disallow: / under User-agent: *. That rule blocks your entire website from every search engine, which is a common cause of a site suddenly disappearing from search results after a redesign or migration.
If you're setting up SEO on a new site, submitting your sitemap is a good next step: How to Submit Your Website's Sitemap to Google Search Console.