What is robots.txt

If you want to hide the page completely from Search, use another method. Use a robots. This won't prevent other pages or users from linking to your image, video, or audio file.

Before you create or edit a robots. Depending on your goals and situation, you might want to consider other mechanisms to ensure your URLs are not findable on the web. If you decided that you need one, learn how to create a robots. Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4. For details, see the Google Developers Site Policies. Documentation Not much time?

Beginner SEO Get started. Establish your business details with Google. Advanced SEO Get started. Documentation updates.

Go to Search Console. General guidelines. For example, you might have a staging version of a page. Or a login page.

These pages need to exist. By blocking unimportant pages with robots. Prevent Indexing of Resources: Using meta directives can work just as well as Robots. The bottom line? Establish your business details with Google. Advanced SEO Get started. Documentation updates. Go to Search Console. General guidelines. Content-specific guidelines. Images and video. Best practices for ecommerce in Search. COVID resources and tips. Quality guidelines. Control crawling and indexing. Sitemap extensions. Meta tags.

Crawler management. Google crawlers. Site moves and changes. Site moves. International and multilingual sites. JavaScript content. Change your Search appearance.

Using structured data. Feature guides. Debug with search operators. Web Stories. Early Adopters Program. Optimize your page experience. This means that regular expressions are usually used only with the Disallow directive to exclude files, directories, or websites. This directive would not index all websites containing the string "autos".

With this directive, all content ending with. Similarly, this can be transferred to different file formats: For example,.

With pages excluded by robots. Too much restriction of the user agents can therefore cause disadvantages in the ranking. A notation of directives that is too open can result in pages that contain duplicate content or that affect sensitive areas such as a login.

When creating the robots. The latter also applies to the use of wildcards, which is why a test in the Google Search Console makes sense.

In this case, webmasters should use the Noindex Meta-Tag instead and exclude individual pages from indexing by specifying them in the header. If errors occur here, web sites can become unavailable, because the URLs are not crawled at all and thus cannot appear in the index of the search engines.

The question of which pages are to be indexed and which are not has an indirect impact on the way in which search engines view or even register websites.