When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have lots of articles on my website. Should I add myself as the Author Rank for seo purposes
Hi All, We have lots of articles on our news page of our website. Should we add one of us as the author to gain Author Rank points so to speak ?.. Have not done anything previously with this but just wondered if we were missing a trick here thanks Pete
On-Page Optimization | | PeteC120 -
Googlebot indexing URL's with ? queries in them. Is this Panda duplicate content?
I feel like I'm being damaged by Panda because of duplicate content as I have seen the Googlebot on my site indexing hundreds of URL's with ?fsdgsgs strings after the .html. They were beign generated by an add-on filtering module on my store, which I have since turned off. Googlebot is still indexing them hours later. At a loss what to do. Since Panda, I have lost a couple of dozen #1 rankings that I've held for months on end and had one drop over 100 positions.
On-Page Optimization | | sparrowdog0 -
How to find google indexed pages
I can't find where the # of indexed pages are on my google analytics. I tried the instructions below, but the index status was not an option on my dashboard. View the Index Status page: On the Webmaster Tools home page, click the site you want. On the Dashboard, click Google Index, and then click Index Status.
On-Page Optimization | | SoftwareMarketing0 -
Duplicate content shown in Google webmaster tools for 301 redirected URLs.
Why does Google webmaster tools shows 5 URLs that have been 301 redirected as having duplicate meta descriptions?
On-Page Optimization | | Madlena0 -
Should I rewrite all my URLs ?
Hi all, I'm pretty new here and this is a question I'm struggling with since years ! All my URLs are very long. Years ago I wanted to put as many keywords as possible but today I'm not sure anymore it was such a good idea. Example: http://www.spirit-of-metal.com/album-groupe-Take_Me_To_Janus-nom_album-Ripping_the_Heart_from_the_Chest_of_the_Earth-l-en.html The problem is I have more than 300K of these pages. I'm afraid to create a huge mess even if I 301 them all to the new pages. What's your opinion ? Is it worth the effort ? Many thanks in advance for your precious help !
On-Page Optimization | | kivanSOM0 -
800 number on website
Hi, My client just sent me an 800 number that he would like to use to replace his number on the website. I know that it is best to keep a local phone number on the website and across all citations for NAP/Local SEO reasons. Is there anywhere that I could still incorporate the 800 number and not have it affect SEO? Thanks, Erin
On-Page Optimization | | HiddenPeak0 -
ECommerce URL's
This is based on a clothing retailer, eCommerce site. In an effort to reduce the length of our product names, we are considering removing terms like long-sleeve, short-sleeve, etc., but leaving that information in the URL. Now, the concern is that we would lose some traction in the SERP's if those descriptive words are left out as the product name is also our page title. Then I think keywords as broad as long-sleeve shirt wouldn't serve us well anyways. One idea we have is that the alt tag on the product image could still display the longer product name that would include long-sleeve, etc. thus having the keyword on the product page. Any ideas or suggestions? Hope this is clear. Seems redundant from a user standpoint to state long-sleeve, etc. in every product name. Thanks - your answers are always so helpful!
On-Page Optimization | | kennyrowe0 -
How can I stop google reading a certain section of text with my H1 tag?
Hey Mozzers, I'm wondering if anybody knows of a way that I can stop google reading a certain part of text within my H1 texts? My issue is that I have individual office pages on my site, but many offices are based in the same city; such as 'London'. I want to keep London within the H1 tag for user experience but I do not want it to be picked up by the search engines and start a canonical issue. I've seen some people say to use document.write or use an image. Does anybody know of a correct way of doing this? Many Thanks.
On-Page Optimization | | Lakeside0