Does using robots.txt to block pages decrease search traffic?
-
I know you can use robots.txt to tell search engines not to spend their resources crawling certain pages.
So, if you have a section of your website that is good content, but is never updated, and you want the search engines to index new content faster, would it work to block the good, un-changed content with robots.txt? Would this content loose any search traffic if it were blocked by robots.txt? Does anyone have any available case studies?
-
If you block the pages from being crawled, you are also telling the search engines to not index the pages (they don't want to include something they haven't looked at). So yes, the traffic numbers from organic search will change if you block the pages in robots.txt.
-
Agreed, that is a better solution, but, I am still wondering if you block something with robots.txt, will that lead to a decrease in traffic? What if we have some duplicate content that is highly trafficked, if we block it with robots.txt, will the traffic numbers change?
-
You certainly don't want to block this content!
One thing I'd consider is the if-modified-since header, or other headers. Here are two articles that explain more about the concept of using headers to tell the search engines " this hasn't changed, don't bother crawling it". I haven't personally used this, but have read about it in many places.
http://www.feedthebot.com/ifmodified.html
http://searchengineland.com/how-to-improve-crawl-efficiency-with-cache-control-headers-88824
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it best to 301 redirect or use canonical Url when consolidating two pages?
I have build several pages (A and B) with high quantity content. Page A is aged and gets lots of organic traffic, ranks for lots of valuable keywords, and has only internal links to this page. Page B is newer (6 months) and gets little traffic, ranks for no keywords, but has terrific content and many high value external links. As Page A and B are related to a similar theme, I was going to merge content from page B onto page A, but don't know which would be the best approach for handling the links going to page B. For the purposes of keep as much link equity as possible, is it best to us a 301 redirect from B to A or use a canonical URL from B to A?
Intermediate & Advanced SEO | | Cutopia0 -
Google is indexing wrong page for search terms not on that page
I’m having a problem … the wrong page is indexing with Google, for search phrases “not on that page”. Explained … On a website I developed, I have four products. For example sake, we’ll say these four products are: Sneakers (search phrase: sneakers) Boots (search phrase: boots) Sandals (search phrase: sandals) High heels (search phrase: high heels) Error: What is going “wrong” is … When the search phrase “high heels” is indexed by Google, my “Sneakers” page is being indexed instead (and ranking very well, like #2). The page that SHOULD be indexing, is the “High heels” page (not the sneakers page – this is the wrong search phrase, and it’s not even on that product page – not in URL, not in H1 tags, not in title, not in page text – nowhere, except for in the top navigation link). Clue #1 … this same error is ALSO happening for my other search phrases, in exactly the same manner. i.e. … the search phrase “sandals” is ALSO resulting in my “Sneakers” page being indexed, by Google. Clue #2 … this error is NOT happening with Bing (the proper pages are correctly indexing with the proper search phrases, in Bing). Note 1: MOZ has given all my product pages an “A” ranking, for optimization. Note 2: This is a WordPress website. Note 3: I had recently migrated (3 months ago) most of this new website’s page content (but not the “Sneakers” page – this page is new) from an old, existing website (not mine), which had been indexing OK for these search phrases. Note 4: 301 redirects were used, for all of the OLD website pages, to the new website. I have tried everything I can think of to fix this, over a period of more than 30 days. Nothing has worked. I think the “clues” (it indexes properly in Bing) are useful, but I need help. Thoughts?
Intermediate & Advanced SEO | | MG_Lomb_SEO0 -
Https Loss of Search traffic
Hey guys, We moved our site to from http to https. We subsequently lost 25% in our search traffic in 1 Month. We changed a few other pieces such as images, added new content etc. Has anyone got any suggestions on how we start to understand what happened? Thanks in advance.
Intermediate & Advanced SEO | | Johnny_AppleSeed0 -
Interlinking from unique content page to limited content page
I have a page (page 1) with a lot of unique content which may rank for "Example for sale". On this page I Interlink to a page (page 2) with very limited unique content, but a page I believe is better for the user with anchor "See all Example for sale". In other words, the 1st page is more like a guide with items for sale mixed, whereas the 2nd page is purely a "for sale" page with almost no unique content, but very engaging for users. Questions: Is it risky that I interlink with "Example for sale" to a page with limited unique content, as I risk not being able to rank for either of these 2 pages Would it make sense to "no index, follow" page 2 as there is limited unique content, and is actually a page that exist across the web on other websites in different formats (it is real estate MLS listings), but I can still keep the "Example for sale" link leading to page 2 without risking losing ranking of page 1 for "Example for sale"keyword phrase I am basically trying to work out best solution to rank for "Keyword for sale" and dilemma is page 2 is best for users, but is not a very unique page and page 2 is very unique and OK for users but mixed up writing, pictures and more with properties for sale.
Intermediate & Advanced SEO | | khi50 -
Where is the best place for Landing Pages to reside on the Home Page?
On this site http://www.austintenantadvisors.com/ I have my main landing pages listed in the navigation under "Types". The reason why I did this is because I am not sure where to insert those on the home page where it does not look spammy to Google and looks natural for users. Obviously they need to appear somewhere on the home page for Google to be able to continue crawling and indexing them. Any thoughts or suggestions would be appreciated.
Intermediate & Advanced SEO | | webestate0 -
Pricing Page vs. No Pricing Page
There are many SEO sites out there that have an SEO Pricing page, IMO this is BS. A SEO company cannot give every person the same quote for diffirent keywords. However, this is something we are currently debating. I don't want a pricing page, because it's a page full of lies. My coworker thinks it is a good idea, and that users look for a pricing page. Suggestions? If I had to build one (which I am debating against) is it better to just explain why pricing can be tricky? or to BS them like most sites do?
Intermediate & Advanced SEO | | SEODinosaur0 -
What is the best canonical url to use for a product page?
I just helped a client redesign and launch a new website for their organic skin care company (www.hylunia.com). The site is built in Magento which by default creates MANY urls for each product. Which of these two do you think would be the best to use as the canonical version? http://www.hylunia.com/pure-hyaluronic-acid-solution
Intermediate & Advanced SEO | | danielmoss
or http://www.hylunia.com/products/face-care/facial-moisturizers/pure-hyaluronic-acid-solution ? I'm leaning on the latter, because it makes sense to me to have the breadcrumbs match the url string, and also it seems having more keywords in the url would help. However, it's obviously a very long url, and there might be some benefits to using the shorter version that I'm not aware of. Thanks in advance for sharing your thoughts. Best, Daniel0 -
Would the use of
Hi, I am wondering on you through relevant to SEO in the following situation. I have a "travel" website and obvisouls as part of that I have a whole list of desitinations. So I have a drop down in my page navigation, which lists all my desitinations. At the moment I see have 2 main options to display the lists as follows: 1/. Perfect Anchors, but not good for usability - IE repeating the word "holiday in a list of 100 destinations, looks spammy for one, and when the headline says "Holiday Destinations", then from a use perspective its pretty pointless and takes away from navigation rather than improves it".
Intermediate & Advanced SEO | | James77
New York Holidays
Las Vegas Holidays 2/. Non Perfect Anchors - But better for usability
New York
Las Vegas So I am thinking - would the use of the title attribute provide a perfect solution?? Or am I wasting my time with this and it is just pointless considering it as an option. EG - what I had in mind was:
3/. Ideal Solution for both SEO and usability??
New York
Las Vegas Thanks for you help in advance.0