Meta robots or robot.txt file?
-
Hi Mozzers!
For parametric URL's would you recommend meta robot or robot.txt file?
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs.And what's the real difference between the two?
Thanks again!
Kay
-
No problem at all
-Andy
-
Thanks Andy!!!
-
Hi Kay,
If you want to disallow access to a page, then add the following to the Robots.txt file:
Disallow: /quickView
Then test this in Webmaster Tools.
If you want to tell Google not to index a page, then you need to do this at the page level using Meta Robots. However, don't do both (at least not at the same time). If you disallow access to a set of pages via Robots.txt and then at a later stage you Meta Noindex, Google won't see this because of the Disallow in the Robots.txt.
It really depends what you are trying to achieve, but it sounds like the Meta Robots is the way to go for you.
-Edit... here is an interesting read for you.
-Andy
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do about new meta description character limit?
Hey Everyone, So as I'm sure everyone has heard the new meta description limits have been increased and I have been going back and forth with whether or not to update my descriptions for my client's websites. I know that Google is now dynamically generating descriptions based off of content, but is it still beneficial to write longer descriptions as well? Will Google display my longer description now if it is ranking well? Rand Fishkin at Moz and others say that you should and other people, including Danny Sullivan, have said you shouldn't worry about it and leave them at between 150-160. My questions is what should I do? I will be focusing heavily on making sure the site's content is very targeted and relevant for when Google dynamically generates the descriptions, but should I still edit my descriptions manually?
Intermediate & Advanced SEO | | kenturley2 -
Titles and Metas disappeared?? Help please!
Hi everyone, Had a bit of a concern today, my weekly report has come through and my crawl issues have skyrockets by over 400! It says my metas and titles are missing but when I check through the site manually they seem to all still be there, I'm getting the same problem when I use screaming frog to crawl the site. I would really appreciate an explanation from someone as to why this is happening as I am quite confused about the situation. Thank you people Charlie Our website is www.homelogic.co.uk 🙂
Intermediate & Advanced SEO | | MintySEO0 -
Handling Multiple Domain 301 Redirects on Single htaccess file
Hello, I have a client that currently that has 9 different markets with different sub-domains on one server (aka one htaccess file.). All the sites have very similar Navigation and some of them contain the same products aka same URLs. The site is using Magento CMS and I'm trying to figure out how to redirect some products that have been removed from one of the stores. The problem I'm running into is when I try to redirect one store url, it redirects all the site's URLs. Example http://store.domain1.com/ http://store.domain2.com/ I'd like to redirect http://store.domain1.com/old-url.html to http://store.domain1.com/new-url.html without making http://store.domain2.com/old-url.html redirect. I've literally been pulling out my hair trying to figure this one out but have had no luck. Does anybody have any ideas on how I could do this without having the sites redirect or create any loops? Any wisdom from you apache experts would be greatly appreciated. Thanks, Erik
Intermediate & Advanced SEO | | Erik-M0 -
When you add 10.000 pages that have no real intention to rank in the SERP, should you: "follow,noindex" or disallow the whole directory through robots? What is your opinion?
I just want a second opinion 🙂 The customer don't want to loose any internal linkvalue by vaporizing link value though a big amount of internal links. What would you do?
Intermediate & Advanced SEO | | Zanox0 -
How does the use of Dynamic meta tags effect SEO?
I'm evaluating a new client site which was built buy another design firm. My question is they are dynamically creating meta tags and I'm concerned that it is hurting their SEO. When I view the page source this is what I see. <meta name="<a class="attribute-value">keywords</a>" id="<a class="attribute-value">keywordsGoHere</a>" content="" /> <meta name="<a class="attribute-value">description</a>" id="<a class="attribute-value">descriptionGoesHere</a>" content="" /> <title id="<a class="attribute-value">titleGoesHere</a>">title> To me it looks like the tags are not being added to the page, however the title is showing when you view it in a browser and if use a spider view tool, it sees the title. I'm guess it is being called from a DB. So I'm a little concerned though that the search engines are not really seeing the title and description. I'm not worried about the keywords tag. Can anyone shed some light on how this might work? Why it might not being showing the text for the description in the page code and if that will hurt SEO? Thanks for the help!
Intermediate & Advanced SEO | | BbeS0 -
Robots.txt
What would be a perfect robots.txt file my site is propdental.es Can i just place: User-agent: * Or should i write something more???
Intermediate & Advanced SEO | | maestrosonrisas0 -
Robots.txt file - How to block thosands of pages when you don't have a folder path
Hello.
Intermediate & Advanced SEO | | Unity
Just wondering if anyone has come across this and can tell me if it worked or not. Goal:
To block review pages Challenge:
The URLs aren't constructed using folders, they look like this:
www.website.com/default.aspx?z=review&PG1234
www.website.com/default.aspx?z=review&PG1235
www.website.com/default.aspx?z=review&PG1236 So the first part of the URL is the same (i.e. /default.aspx?z=review) and the unique part comes immediately after - so not as a folder. Looking at Google recommendations they show examples for ways to block 'folder directories' and 'individual pages' only. Question:
If I add the following to the Robots.txt file will it block all review pages? User-agent: *
Disallow: /default.aspx?z=review Much thanks,
Davinia0 -
Soft 404's from pages blocked by robots.txt -- cause for concern?
We're seeing soft 404 errors appear in our google webmaster tools section on pages that are blocked by robots.txt (our search result pages). Should we be concerned? Is there anything we can do about this?
Intermediate & Advanced SEO | | nicole.healthline4