Meta robots or robot.txt file?
-
Hi Mozzers!
For parametric URL's would you recommend meta robot or robot.txt file?
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs.And what's the real difference between the two?
Thanks again!
Kay
-
No problem at all
-Andy
-
Thanks Andy!!!
-
Hi Kay,
If you want to disallow access to a page, then add the following to the Robots.txt file:
Disallow: /quickView
Then test this in Webmaster Tools.
If you want to tell Google not to index a page, then you need to do this at the page level using Meta Robots. However, don't do both (at least not at the same time). If you disallow access to a set of pages via Robots.txt and then at a later stage you Meta Noindex, Google won't see this because of the Disallow in the Robots.txt.
It really depends what you are trying to achieve, but it sounds like the Meta Robots is the way to go for you.
-Edit... here is an interesting read for you.
-Andy
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta description duplication
Hello, What does google mean by don't duplicate your meta description. For example if I my meta says : Stunning golf holidays in Florida , call xxxx and book today. and I have another page with golf holiday but in ireland this time. If I write Stunning golf holidays in Ireland , call xxxx and book today. Is it considered duplicate ?
Intermediate & Advanced SEO | | seoanalytics0 -
Search Results Pages Blocked in Robots.txt?
Hi I am reviewing our robots.txt file. I wondered if search results pages should be blocked from crawling? We currently have this in the file /searchterm* Is it a good thing for SEO?
Intermediate & Advanced SEO | | BeckyKey0 -
URLs with parameters + canonicals + meta robots
Hi Moz community! I'm posting a new question here as I couldn't find specific answer to the case I'm facing. Along with canonical tags, we are implementing meta robots on our pages (e-commerce website with thousands of pages). Most of the cases have been covered but I still have one unanswered case: our products are linked from list pages (mostly categories) but they almost always include a tracking parameter (ie /my-product.html?ref=xxx) products urls are secured with a canonical tag (referring only to the clean url /my-product.html) but what would be the best solution regarding the meta robots? For now we opted for a meta robot 'noindex, follow' for non canonical urls (so the ones unfortunately linked from our category/list pages), but I'm afraid that it could hurt our SEO (apparently no juice is given from URLs with a noindex robots), and even maybe prevent bots from crawling our website properly ... Would it be best to have no meta robots at all on these product urls with parameters? (we obviously can't have 'index, follow' when the canonical ref points to another url!). Thanks for your help!
Intermediate & Advanced SEO | | JessicaZylberberg0 -
Robots.txt Blocking - Best Practices
Hi All, We have a web provider who's not willing to remove the wildcard line of code blocking all agents from crawling our client's site (user-agent: *, Disallow: /). They have other lines allowing certain bots to crawl the site but we're wondering if they're missing out on organic traffic by having this main blocking line. It's also a pain because we're unable to set up Moz Pro, potentially because of this first line. We've researched and haven't found a ton of best practices regarding blocking all bots, then allowing certain ones. What do you think is a best practice for these files? Thanks! User-agent: * Disallow: / User-agent: Googlebot Disallow: Crawl-delay: 5 User-agent: Yahoo-slurp Disallow: User-agent: bingbot Disallow: User-agent: rogerbot Disallow: User-agent: * Crawl-delay: 5 Disallow: /new_vehicle_detail.asp Disallow: /new_vehicle_compare.asp Disallow: /news_article.asp Disallow: /new_model_detail_print.asp Disallow: /used_bikes/ Disallow: /default.asp?page=xCompareModels Disallow: /fiche_section_detail.asp
Intermediate & Advanced SEO | | ReunionMarketing0 -
What is better for Meta description ??
Hi everybody, I noticed that a lot of websites prefer their meta description would be the first words of the content inside.
Intermediate & Advanced SEO | | roeesa
I on the other hand thought that google will prefer the meta description to be like a peek to what going to be inside.
anyone can explain me, what is better? Thanks 🙂0 -
Duplicate meta descriptions
Hi All Does having quite a few Duplicate meta descriptions hurt SEO. I am worried that I have too many and thinking this could be the reason for my recent drop in search visibility. Thanks in Advance. Andy
Intermediate & Advanced SEO | | Andy-Halliday0 -
Does having a file type on the end of a url affect rankings (example www.fourcolormagnets.com/business-cards.php VS www.fourcolormagnets.com/business-cards)????
Does having a file type on the end of a url affect rankings (example www.fourcolormagnets.com/business-cards.php VS www.fourcolormagnets.com/business-cards)????
Intermediate & Advanced SEO | | JHSpecialty0 -
Meta Description In Blog Feed
The SEOmoz crawl tool is giving me a lot of crawl errors because my blog feed and my blog tags do not have meta descriptions. Can you even give this type of content meta descriptions? If so how can you do it, as this content is created dynamically by Wordpress?
Intermediate & Advanced SEO | | MyNet0