Meta robots or robot.txt file?
-
Hi Mozzers!
For parametric URL's would you recommend meta robot or robot.txt file?
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs.And what's the real difference between the two?
Thanks again!
Kay
-
No problem at all
-Andy
-
Thanks Andy!!!
-
Hi Kay,
If you want to disallow access to a page, then add the following to the Robots.txt file:
Disallow: /quickView
Then test this in Webmaster Tools.
If you want to tell Google not to index a page, then you need to do this at the page level using Meta Robots. However, don't do both (at least not at the same time). If you disallow access to a set of pages via Robots.txt and then at a later stage you Meta Noindex, Google won't see this because of the Disallow in the Robots.txt.
It really depends what you are trying to achieve, but it sounds like the Meta Robots is the way to go for you.
-Edit... here is an interesting read for you.
-Andy
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt & Disallow: /*? Question!
Hi, I have a site where they have: Disallow: /*? Problem is we need the following indexed: ?utm_source=google_shopping What would the best solution be? I have read: User-agent: *
Intermediate & Advanced SEO | | vetofunk
Allow: ?utm_source=google_shopping
Disallow: /*? Any ideas?0 -
Meta Description
Hello, For example let say I do hiking tour in different regions and all my pages are presented the same way with the highlights, hotels, what is included, the price, the level and the dates. I guess that across my pages the meta description is going to be the same, the only thing that is going to change is the destination. Is it ok to do it this way ? I know it isn't recommend to do duplicate but in this type of configuration I have no idea on have to do different meta knowing all the pages present the same things. Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Does Google ignore duplicate meta descriptions?
Hi there SEO mozzers, I am dealing with a website that has duplicate meta descriptions (we know is bad).As a punishment, Google totally ignores the meta descriptions and picks content from the website and displays it in SERP. I already read the https://moz.com/blog/why-wont-google-use-my-meta-description but I was wondering if there is more information/knowledge out there. Any tips are appreciated!
Intermediate & Advanced SEO | | Europarl_SEO_Team0 -
SERP display switching between normal meta description and 15+ items
The site, www.myrtlebeach.com has been having an issue with the way it displays in search results for the keyword "Myrtle Beach hotels". It is showing as a bulleted/itemized list similar to what's mentioned in this Moz article I'll begin with a little background. When I started working with the site it would display in SERPs as: Myrtle Beach Hotels - Reviews, Deals, & Photos - MyrtleBeach.com
Intermediate & Advanced SEO | | Fuel
www.myrtlebeach.com/hotels/
70+ items - Compare Lowest rates & see reviews on Myrtle Beach Hotels
from $76. Holiday Inn Club Vacations Myrtle Beach - South Beach offers a ...
from $27. Located among the south end Myrtle Beach hotels, Holiday Sands ... We did not want the site showing as an itemized list with 70+ items. We wanted it to show with the meta description we provided.
Our first attempt at getting the SERP to display our normal meta description was to simply change the meta description. That didn't work. Our second thought was to use pagination to reduce the number of items on the page. A few days after we implemented pagination we saw our normal meta description displaying in the SERP. Shortly after that we saw the SERP had reverted, but this time was showing 15+ items rather than 70+. This is when began seeing the SERP display change between a normal meta description and 15+ items. In another effort to stop the 15+ items from displaying in SERPs, we added relevant blog content like "Top 10 Oceanfront hotels" and "Best Kid Friendly Hotels" to break up the hotel listings on http://www.myrtlebeach.com/hotels/. Again, our normal meta description displayed in the SERPs for the next few days, but shortly after reverted back to 15+ items. Since then we have been seeing the SERP switch between our normal meta description and 15+ items with no rhyme or reason. Because our listings are not using , I'm not really sure why the site would be displaying this way. Since I have been regularly monitoring the SERP for the keyword "Myrtle Beach hotels", myrtlebeach.com/hotels/ has ranked as high as 5 and as low as 10. I open an Incognito Window and I take screenshots almost daily. I then record how the site was displayed in the SERP and its rank. I also look at organic visits and a Value Per Entry metric I've created. (I looked at Value Per Entry to determine if someone seeing 15+ items was more apt to convert) Average Visits on days with a normal meta description - 182
Average Visits on days with itemized 15+ description - 174 Average Value Per Entry on days with a normal meta description - 131
Average Value Per Entry on days with itemized 15+ description - 120 Average Rank on days with a normal meta description - 7
Average Rank on days with itemized 15+ description - 6 This data shows that performance is better on days when the normal meta description is displayed regardless of rank. I have linked 2 screenshots for reference. The 2-6-14 screenshot shows the SERP display with 15+ items and the 2-7-14 screenshot shows the SERP display with the normal meta description we provided. My questions are:
1. How does Google determine if a site should display in SERPs with an itemized or bulleted list?
2. Is there something else I need to change on myrtlebeach.com/hotels/ to prevent it from displaying in SERPs with 15+ items? m4znToY PRdDXZf0 -
Robots.txt, does it need preceding directory structure?
Do you need the entire preceding path in robots.txt for it to match? e.g: I know if i add Disallow: /fish to robots.txt it will block /fish
Intermediate & Advanced SEO | | Milian
/fish.html
/fish/salmon.html
/fishheads
/fishheads/yummy.html
/fish.php?id=anything But would it block?: en/fish
en/fish.html
en/fish/salmon.html
en/fishheads
en/fishheads/yummy.html
**en/fish.php?id=anything (taken from Robots.txt Specifications)** I'm hoping it actually wont match, that way writing this particular robots.txt will be much easier! As basically I'm wanting to block many URL that have BTS- in such as: http://www.example.com/BTS-something
http://www.example.com/BTS-somethingelse
http://www.example.com/BTS-thingybob But have other pages that I do not want blocked, in subfolders that also have BTS- in, such as: http://www.example.com/somesubfolder/BTS-thingy
http://www.example.com/anothersubfolder/BTS-otherthingy Thanks for listening0 -
Should comments and feeds be disallowed in robots.txt?
Hi My robots file is currently set up as listed below. From an SEO point of view is it good to disallow feeds, rss and comments? I feel allowing comments would be a good thing because it's new content that may rank in the search engines as the comments left on my blog often refer to questions or companies folks are searching for more information on. And the comments are added regularly. What's your take? I'm also concerned about the /page being blocked. Not sure how that benefits my blog from an SEO point of view as well. Look forward to your feedback. Thanks. Eddy User-agent: Googlebot Crawl-delay: 10 Allow: /* User-agent: * Crawl-delay: 10 Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ # Allow Everything Allow: /*
Intermediate & Advanced SEO | | workathomecareers0 -
Canonical Meta Tag Best Practices
I've noticed that some website owners use canonical tags even when there may be no duplicate issues.For examplewww.examplesite.com has a canonical tag.......rel="canonical" href="http://www.examplesite.com/" />www.examplesite.com/bluewidget has a canonical tag.......rel="canonical" href="http://www.examplesite.com/bluewidget/" />Is this recommended or helpful to do this?
Intermediate & Advanced SEO | | webestate0 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0