Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is the HTML content inside an image slideshow of a website crawled by Google?
I am building a website for a client and i am in a dilemma whether to go for an image slideshow with HTML content on the slides or go for a static full size image on the homepage. My concern is that HTML content on the slideshow may not get crawled by Google and hence may not be SEO friendly.
On-Page Optimization | | aravinn0 -
I want to check which pages have been crawled
I would like to find out which pages have been crawled by seomoz on my site
On-Page Optimization | | seoworx1230 -
What's the best way to handle crawling of photo gallery?
When you have a photo gallery with many search filters and loads and loads of pages, is it best to block all the filters and use google's pagination code? Ex: http://photo.net/gallery/photocritique/filter This site has pages for many different queries. While the page titles are unique, the pages are showing duplicated content.
On-Page Optimization | | cakelady0 -
Dealing with updating blog posts
I run a travel and culture blog which means that I write about a lot of upcoming events which recur each year. Usually I title (and slug) the page with the event name and date. When it comes to update the article the next year, sometimes it's as little as changing the date, other times more has changed and it needs to be substantially re-written. Until now, what I've done is update the title, content, and then re-posted (sometimes altering the slug where it's needed to be done). Sometimes it works fine and Google keeps me ranking well, but other times the changes dont get such a great response. I have these options (as far as I can see). Which do you think is best? 1. To create a new article each year and put a message at the start of the previous one to say, click here to read about the 2012 event 2. To continue what I'm doing updating, changing the slug, and re-posting (ie changing the date). 3. To write a new article and insert a 301 redirect. I need to make sure the article appears as a new article in my RSS feed and also on the homepage. Look forward to your ideas! Thanks
On-Page Optimization | | ben10000 -
There are companies who evaluate what effect the penguin update had on a website. Is this possible and is it a good investment ?
I have been hit by the penguin update. I have found companies who for $300 will evaluate my site for potential problems. Is this possible and is it worth the investment
On-Page Optimization | | MobileVet0 -
SEO Moz crawl has 3 missing page title errors when they are clearly there.
My SEO Moz crawl today has highlighted for errors where page titles are empty missing. For example: http://www.musicliveuk.com/live-acts/hire-wedding-entertainment/wedding-entertainment-kent This page clearly has a title as do the other 3. Is it a bug in the system or am I missing something?
On-Page Optimization | | SamCUK0 -
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them.
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them. http://pro.seomoz.org/campaigns/95663/issues#notice-issues
On-Page Optimization | | cyaindc0 -
Pages crawled
I noticed there is a limited in the number of pages crawled on galena.org? Will this number increase over time?
On-Page Optimization | | nskislak240