Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should We Wait To Launch a Redesigned Site After Google's Core Web Vitals & Page Experience Algo Update
We are redesigning our WordPress site (over 1300 posts and pages) and are on schedule to launch in May of 2021. Should we wait for after Google's Core Web Vitals & Page Experience algorithm update?
On-Page Optimization | | opiates0 -
We recently updated a large guide that takes the place of the original. The original has some nice organic traffic to it and I don't want to risk losing it. Should I 301 redirect to the new version, or update all the info directly on the original page?
We don't have a lot of content that garners much non-branded organic, so this is something I don't want to risk losing. We do not have a whole lot of external links into the page either.
On-Page Optimization | | AFP_Digital1 -
Crawl errors
Hi I have the following errors on my site and was wondering would it help improve my ranking to fix : Missing Meta Description Tag 137Duplicate Page Title 17Title Element is Too Long 6Temporary Redirect 3
On-Page Optimization | | WallerD0 -
How much content does Google Crawl on your site?
Hi, We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us? Thanks,
On-Page Optimization | | mdorville
Matt0 -
What's the best way to handle crawling of photo gallery?
When you have a photo gallery with many search filters and loads and loads of pages, is it best to block all the filters and use google's pagination code? Ex: http://photo.net/gallery/photocritique/filter This site has pages for many different queries. While the page titles are unique, the pages are showing duplicated content.
On-Page Optimization | | cakelady0 -
Pages crawled dropped like a stone
set up a new campaign seomoz crawled the site and did about 1500 pages the last crawl it now only lists 1 page I can see anyway to see why, the site has not changed so why has the number of pages dropped?
On-Page Optimization | | spiralsites0 -
Does anyone know an "updated" all in one cheat cheat for on page SEO
howdy guys, after panda, and penguin, i wanted to know if anyone has any good references for a cheat sheet that has ALL the on page SEO factors like title tag, image tags, H1-H6, etc, the basics, but since the zoo got loose I dont know if things have changed for on page SEO. For example what is the optimal KW density we should be using etc. i found this list : http://www.vaughns-1-pagers.com/internet/google-ranking-factors.htm but it seems to be a bit outdated, if someone can share something similar to this list but something more recent, like a few weeks old, i think that would help the community out a great bunch! thanks guys
On-Page Optimization | | david3050 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0