Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How often is your domain authority updated?
I can't seem to figure out how often our domain authority is updated - it seems random, do you know typically when this happens? Thanks!
On-Page Optimization | | regineraab0 -
Site update and what to do about current keyword rank
Hi im in the process of giving my site a major update as its only one page and not responsive . The the new site will have a homepage with a list of my services with a small description of each and each service will link to its own page. My one page site at the moment ranks number one in my area for a low competition keyword (moz keyword score of 13%) of course this sends me very little traffic. With my new site im also going for more keywords but these will be Moz KWD 30% and 40%. I know this will be quite hard and take some time but i'm pretty sure I know what I need to do to get there. Now my question is what do i do with my current home page (only page) that ranks for that low comp keyword?
On-Page Optimization | | juun
I dont want to lose my rank for that, so do i make a new page on my new site that is optimized for that keyword, but then I don't want to 301 my homepage to that so I guess as its such an easy word to rank for my new page would soon rank for that? What are your thoughts and advice please, thanks in advance.1 -
Is the HTML content inside an image slideshow of a website crawled by Google?
I am building a website for a client and i am in a dilemma whether to go for an image slideshow with HTML content on the slides or go for a static full size image on the homepage. My concern is that HTML content on the slideshow may not get crawled by Google and hence may not be SEO friendly.
On-Page Optimization | | aravinn0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
On Page Optimization not updating after SSL change
Last week we changed our URLs to be only HTTPS (they were a mixture of both for a couple of weeks). We did this by adding a 301 redirect from HTTP to HTTPS. However, the On Page Optimization is now showing the same URL/Keyword twice for a few of our pages. One has a grade A, one has a grade B. The grade B complains about "Rel Canonical" not matching, but the other doesn't. This is pretty confusing - should it not have crawled the pages and noticed the 301 redirects and combined the entries or removed the HTTP ones?
On-Page Optimization | | RitchieF19800 -
Number of pages crawled in dropping from 4 to 2
The report on our campaign shows that up to 2 pages are being crawled now from 4. However, our site has more pages than this. We recently inserted code to allow crawlers. what can we do to resolve this? please assist.v
On-Page Optimization | | seoworx1230 -
Website Changes Updated in Google...
How long does it take for Google to recognize SEO changes that I have made to my website?
On-Page Optimization | | AubbiefromAubenRealty0 -
Does anyone know an "updated" all in one cheat cheat for on page SEO
howdy guys, after panda, and penguin, i wanted to know if anyone has any good references for a cheat sheet that has ALL the on page SEO factors like title tag, image tags, H1-H6, etc, the basics, but since the zoo got loose I dont know if things have changed for on page SEO. For example what is the optimal KW density we should be using etc. i found this list : http://www.vaughns-1-pagers.com/internet/google-ranking-factors.htm but it seems to be a bit outdated, if someone can share something similar to this list but something more recent, like a few weeks old, i think that would help the community out a great bunch! thanks guys
On-Page Optimization | | david3050