Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can the Lightboxes on My Site be Crawled?
I'm trying to optimize my site, but I have lightboxes and I don't know if they are visible to the search engines. If they aren't, could you suggest something that I could do? THANK YOU so much!!!!! My site is lymphexpo.com
On-Page Optimization | | bosleypalmer0 -
Single Page on my client's website is not crawling and indexing new changes. What could be the possible reason?
I made several changes on client's website on different pages, changed titles, add content on few pages, moved blog from subdomain to sub directory. Everything is crawled but there is one page on the website (not part of the blog) that isn't getting crawled in Google and picking up changes. The last crawl of the website is 2 days back whereas that page was last crawled on 30th sep. I just wanted to know the possible reasons and has anyone encountered this before?
On-Page Optimization | | MoosaHemani0 -
Disavowed links, updated website etc - still no ranking improvements
Hi, Could anyone take a look at www.artificialgrass4u.co.uk - a few years ago it used to rank highly for 'artificial grass' ... then when Google rolled out its algorithms punishing websites with poor links it lost all it's rankings. We've disavowed almost all of the bad links, and have been adding new optimised content etc over the past few months but rankings still haven't improved. Is there anything I'm missing? Thanks
On-Page Optimization | | icansee0 -
Does this popup get crawled?
Hi, We have a popup on our site that shows examples of different trash cans that can be used for the bags we sell. Here's an example of a page that has the "common cans popup" http://www.plasticplace.com/3-gallon-high-density-6-mic-17x18-clear-trash-bags (This is a screenshot of where to click: http://screencast.com/t/8AvoktAcXtM) Is this content crawled by google? This is how it looks in our source code: view-source:http://www.plasticplace.com/gallon-size/20-30-gallon-trash-bags (start at line 913) Thanks!
On-Page Optimization | | EcomLkwd0 -
How do I get rid of crawl errors?
I recently revamped all my category pages to make them Google friendly. I did a lot of keyword research and ended up deleting categories and renaming categories. Now Google Webmasters is showing a ton of 404 errors. How can I get rid of them? Do I have to remove the URL one by one with Google Webmasters? Is it something that won't hurt my rankings and will go away in time?
On-Page Optimization | | dealblogger0 -
How much content does Google Crawl on your site?
Hi, We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us? Thanks,
On-Page Optimization | | mdorville
Matt0 -
Reducing number crawl-able links?
Hello, I just like to ask for best practice when it comes to reduce number of internal links on a site with a mega menu. Since the mega menu lists all categories and all their subcategories it creates a problem when all categories are linking to all categories directly.. Would the method below reduce the number of links and preventing the link juice flowing directly from category to category? [(link built with JavaScript and the html5 "data-" attribute) Thinking of using these links to categories in the menu not directly below the parent category.](#)
On-Page Optimization | | AJPro0 -
How long after a URL starts showing a 404 does Google stop crawling?
Before hiring me to do SEO, a client re-launched their site and did not 301 the old URLs to the new. Only the home page URL stayed the same. For a month after the re-launch, the old URLs returned a 404. For the next month, all 404 pages (basically any non-existent URL) were 301'd to the home page. Finally, 2 months after launching, they properly 301'd the old URLs to the new. Now, the new URLs are not ranking well. I assume it's too late to realize any benefit from the 301's, just checking to see if anybody has any insight into how long Google keeps trying to crawl old/404/improperly 301'd URLs. Thanks!
On-Page Optimization | | AndrewMiller0