Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Report shows Internal links as zero
We have recently updated our website (https://www.payumoney.com/) and created new pages like "https://www.payumoney.com/free-payment-gateway-India.html". we have internally linked all the pages, but when I check the crawl report from MOZ it shows internal links as 0 and these pages have the PA of 1 and not increasing. I am unable to find out the reason why is it happening?
On-Page Optimization | | payu_india0 -
PDF Instructions come up in Crawl report as Duplicate Content
Hello, My ecommerce site has many PDF instruction pages that are being marked as duplicate content in the site crawl. Each page has a different title, and then a PDF displayed in an iframe with a link back to the previous page & to the category that the product is placed in. Should I add text to the pages to help differentiate them? I included a screenshot of the code that is on all the pages. Thanks! Justin 9tD9HMr
On-Page Optimization | | JustinBSLW0 -
Crawl with cach problem
Hello, My Crawl results in Seomoz shows me that i have few thousands of 302 direct problem, bucause it was crawling links like http://www.sposae.com/abito-sposa-g2026-pr-347.html?action=buy_now and it would be redirected automatically to http://www.sposae.com/cookie_usage.php because of cookie not activated from the user. Now I'm wondering if this is an issue to be solved or just a minor thing without impact. Thanks
On-Page Optimization | | angelowei0 -
Content Update
Hello, If I update the existing content i.e.I added some content to the already existing indexed content in a post,how will it effect SEO wise? Venkee
On-Page Optimization | | Venkee0 -
Error is not going away and crawling
I have fixed an error but its still showing in red as error. Im totally new to SeoMoz and to SEO in general so im not sure how this tool works. Did I fix it correctly or not if its still showing? It was a broken link and now it links up to another page. Do I just have to wait? My website only has 8 pages and on the dashboard it says crawled 8 pages but it takes up to a week for a full crawl? Im really confused. Thank you in advanced!
On-Page Optimization | | Pixeltistic0 -
Did anyone Rankings drop massively last weekend ...Is this new google update ?
Hi all, My Site rankings took a battering in the past week , From what I have read, I know that google have updated their algorithm and supposedly it only affects less than 1% of queries but I was very surprised to have fallen in that list... Just wondered, am i the only site to have been hammered this past week ? or is it more of a case that maybe the new algorithm means I need to be more savvy with our SEO. Just posting a quick general consensus question ... thanks Sarah
On-Page Optimization | | SarahCollins0 -
The crawl diagnosis indicated that my domain www.mydomain.com is duplicate with www.mydomain.com/index.php. How can I correct this issue?
How can I fix this issue when crawl diagnosis indicated that my www.mydomain.com is duplicate with www.mydomain.com/index.php? That suppose to be the same page and not duplicate, right?
On-Page Optimization | | jsevilla0 -
Does the seomoz crawler that crawls for the onpage reports have a set ip?
I would like to test my site, but its not launched yet and don;t want anybody to see it. But I can allow myself and other to view the site if I have there ip address. So does the seomoz crawler have a static one or range? James
On-Page Optimization | | BarefootJames0