Crawl Diagnostics Updates
-
I have several page types on my sites that I have blocked using the robots.txt file (ex: emailafriend.asp, shoppingcart.asp, login.asp), but they are still showing up in crawl diagnostics as issues (ex: duplicate page content, duplicate title tag, etc). Is there a way to filter these issues or perhaps there is something I'm doing wrong resulting in the issues that are showing up?
- Ryan
-
Hi Ryan,
try to move the sitemap to the end and leave a space before it. something like this:
User-agent:*
Disallow: /cgi-bin/
Disallow: /ShoppingCart.asp
Disallow: /SearchResults.asp...
...
Disallow: /mailinglist_subscribe.asp
Disallow: /mailinglist_unsubscribe.asp
Disallow: /EmailaFriend.asp -
I added the pages that it was suggesting to the robots.txt file:
http://www.naturalrugco.com/robots.txt
Most of the pages listed in the high priority errors within moz analytics crawl diagnostics are the emailafriend.asp pages which I've disallowed. Ex: http://www.naturalrugco.com/EmailaFriend.asp?ProductCode=AMB0012-parent
-
Hi Ryan,
At the end of this page you will find several ways to block Roger bot from indexing pages: http://moz.com/help/pro/rogerbot-crawler
I hope it helps,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Updating url name of a key page
One of my key pages has a url name that is completely non descriptive ('About') with no keywords in it that match the important content on that page. Should I create a new page with a proper url name and move the contents there (lose the authority of the current page), or create the url name only and redirect to the existing About page?
On-Page Optimization | | ppopov0 -
To update or not to update news URLs ?
We manage a huge daily news website in my small country - keeping this a bit mysterious in case competitors are reading 🙂 Our URL structure is www.companyname.com/news/categoryofnews/title-of-article?id=articleid In this hyperreactive news world, title of articles change frequently (may be ten times a day for the main stories). The question we debate is : should we reflect the modification of the title in the URL or not ? Example : "Trump says he wants to ban search engines" would have URL http://www.companyname.com/news/entertainment/Trump-says-he-wants-to-ban-search-engines?id=12345678 Later in the day the title becomes "Trump denies he suggested banning search engines". Should the URL be modified to http://www.companyname.com/news/entertainment/Trump-denies-he-suggested-banning-search-engines?id=12345678 (option A) or not (option B) ? In Google News it makes no difference because of the sitemap, but in Google organic things are different. At present (option B in place), Google apparently doesn't see that the article has been updated, and shows the initial timestamp which is visually (and presumably SEOwise) not good : our new news looks like old news. Modifiying the URL would solve that issue, but could, may be, create another one : the new URL, being considered a new article, would lose, the acquired weight of the previous one in terms of referrals, social trafic and so on. Or not ? What do you think is the best option ? Thanks for your expertise, Yves
On-Page Optimization | | yves678901 -
Dealing with updating blog posts
I run a travel and culture blog which means that I write about a lot of upcoming events which recur each year. Usually I title (and slug) the page with the event name and date. When it comes to update the article the next year, sometimes it's as little as changing the date, other times more has changed and it needs to be substantially re-written. Until now, what I've done is update the title, content, and then re-posted (sometimes altering the slug where it's needed to be done). Sometimes it works fine and Google keeps me ranking well, but other times the changes dont get such a great response. I have these options (as far as I can see). Which do you think is best? 1. To create a new article each year and put a message at the start of the previous one to say, click here to read about the 2012 event 2. To continue what I'm doing updating, changing the slug, and re-posting (ie changing the date). 3. To write a new article and insert a 301 redirect. I need to make sure the article appears as a new article in my RSS feed and also on the homepage. Look forward to your ideas! Thanks
On-Page Optimization | | ben10000 -
Crawl Diagnostics not working?
i've been in the crawl diagnostics of my website. I only have 1 page crawled and no errors. Last Crawl Completed: Jul. 12th, 2012 Next Crawl Starts: Jul. 19th, 2012 Do you have an idea of how to fiw it? Thanks a lot
On-Page Optimization | | Ericc220 -
How to force a refresh after on-page optimisation update
After updating areas highlighted in the On-Page Optimization report even after clicking the [Grade My On-page Optimization] the results don't refresh or reflect the changes eg The h1 tag does include the exact search term and there is bolded examples of the keyword phrase but report says not! Is there a way to force an update or is it a time related issue?
On-Page Optimization | | RobWillox0 -
There are companies who evaluate what effect the penguin update had on a website. Is this possible and is it a good investment ?
I have been hit by the penguin update. I have found companies who for $300 will evaluate my site for potential problems. Is this possible and is it worth the investment
On-Page Optimization | | MobileVet0 -
Original content and the Google Panda Update
We are an online furniture store with about 1300 products on the site, and we mostly use the catalogue descriptions for the product. Recently I have been reading about One Way Furniture: http://ecommerceprnews.com/e-commerce_articles/2011/03/one-way-furniture-shifts-toward-quality-content-after-google-panda-update-201928.htm They are a big american online furniture which seemed to have lost about a 3rd of there traffic due to being punished in the panda update. Now it seems they are blaming the fact they use they use catalogue descriptions for the product (like us), and now they are going to rewrite all their product descriptions. We are a small company and rewriting 1300 products (meaningfully) is no small task. Looking at our own traffic we have taken a small slump since feb after about 18 months of general increased month on month traffic ( bar seasonal dips and boost), but we didn't have a "fall of the cliff" like One Way Furniture. But have been expanding into other areas (and there for new keywords), so we had expected to be increasing our traffic. So the question is, how important is unique content for all our products? is it worth all the time and money to fix all the pages? Our plan is to make sure our category pages (and there for landing pages) have unique content, would that be enough on its own, or are the product pages damaging the site over all?
On-Page Optimization | | eunaneunan0 -
Not making a change of the 100's in crawl Diagnostic
Based on the PRO crawl Diagnostics – if we don’t make a change on 1 page, does that just affect the SEO on that one page, or does it affect the SEO on all pages of the site? E.g. If we get a “Too many on page links” for a certain page that we don’t really want to rank for – does not fixing that particlaur page affect the site as a whole? Hope I explained this ok..
On-Page Optimization | | inhouseninja0