Crawl Diagnostics - unexpected results
-
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site.
It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server.
It appears that the Crawler doesn't think that it's running on the live site.
For example
http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html
will go to a Product Not Found page, and therefore Duplicate Content errors are produced.
Running
http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html
produces the correct product page and not a Product Not Found page
Any thoughts?
-
Hi Nordichouse,
Sorry it took awhile for me to get back to you on this.
I agree with the SEOmoz techs, it doesn't matter if it is a crawler or a actual person, if you go to an invalid url you should be redirected as 301 to the actual page. If the product doesn't exist it should not allow for superfluous urls.
So basically what you should have is if the product exist then the site redirects to the correct URL. If it doesn't exist then send any query for that product to the same page and display the oscommerce product not found message. By doing this you prevent the system from creating upteenthousand urls for each product.
If you want to test what I mean you can visit our store a www.rubberstore.com/catalog and try a few urls like:
catalog/nipple-clips-p-1000.html
we don't have a product with the id of 1000 so you'll get redirected to the not found message and the root page
-p-1000.htmlhowever if you try:
catalog/a-fake-url-p-29.html
you'll get redirected to our actual product page matching this product id.Hope that makes since. All this is done with the .htaccess url re-writter I posted above.
-
Don
Yes, that is how it is done and there is no problem with that. The above is just how inbound URLs get processed.
The issue here is how the crawler works. The only possible way for this particular URL to be generated is for a certain parameter to be appended to the URL - and that would be unusual (unless SEOmoz techies tell me different)
Alan
-
Did you ever have a product with the id of 1140? If you look at your products table just check the auto number in the product_id column..
If you did and it was live at some point it could be finding the old product based on the old url it used to have.
If you never made that product live then I don't know how a crawler could of found a product that doesn't exist unless they starting using some technology that I'm unaware of.
Since you said you use OSC this what we use to deal with the problem I outlined above..
Begin Ultimate SEO V2.2d
Options +FollowSymLinks
RewriteEngine On# RewriteBase instructions
# Change RewriteBase dependent on how your shop is accessed as below.
# http://www.mysite.com = RewriteBase /
# http://www.mysite.com/catalog/ = RewriteBase /catalog/
# http://www.mysite.com/catalog/shop/ = RewriteBase /catalog/shop/# Change the following line using the instructions above
RewriteBase /catalog/RewriteRule ^(.)-p-(.).html$ product_info.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-c-(.).html$ index.php?cPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-m-(.).html$ index.php?manufacturers_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-pi-(.).html$ popup_image.php?pID=$2&%{QUERY_STRING}
RewriteRule ^(.)-by-(.).html$ all-products.php?fl=$2&%{QUERY_STRING}
RewriteRule ^(.)-t-(.).html$ articles.php?tPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-a-(.).html$ article_info.php?articles_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-au-(.).html$ articles.php?authors_id=$2&%{QUERY_STRING}
#RewriteRule ^(.)-pr-(.).html$ product_reviews.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-pri-(.).html$ product_reviews_info.php?products_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-f-(.).html$ faqdesk_info.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-fc-(.).html$ faqdesk_index.php?faqPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-fri-(.).html$ faqdesk_reviews_info.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-fra-(.).html$ faqdesk_reviews_article.php?faqdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-i-(.).html$ information.php?info_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-links-(.).html$ links.php?lPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-pm-([0-9]+).html$ info_pages.php?pages_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-n-(.).html$ newsdesk_info.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-nc-(.).html$ newsdesk_index.php?newsPath=$2&%{QUERY_STRING}
RewriteRule ^(.)-nri-(.).html$ newsdesk_reviews_info.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-nra-(.).html$ newsdesk_reviews_article.php?newsdesk_id=$2&%{QUERY_STRING}
RewriteRule ^(.)-po-([0-9]+).html$ pollbooth.php?pollid=$2&%{QUERY_STRING}End Ultimate SEO V2.2d
You may try it to see if it helps fix your issue.
-
Thanks, Don
You are right in your analysis - it is osC, but highly modified by myself. Yes, it does redirect.
That, however, is not the point. On the live site, the URL containing 1140 (for example) is never generated.
The mystery is how the Crawler can find something that isn't there! Magic.
Alan
-
Hi nordichouse,
You may want to check with your CMS provider. The urls are similar to Oscommerce which I'm experienced with, but I can see that isn't an Oscommerce setup. The system should have some sort of URL re-writer to deal with this problem.
The issue that I see is the system actually doesn't care what you type in between .co.uk/ and -p-1140.html
For example try this url to get a valid product..
http://www.nordichouse.co.uk/nipple-clips-p-1000.html
which is the same as
http://www.nordichouse.co.uk/-p-1000.html
But should 301 redirect to: http://www.nordichouse.co.uk/linen-style-collection-p-1000.htmlOscommerce has a URL 301 re-writer that prevents the system for using incorrect URL's I would hope your system does as well.
I'm not trying to avoid helping you, but the without an exact knowledge of how the system handles URL's it generates it is hard to troubleshoot, however since it is a CMS somebody who works on it should already have this knowledge.
My best,
Don
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Having 1 page crawl error on 2 sites
Help! A few weeks back, my dev team did some "changes" (that I don't know anything about), but ever since then, my Moz crawl has only shown one page for either http://betamerica.com or http://fanex.com. Moz service was helpful in talking about a redirect loop that existed, and I asked my team to fix it, which it looks to me like they have. Still, 1 page. I used SEO Book's spider tool and it also only sees 1 page, and sees the sites as http://https://betamerica.com (for example), which is just weird. I don't know enough about HT Access or server stuff to figure out what's going on, so if someone can help me figure that out, I'd appreciate it.
Moz Pro | | BetAmerica0 -
Crawl Stats Have Dissapeared
Hi SEOmoz I received an email today that another scan has been performed but when I log into my account all the tracking details have disappeared? States Pages crawled N/A. Can someone please help? Temporary problem? Website www.vintageheirloom.com Thanks
Moz Pro | | well-its-1-louder0 -
Is seomoz rogerbot only crawling the subdomains by links or as well by id?
I´m new at seomoz and just set up a first campaign. After the first crawling i got quite a few 404 errors due to deleted (spammy) forum threads. I was sure there are no links to these deleted threads so my question is weather the seomoz rogerbot is only crawling my subdomains by links or as well by ids (the forum thread ids are serially numbered from 1 to x). If the rogerbot crawls as well serially numbered ids do i have to be concerned by the 404 error on behalf of the googlebot as well?
Moz Pro | | sauspiel0 -
In my crawl diagnostics, there are links to duplicate content. How can I track down where these links originated in?
How can I find out how SEOMOz found these links to begin with? That would help fix the issue. Where's the source page where the link was first encountered listed at?
Moz Pro | | kirklandsl0 -
Open Site Explorer results vs. Google Webmaster Tools results
I've been comparing the links to my domain that OSE and GWT show and GWT shows many more links than OSE. Can anyone explain the difference? Does Google report no follow links that OSE does not?
Moz Pro | | cartersnipes0 -
How can i get seomoz to crawl a campaign on demand
hi how can i get seomoz to crawl a campaign on demand instead of on a weekly basis? For example i have corrected some error warnings and on page elements and would like it to re crawl the site sooner to see how the corrections have worked? thanks
Moz Pro | | Bristolweb0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0 -
Pages Crawled: 250 | Limit: 250
One of my campaigns says: Pages Crawled: 250 | Limit: 250 Is this because it's new and the limit will go up to 10,000 after the crawl is complete? I have a pro account, 4 other campaigns running and should be allowed 50,000 pages in total
Moz Pro | | MirandaP0