Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
-
Moz crawler is crawling all pages like this:
- http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date
- http://www.xxxx.com/?product_count=100&product_order=desc&paged=1
- http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid
Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us.
Thanks.
-
The immediate solution is use your robots.txt file to block the Moz crawler from crawling URLs with parameters. Pamela.
User-agent: rogerbot
Disallow: /*?utmThose pages are coming from the bot trying to follow links to all the different ways product pages can be sorted. You'll want to insure Googlebot isn't having the same problem.
Hope that helps;
Paul
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On-Page Grader URL inaccessible when copy/pasted but not when edited
Hi!, I've looked through multiple topics on this but none quite seem to fit what's going on - hopefully someone can help! I get the error message 'Sorry, but that URL is inaccessible.' when I copy and paste a url from my site into the search e.g. http://www.orbussoftware.com/enterprise-architecture/ However if I edit this to https the search completes fine. Since we redesigned our site approx 6 months ago, we've found most of our rankings have completely dropped off, and now I'm getting this error I'm wondering if it has something to do with how our site is structured? If I'm getting this error with Moz does that mean Google could be having issues too? Or is it all just a strange quirk? Thanks!
Moz Bar | | JennaOrbus0 -
Is there a tool that works like crawl test that allows more than 3000 pages?
I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site. It would probably be in the neighborhood of 10,000 urls. Does anyone know of a free tool and if not is there a paid tool that will do this?
Moz Bar | | bradwayland0 -
Is there a way to track mobile rankings vs desktop rankings in Moz?
With the new release of Google's mobile algorithm we want to start tracking keywords mobile vs desktop. Any suggestions?
Moz Bar | | TicketCity3 -
Duplicate content reported for totally different pages
Hi, The Moz report is showing just over 21,500 duplicate page issues on our site. This is more or less every page we have. However when I look at the pages it says are duplicates they are totally different (it could for example report that a news page for 2009 is the same as a product page just added which has no relation when you read the content or view the page). What sort of thing could it be picking up as duplicate content? I assume it must be something in the HTML for the site rather than the actual page content as there is no cross over at all on the pages highlighted. The only issue I can currently identify is that the menu for the mobile version of the site has a huge number of internal links which I will cut down. If the tools purely look at HTML content this could be seen as duplicate but shouldn't it be clever enough to realise what is content and what is site structure? Thanks,
Moz Bar | | TW-Steve0 -
Does moz Ranking report canadian rankings?
Hi I would like to know Moz rankings can report canadian ranking or does it just focuses on US market search engines only? If it does also focuses on the canadian market, how do you set it up? Thanks
Moz Bar | | Ideas-Money-Art0 -
Issues adding keywords to MOZ analytics
Hi All 🙂 I've started a couple of campaigns in MOZ Analytics, one accepts keywords the other looks like it does but doesn't actually add them.. I'm pasting a group of keywords, about 30, all separated with comma's. Doing exactly the same to one account that congratulated me straight away and added them to the list below, yay! The other shows the spinning wheel for a few seconds and does nothing further, I've waited a good 24 hours and they keywords have not appeared. Any help appreciated! Thanks 🙂
Moz Bar | | spanda0 -
Moz showing warnings for each dynamic link despite canonicalization?
As you can see in the attached image, Moz is showing a warning for each dynamic URL despite a rel=canonical tag. Is this by design? If so, it is frustrating seeing as it is really just the one page with many links . . . 9h5oDmr.png
Moz Bar | | BlueLinkERP0 -
Duplicate Page Content Report on MOZ
Hi, I am just wondering as to the accuracy of this report - does it pick up all the duplicate on page content? Or is there a limit? We have an ecommerce store with a lot of copied and pasted descriptions - just wondering if there is a limit on how much the moz crawler picks up? In other words, once we fix what MOZ has detected, will there be more detected because it is limited to display say up to 200?? Hope you understand what I mean. Thanks
Moz Bar | | bjs20100