Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?
-
Well, basically that's the question Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?
This is, I have more than 10.000 pages on the website, and I am not interested in having reports for many of them, but I still wanna get SEO visits on them, so I want Google to crawl it easily...
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hi Matt,
Absolutely, you could do this by adding a special part to your robots.txt file specified to the user agent rogerbot which is used by SEOMoz for their spider. This could be done with for example:
User-agent: rogerbot Disallow: */anythingyouwanttoexcludeforroger/*
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I combine SEOmoz Campaigns.
Basically I am in internal SEO. We subscribe to SEOmoz and we have 5 campaign spots. each with their own allocation of keywords. Ideally I only want to track our site and not 4 other's. I've currently setup the campaign a couple of times with different keywords, but it would be much better if I could choose what allocates to what. I.e. 2 campaign tokens to one site and 3 to another or just 5 to one site. Is this possible, and if not why not?
Moz Pro | | DaveDawson0 -
404 errors in SEOMoz crawl tool
I currently have several 404 errors in the latest crawls from SEOMoz. Here is an example of the error. http://dealerplatform.com/blog/2011/10/23/videos-for-auto-dealers/www.dealerplatform.com/ In all cases the error is a result of www.dealerplatform.com being added to the real url. Anyone seen this before? The site is a wordpress mutlisite. I don't see where this incorrect link is showing up anywhere on the website. Any advice would be helpful. Thanks
Moz Pro | | Chris_Gregory0 -
How can authority of the page be higher then authority of the domain?
In the opensite explorer most of the links that I see have a higher domain authority and lower page authority which makes sense to me because I thought that domain authority is an aggregate of its pages authority. But apperently I was wrong because sometimes I see that the page authority is greater then its domain? How could that be?
Moz Pro | | SirMax0 -
Only crawling one page
Hi there, A campaign was crawling fine, but at the last crawl, for some reason, SEOmoz can only crawl one page... any ideas? If I run a custom crawl I still access all of the site's pages.
Moz Pro | | harryholmes0070 -
Can you change crawl day of week?
Can I somehow sync the day of the week for each of my campaigns' crawls, so that all campaigns are updated on the same day?
Moz Pro | | ATShock0 -
MOZ Crawler only crawling one page per campaign
We set up some new campaigns, and now for the last two weekly crawls, the crawler is only accessing one page per campaign. Any ideas why this is happening? PS - two weeks back we did "upgrade" the account. Could this have been an issue?
Moz Pro | | AllaO0 -
Crawling One Page
I set up a profile for a site with many pages, opting for setting up as a root directory. When SEOMoz crawled, they only found one page. Any ideas for why this would be? Thanks!
Moz Pro | | Group160 -
How is SeoMoz different?
In the past I heard that other SEO type crawling software like WP Gold could seriously harm your rankings in the search engines - especially Google. How is SeoMoz different ?
Moz Pro | | NeilTompkins0