Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Backlinks are indexed in Ahrefs But Not Indexed in MOZ. Why?
I Create backlinks for my website to Increase DA But Backlinks are indexed and Ahrefs Show that backlinks but MOZ is showing backlinks. I am confuse Can you explain?
Link Building | | Seogamesokay1a0 -
Should I built a microsite for client's new service or add a new section to an existing new website?
We built a new website for one of our clients. Covering services A B C mostly commercial and industrial B2B construction service. We bought him a brand new domain called serviceAcompanyname.com The website uses a brand new domain and as no DA and PA yet. The client wants a new section added to the Website concerning a new service ( we are going to call it service D) which is still a local service but offered to Home owners. Should we buy a new domain called serviceD-region-companyname.com and make a microsite covering the topic and work on backlinks and links in parallel for both sites, or would we be better off just adding a new section to the existing website and work on the main DA. Will it be easier in the future to enhance 2 different domains or a single one even though all services are not targeting the same audience.
Link Building | | escteam0 -
Why Links from Top Level Domains doesn't Pass Link Equity?
Hello, I have a doubt about Equity-Passing links report of OpensiteExplorer.org According to Sam Weber http://moz.com/community/users/432678 links which pass value from one page to another including followed 301 and Meta refresh links are under Equity-Passing links. I am surprising after looking at the links report generated by opensiteexplorer.org. Most of the article and directory links which have low DA and PA are under Equity-Passing links. Whereas websites like EzineArticle or Articlebase which have good DA and PA are not found in either Equity-passing links or under only nofollow category. Please suggest me if the report of Opensiteexplorer is not good enough or the links from the site like Ezine Article doesn’t pass the link equity. Thanks.
Link Building | | TopLeagueTechnologies0 -
Back linking to t foreign sites
One of our major competitors seems to be linking to an Asian speaking website that has a blog/product review formant. I was wondering how they are achieving this. Are there any non English /Asian sites worth submitting to and what is the best way to go about this if English is the only language you speak! And of course is it worth while doing this from an SEO perspective.
Link Building | | Hardley10 -
What's a really good example of a linkbait-y Category / Subcategory hierarchy?
Rand makes a really great point in this 2009 post about the shape of crawl paths: "#4. Craft navigation / category pages that are worthy of links. If you can make these pages worthy of links and attention, you drive PageRank and crawl priority further down your site's architecture into the content (and signal the engine that ALL your pages are important." Which makes sense, intuitively, because you'd like link juice to flow directly and undiluted to your money pages. "Here's all my Green Widgets, Roger: they're all right here. While you're at it, here's a related blog post—'5 Ridiculously Awesome Things Every Green Widget Buyer Should Know'—and, oh look! Would you like to see my Blue Widgets as well?" In practice, though, the Home » Widgets » Green Widgets doesn't sound all that alluring. Useful, absolutely, for UX, but not for getting links. Anyone have some favorite examples of Category / Subcategory hierarchies that do well as link-bait? Client is a marketing agency dealing in the technical arcana of databases and ad serving, so their money pages won't be as specific as a Green Widget or a Miami Hotel. Their site isn't huge, and the pages will be extensively interlinked, so the emphasis has more to do with link juice / page authority than indexation. But I'm wondering if it could it be smart to replace a generic "Services" category with a KW-rich drop-down menu of "Marketing Solutions" (i.e. 'Increase Customer Retention') and link each landing page to a relevant charcuterie of services, white papers, webinars, case studies, etc., rather than keeping these pages in their respective silos—even as they link horizontally to related services?
Link Building | | sweetfancymoses2 -
Changing url from non www to www.
basically I have realised i have more page authority and links going to www.onestopmuscle.co.uk instead of onestopmuscle.co.uk. I use wordpress and have no clue how to make sure onestopmuscle.co.uk redirects to www. version. anyone have any ideas? I don't want to mess about with files or i'll most likely make the blog into a disaster.
Link Building | | FLEAR0 -
Why aren't my links being indexed
I am building backlinks from similar sources to 2 different domains. Domain A seems to get its linked indexed within a day or 2, while Domain B has virtually identical links that haven't indexed for over a month. Domain A is older and has more authority, and Domain B is hosted on a different server. Any insight into why this is happening? It's very frustrating.
Link Building | | insitegoogle0 -
April Fool's for Link Building
Is there a better day for link baiting than April 1st? The time has come for everyone to share their April Fool's Day link building campaigns! This is our second attempt at capitalizing on April Fools with the first attempt (2010) falling flat. http://bigfi.sh/aFd2011
Link Building | | RyanOD0