Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Different number of backlinks (Search console - Majestic)
Hi there, I saw a notable difference between the number of backlinks from ref domains to my website in search console and majestic. I basically have two questions: why such a difference? And which tool gives me the most accurate number between these two? Thank you!
Link Building | | skrauss0 -
What Makes A 'Natural Link Profile'?
I find it hard to recognize unnatural link patterns when the links to my site are so familiar. It can be hard to see the wood for the trees! How many links from one site is too many? Does it depend on the size of the site? Thanks for your advice on this.
Link Building | | T0BY0 -
Can 302's Negate Spam Link Profile?
To make a long story short, the previous SEO company we hired spammed us on LOTS of crappy links, many of which are still indexed. We're currently in the act of removing these links before we receive a manual spam penalty/notification + building new, stronger links to balance out our profile. While most of these garbage links were sent to our homepage, many are linking our Services page (and a bunch to our .com/sitemap.html for some reason, lol). Now we're in the midst of updating our entire website and the permalinks are going to change. While I'd normally 301 our old links to the new ones, Id rather not bring the horrible link profile with it if at all possible. Would a simple 302 redirect effectively dodge the bad juju from these spam links to our Services page since they pass 0 juice? Any thoughts on this would be appreciated. Thanks!
Link Building | | kirmeliux0 -
What's more important page authority Vs domain authority?
Hello everyone, I am fairly new to SEO so I'm still trying to get my head round everything, I am currently looking into some back links .. well looking at competitor's back links to copy. I was just wondering what's more important page authority or domain authority? So for example if a page has a page authority of 50 and a domain authority of 10 is that better than if a page have page authority 10 and a domain authority of 50. Thanks so much in advance!
Link Building | | vanplus1 -
Should we use 'disavow' if we dramatically dropped serps after Oct 5th, but did not receive a warning?
We dramatically dropped serps after Oct 5th, but did not receive a warning about bad links on google webmasters. Is it a good idea to find and collect the worst links and disavow them? As there was no warning we can only guess links were the problem, but it seems likely. However we do not want to send any requests to google if no warning was sent. Can disavow be used in this way, or should we do more manually? (We are hoping remove links + increase very solid links + optimize for key words will help revive serps) Many thanks. James
Link Building | | Quime0 -
When's Linkscape update going to go live?
I've been keeping an eye on the Linkscape update calendar and am a bit confused! There was meant to be an update yesterday, but Open Site Explorer is still showing "Last index update: 8/14/12". When will the new data reach OSE? Been doing lots of link building and want to know if my efforts have been worth it! Any info greatly appreciated. Thanks! Alex
Link Building | | reddogmusic1 -
What's your best link, and where is it from?
I'll start with ours, it's a BBC article which listed our website in the resources box, keyword rich 🙂
Link Building | | tomcraig860 -
SEO MOZ LINK BUILDING TOOLS
So I learnt that in link building you need to send an email to someone from where you want the inbound link coming in from. In this case, are there any tools withing SEO moz that allow you toactually search for links and also provide you the contact email? Thank you, Vijay
Link Building | | vijayvasu0