Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlinks on Moz not on Google Search Console
Moz is showing thousands of backlinks to my site that are not showing up on Google Search Console - which is good because those links were created by some spammer in Pakistan somewhere. I haven't yet submitted a disavow report to Google of well over 10K links because the list keeps growing every day with new backlinks that have been rerouted to a 404 page. I have asked Google to clarify and they put my question on their forum for an answer, which I'm still waiting for - so I thought I'd try my luck here. My question... If Moz does not match Google Search Console, and backlinks are important to results, how valid is the ranking that Moz creates to let me know how I'm doing in this competition and if I'm improving or not. If the goal is to get Google to pay attention and I use Moz to help me figure out how to do this, how can I do that if the backlink information isn't the same - by literally over 10 000 backlinks created by some spammer doing odd things... They've included the url from their deleted profile on my site with 100s of other urls, including Moz.com and are posting them everywhere with their preferred anchor text. Moz ranking considers the thousands of spam backlinks I can't get rid of and Google ignores them or disavows them. So isn't the rankings, data, and graphs apples and bananas? How can I know what my site's strength really is and if I'm improving or not if the data doesn't match? Complete SEO Novice Shannon Peel
Link Building | | MarketAPeel
Brand Storyteller
MarketAPeel0 -
Is too high a frequency of 'money' keywords backlinks considered factor for Penguin penalties, even if the money keywords are on reputable pages ?
Is too high a frequency of 'money' keywords backlinks (eg. a money keyword backlink for moz.com would be "Seo tools") a considered factor for Penguin penalties even if the money keywords are only on reputable pages, with decent PA, DA and trust ?
Link Building | | jpeg800 -
Ugh content! I don't get it
Everyone on this website says the proper way to build links is with great content. I don't get that because I am trying to build links for a classified ads website and there is no "great content" for any of the classified ads websites. None of them have blogs or any content other than the stuff people are selling. When I look at my competitors inbound links using open site explorer, they are all junk. Most of the links listed in open site explorer are dead /broken links and the ones that are not dead links are spammy, low quality links including many link exchanges. So, I don't get the "great content to build links" route because in my vertical it's not being done. Suggestions on how to build links in this situation?
Link Building | | Noob150 -
Does paying a reviewer for an impartial review violate Google's guidelines?
When a company pays for an impartial review from a website, should these links be no-followed? I am confident that paid positive reviews are seen as a manipulation of search, but is paying for an impartial review okay?
Link Building | | RG_SEO0 -
301 redirect of PR 6 domain to PR 1 site, Will it improve my site's seo?
I have a pagerank 1 website but i recently purchased one old pagerank 6 domain, if I 301 redirect this pr6 domain to my currrent pr1 site, will it will increase my site's pr and improve seo ?
Link Building | | IndiaFPS0 -
Many competitor's backlinks are in content anchor links. Ho do I get these same links?
Hi I managed to open up OSE. I'm finding that much of the competition's backlinks are in content anchor text links. Am I supposed to get backlinks from these same pages using the same anchor text but linking back to my page or is that allowed? If so, how do I get these in content blog post links? Thanks. Sunil.
Link Building | | sunilmuse0 -
Is it ever the case that a link isn't worth having?
We have been asked to provide a link on our website to a website which provides us with recycled computers to use (our company brand is based around recycling). They have agreed that we will get a reciprocal link back to our site. However the comparisons show that - perhaps this isn't worth it? Our page authority: 39, theirs 26. Our domain authority: 41, theirs 14. Our linking root domains: 41 theirs 7. Finally their links are in the # [li] list [/li] # format I guess my question is threefold: 1 - Will they get significantly more out of this linking opportunity than we will? 2 - Is it worth having the link anyway? (Can it hurt us in any way?). 3 - Does having links in the # [li] list [/li] # format effect the power of the links contained? Thanks.
Link Building | | Benj250 -
Does it pay to change link text internally?
Most of my internal pages have their best links from within our site (of course we are trying to change that). Is it worth the effort to sculpt the link text to show varied text instead of most all showing the same link text? Or is that only important from external links?
Link Building | | joemas990