Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are there less backlink domains in Moz vs. Semrush?
For our domain studyville.com, Semrush is reporting 46 linking domains, and Moz is reporting 7. Does anyone know where there is such a large discrepancy?
Link Building | | shelbythomas0 -
My Backlinks are indexed in Ahrefs But Not Indexed in MOZ. Why?
I Create backlinks for my website to Increase DA But Backlinks are indexed and Ahrefs Show that backlinks but MOZ is showing backlinks. I am confuse Can you explain?
Link Building | | Seogamesokay1a1 -
How does a business have a link back where I can't see the anchor text?
So, I'm running competitor websites through Open Site Explorer and following the link from Moz to where their link is coming from. It has the anchor text 'Christmas Gifts'. When I go to that page there is no mention of this anchor text. No mention of that persons website. No display ad's that could be providing that link. How is this link pointing to their page?
Link Building | | Byron_Bay_Gifts0 -
Sitemap.xml change to anchor links (converter)
Hi all, I would like import the sitemap.xml to an anchorlinks list. List with links navigatetion html ^^. Is there a posibility to use a tool or in excel? Because I will use 5000 products and 1000 brands.
Link Building | | Dreamgame20160 -
Should I built a microsite for client's new service or add a new section to an existing new website?
We built a new website for one of our clients. Covering services A B C mostly commercial and industrial B2B construction service. We bought him a brand new domain called serviceAcompanyname.com The website uses a brand new domain and as no DA and PA yet. The client wants a new section added to the Website concerning a new service ( we are going to call it service D) which is still a local service but offered to Home owners. Should we buy a new domain called serviceD-region-companyname.com and make a microsite covering the topic and work on backlinks and links in parallel for both sites, or would we be better off just adding a new section to the existing website and work on the main DA. Will it be easier in the future to enhance 2 different domains or a single one even though all services are not targeting the same audience.
Link Building | | escteam0 -
How do sites have so many 'total links'?
I've been analyzing some of our competitors: essayedge.com and papercheck.com Both sites have a large number of 'total links'... about 93,000 each. The former has about 1,200 linking root domains while the latter only has 195. Even for 1,200 linking root domains, 93k total links seems like a ton to me. Our site has 101 linking root domains and only 299 'total links'. I'm quite new to this whole SEO game and admittedly still learning a TON. Am I missing something here? How do sites generate so many links? This seems nuts to me. Thanks for any help!
Link Building | | TBiz0 -
JavaScript is crawled by search engines, isn’t it? Does it mean that links embedded in JavaScript pass link juice?
I wonder If links embedded in JavaScript from an external Website pass link juice to the linked page and thus have a positive effect on google rankings. I read that JavaScipt is craweld. Does it mean that also the link juice is passed? I'm looking forward to your answers.
Link Building | | Tabea0 -
What's the best way to find some guest blogging opportunities?
In the forum's experience, what have you found the best way to find guest blogging opportunities? e.g. google search syntax that has worked well and other more diverse methods would be appreciated! Thanks
Link Building | | PeterAlexLeigh0