Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ahrefs Backlinks VS Moz Backlinks
HI Team and members We have website related to jobs in Pakistan and we purchased 5 niche edits backlinks to a specific page " MES Jobs 2021" In ahrefs it show only 2 backlinks to that page but in moz 0 backlinks but in actually there are 5 links. Why this happen in moz? We actually love this tool used for Keywords research for jobs in Pakistan find some cool keywords using moz but we facing backlinks issues in it.
Link Building | | AliHassanbinali1 -
How can i get more moz D.A on my website?
Hello there, I want to increase my Domain authority but how can i do that? please suggest to me some tips here is my website: gangstarmodapk.com And also how can make sure that this can help in ranking? what parameter are in background and what should be followed for better performance.
Link Building | | seopack.org4564x1 -
Link Building/ Off page strategy. Do we have already have a 'hook' or should we create more specific content.
Hi Guys. As part of our link building campaign I've been creating a database of people who might link to our site. We're a diary company. So I’ve been focusing on people interested in stationary, paper, pens, etc who also blog. We have a unique product at TOAD diaries, it's essentially an online tool that allows you to design your own diary. You can choose size, colour, duration, etc and also personalise the cover. So the product (we think) is very compelling and interesting. First of it’s kind. Check it out here: http://www.toaddiaries.co.uk/designer/diary/a5/everyday-diary-week-across-2-pages/coil-bound/toad-plum/12/1/8/2014 So.... question. Do you think that the site itself would considered 'good content'? i.e) It's already a very interesting idea that's worth linking to. Or Would it be better to create a high quality engaging blog (with info-graphics etc) that really speaks to that community of people? Say, about our love of the humble paper diary, and why it's still useful. Then use that blog content to try get links? You thoughts would be really appreciated. Thanks in advance. Isaac.
Link Building | | isaac6630 -
Is it better to link back to the root domain, or the specific page you're optimizing for?
I'm working on a project our site - building links for iPhone Repair. For the term "iphone repair miami" - is it more ideal to link to the root domain (xxx.com) or to the subpage that is about iPhone Repair (xxx.com/iphone-repair)? I would imagine the latter - but obviously that page has less authority than the root. Thanks!
Link Building | | preemo0 -
Open Site Explorer - Finding 404s from my competitor's external linking Root Domains
I want to find websites that have linked 404s to my competitors. My goal is to contact webmasters who have linked to my competitors with 404s, but I cannot figure out how to get Open Site Explorer to give me that data. I can easily see this in GWT for my own site, but I need this for my competition instead. Does anybody know a quick way to get this information?
Link Building | | Francisco_Meza0 -
301 to main page or deeper page for best 'link juice'.
I have two rafting websites. One ranks high for one set of keywords. The other ranks okay for a different set. I want to redirect the whole second site to the first to give the first more link juice and hopefully lift the first site even more in the rankings. Would it be better to link to a deeper page of the 1st site. I have a page that does well for some long tail keywords or would it be better to link to the main page. Or is it a bad idea over all.
Link Building | | tkobrien0 -
Finding competitor's 301's
Is there a way to find out what domains a competitor has 301'd to their main site? I am wondering if a competitor has hidden back links using a 301'd site.
Link Building | | EugeneF0 -
April Fool's for Link Building
Is there a better day for link baiting than April 1st? The time has come for everyone to share their April Fool's Day link building campaigns! This is our second attempt at capitalizing on April Fools with the first attempt (2010) falling flat. http://bigfi.sh/aFd2011
Link Building | | RyanOD0