Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlinks on sites that aren't relevant, how does google determine relevancy?
I have a competitor who I've talked about on here before about how they passed us in ranking for our main keywords and that they were a new site ( less than 6 months old ). Well after further digging a few things were found out: Our site seems to have a penguin penalty causing our keywords ( a group of synonyms as well ) from ranking better than rank 30. As well our competitor has been using this weird backlink tactic of using SEO sites ( a friend? ), a parent site to prop it up ( footer links and others ) as well has links inside articles that aren't relevant as in having a commercial niche page inside a lawyer site page talking about phone dialing. Then the low DA directory sites. I'm curious as to if penguin algo will catch them once it comes out, if not penguin a manual review would certainly catch their tactics, but this site is doing everything else other than proper SEO. What do you think?
Link Building | | Deacyde0 -
My website Moz Page Authority
My website is Best Comedy Tickets. The current homepage has a Page Authority is 41, mR : 4:53 and Domain Authority is 31. However all my sub pages are basically non existent. All the other pages on the site are a Page Authority of 1, mR : 0.00. Why is this and how do I increase this ? Are there any tips or tricks to increase subpages ? I appreciate all the feed back and am grateful for any tips or helpful advice.
Link Building | | JosephSantiagoNYC0 -
What can I do to improve my site's local search ranking?
I've read a lot about the "Magic 7" ranking factors for local search, and I feel like in a lot of ways, my site (www.imageworkscreative.com) is doing all the right things and still not having terrific results. The only thing I can think of to really improve upon is how often our local keywords are mentioned on our pages - maybe setting up a local blog or something similar would be helpful? Adding footers that include the names of local cities? Does anyone have any pointers for me? We would really like to rank for things like dc web design, web design va, etc. - and fast. We have (recent) links from two local Chambers of Commerce and we're working on getting accredited by the BBB. I feel like those listings will help us, but there has to be more we can do.
Link Building | | ScottImageWorks0 -
Should you ever change your anchor text ?
Hello I have a question about anchors. I have done all my own seo over the last 3 years, with tools from various sites. I had an seo audit done about 1 month ago and was told my link profile was very natural. They had one recommendation. To go back over my link profile and ask some webmasters if they would change the anchor from the name of my site or my url to a more seo friendly phrase. This seemed logical. I never did a lot of anchor text just name or site or url. Anyway, over last 4 weeks I have messaged several webmasters and asked to have anchor text changed to something along the lines of the keywords Im targeting. Tedious task to go through all the links but I changed several anchors to what was recommended. I was also out link building at same time. These last links and I got several of them all natural links after 16 hours work days. Are all will seo friendly anchors, because as Ive gotten more experienced my links have gotten more in lines of what is "seo friendly" or at least I hoped. I asked one webmaster to change my anchor and he warned me I would be slapped with a penguin penalty and wouldn't recommend I do this. I have already done this to several of my links. Then today the new seomoz update came up and I was down on DA and PA by 3-4 points. Do these have anything to do with one another and have I been given bad advice and can I fix it if I have ? Sorry about long post just a little confused. I don't want to step into penalty land and not know I did.
Link Building | | New1000ad0 -
Changing url from non www to www.
basically I have realised i have more page authority and links going to www.onestopmuscle.co.uk instead of onestopmuscle.co.uk. I use wordpress and have no clue how to make sure onestopmuscle.co.uk redirects to www. version. anyone have any ideas? I don't want to mess about with files or i'll most likely make the blog into a disaster.
Link Building | | FLEAR0 -
Why aren't my links being indexed
I am building backlinks from similar sources to 2 different domains. Domain A seems to get its linked indexed within a day or 2, while Domain B has virtually identical links that haven't indexed for over a month. Domain A is older and has more authority, and Domain B is hosted on a different server. Any insight into why this is happening? It's very frustrating.
Link Building | | insitegoogle0 -
Why aren't my backlinks showing up?
I've recently switched hosting my site on www.vamospaella.com to www.vamospaella.co.uk (using a 301 redirect) and I've been building backlinks. According to Majesticseo, vamospaella.co.uk has 38 backlinks with 8 referrring domains and vamospaella.com has 15 external backlinks from 11 referring domains. Some of these backlinks date back a month, while others have shown up in the last week. However, I have seen no improvement in my Google rankings for my keywords. Why is this?
Link Building | | RMelly0 -
How good is a backlink that's in the footer
Hello, The strongest site in our industry (according to domain authority and excluding wikipedia) said that they would put a sitewide link to us in their footer. We're good friends with them. It would be right next to the copyright. Our site is nlpca (dot) com The partner site is nlpu (dot) com The link will say something like "More NLP Training" with the "NLP" as the link. We're targeting the keyword "NLP" How much will this move us up for the keyword "NLP"? Right now we're on the 3rd page for that term. I also want to make sure that it's a white hat move. Thanks!
Link Building | | BobGW0