Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at help@moz.com.
Cheers!
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking keyword and links
I have Created the website 3 months ago related to APk Games. And updating content daily and linking but the linked results in Moz are not showing Properly. my site have some ranking keywords on google at 1st but moz showing 0 ranking keywords. my question is how many time will take moz to show actual results. Thanks
Link Explorer | | khannnnnndnnnnnnnnnnss0 -
My link research tool is not working
Hello Dear i am facing a problem that my link research tool is not working is there any body else who has the same problem like me need help
Link Explorer | | Arenapile0 -
URL column of Link Explorer reports do not have the protocol (http or https)
I exported from Link Explorer. In the URL column, there is no protocol for each URL. Isn't the protocol part of what makes a url? I'm crawling each with http:// and https:// but it feels like this info could just be in the report. Am I missing a step?
Link Explorer | | Hyper-Dog0 -
Inbound links that does not appear in open site explorer
There are several links to my website that are old, and are not indexed to locate in Open Site Explorer. Which may be the problem from appearing? This is shown in webmaster tools
Link Explorer | | hectordlux0 -
Learn how to use Moz's Spam Score metric to identify high risk links. Get your Daily SEO Fix.
Almost every site has a few bad links pointing to it but risky links can have a negative impact on your search engine rankings. Watch The Moz Daily SEO Fix: How to Use Spam Score to Identify High Risk Links to learn how to spot those spammy links and what to do with them. And, if you have more questions about Spam Score, check out Rand’s blog post: "Spam Score: Moz’s New Metric to Measure Penalization Risk." This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Link Explorer | | kellyjcoop3 -
Confused about the link metrics showing up in MOZ & OSE
For my campaign "itelligence Applications" which is for site itelligencegroup.com/uk/ - MOZ seems to be giving me different stats on links, depending on which tab I look it in the Link Analysis section. Then when I look at OSE, the numbers are totally different again, not just a little bit different. Example: In MOZ Link Analysis, on the "inbound links" tab, it says there are 129 Root Domains and 6832 Total External Inbound Links. If I then look at the Competitive Metrics screen, it says there are 46,284 external followed links and 240 total linking root domains. If I then go to OSE for the same site, it says there are 14 Root Domains and 686 total links. I have no idea why these numbers vary so much or how to even report them to my client. Am I missing something here? Any help would be gratefully received! Maureen
Link Explorer | | mfrgolfgti0 -
How can I use Open Site Explorer to Identify why my own URL's are appearing as inbound links
I'm a new member of Moz, and am trying to understand why I am seeing my sites URL's in my Inbound Link results in Open Site Explorer. There are quite a few, and they are all product category URL's that are generated by my stores database. This is not happening with .html URL's. Any thoughts?
Link Explorer | | Artfx0 -
Removing the clutter of site-wide links
I have a multi-part question with regard to the moz link index and some presentation suggestions. Firstly I would be interested to know how the link index treats site-wide links with regard to metrics such as DA, and PA. We all know that it is highly likely that SE's are unlikely to pass full link value across from sitewide links, and therefore it would make sense for Moz values to account for this as well - if they do not already. One annoying thing that also relates to sitewides is that they tend to clutter the much of the information presentation in a few of the tools (you can't see wood for trees as it were). This is most prominent in the "Just Discovered" page - if you have a sitewides on a large site, you can often find that this screen is just totally filled with these links as they are found. It would be very useful to be able to filter these out, as they are of little interest - currently I can't see a way of filtering them out. A further value where they create to much noise is the 'Total Links' value. Where sitewides are included in this value, the value actually becomes pretty meaningless as you can find that the majority of that value is sitewides. It would therefore be useful if there was another value for 'Total Links - Excluding Sitewides' where maybe value of 1 was just added to the count for a site wide
Link Explorer | | James770