Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at help@moz.com.
Cheers!
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sudden surge in discovered links
Hi everyone, Recently Moz discovered a lot of old backlinks to my url. Some of them don't even exist anymore. I don't understand how Moz has found these links, even if they don't exist? And why did Moz find them suddenly all at once? Can anyone help me with this question? Thanks!
Link Explorer | | Online_Marketing_Monkey0 -
URL column of Link Explorer reports do not have the protocol (http or https)
I exported from Link Explorer. In the URL column, there is no protocol for each URL. Isn't the protocol part of what makes a url? I'm crawling each with http:// and https:// but it feels like this info could just be in the report. Am I missing a step?
Link Explorer | | Hyper-Dog0 -
No link to Open Site Explorer from Campaigns tracked keywords?
Hi there, The Guide to tracking your keyword rankings in Moz Pro states that "Click on the keyword to analyze it, the URL to be taken to that page in Open Site Explorer or the Optimize link to get a Page Optimization Score with recommendations.". But my URLs all target to the specific page on my site. I figure it would be nice to dive deeper in analysing specific pages, using Open Site Explorer Is just the link with Open Site Explorer broken, or is the Guide out of date? Thanks, Arnout
Link Explorer | | BentoPres1 -
Campaign shows website links from Https. My site is not https but http:// HELP
When looking at my web campaign, I have inbound links pointing from every page located on my menu from https:// I do not have an https. website in any way. The weird part is that when I click on the link, it is a replica of my homepage, although in text only format. I have attached a picture for reference. Why and how could my top links be coming from this? I do not have any inbound links non https:// from my website Thank you! BNHHc5u
Link Explorer | | Morg56850 -
Why do links from UCAS to London South Bank University not get reported as external links
Links on the following page at UCAS http://search.ucas.com/provider/1504/london-south-bank-university?Vac=1&AvailableIn=2015&ProviderQuery=london south bank university&ret=providers do not appear to be listed as external links in Open Site Explorer reports. For example there is a link on the UCAS page to http://www.lsbu.ac.uk/schools-departments . But when I check the Open Site Explorer report for http://www.lsbu.ac.uk/schools-departments no link from UCAS is recorded.
Link Explorer | | southbankuni0 -
Psst Did you see Open Site Explorer's New Link Building Opportunities Section??
Check out Rand's blog post about it for all the details: http://moz.com/blog/open-site-explorers-new-link-building-opportunities-section. We'd love for you to check it out and leave us your feedback! Thanks,
Link Explorer | | jennita
Jen3 -
Why does the number of the total external links and the followed linking root domains between Open site explorer and the campaign in Moz pro doesn't match?
Opensiteexplorer returns 52.473 total external links and 715 followed linking root domains while my campaign in Moz analytics return 35.899 total external links and 1092 followed linking root domains. Does anyone knows how this is possible?
Link Explorer | | ConversionMobstars0 -
Service riddled with bugs, no support, and no one cares
I upgraded my account to Large on Sept, 21st from basic. On Oct 3 I raised a ticket to support stating that my account was not utilizing the full crawl. I was informed that Sam would make the change for me and I would see the change in the next crawl. I waited until the next crawl and nothing changed. I raised another ticket stating what took place in the first ticket and that the crawl limit was changed and that we were still having trouble crawling 50,000 pages, the site is 59,000 index-able pages. As part of the second ticket I indicated that I was also experiencing trouble with Moz Analytics only allowing data from Sept 9th to the 15th and I could not change dates. I received a response to the second ticket 27 hours later from David. David requested that I disconnect GA and reconnect it. I told him that I had already done this but that I would do it again. I also asked for a credit to my account because I had not been able to use what I was paying for. David responded stating that he would forward this to an engineer for a response, I waited until the 21st, I then raised another ticket. I immediately received a automated response and have not had anyone response to the third ticket nearly 5 days later. It saddens me to see that Moz has gone considerably down hill in their desire to furnish tools to SEO's that work. No one seems to care. As part of the cancellation process you are furnished a phone number to call for help. I called the number and after holding for 4-5 minutes I was asked to leave a voicemail which I did explaining my situation. No response yet. I was left with really nothing else to do other than finish the cancellation and chargeback all of the service charges since my upgrade. Sorry for the rant but hope this helps others.
Link Explorer | | nhnf85a1