Specific external links
-
So...we have an A+ rating with the BBB and have for some time. We have a link from their page. We also use sites like pricegrabber and reseller rankings and have links from those.
What I can't figure out is why links from the BBB and Pricegrabber don't even show up in open site explorer but the reseller ratings links are everywhere. The BBB one is easily the oldest and the other two are roughly the same age (i.e. at least a couple years old.)
-
The Open Site Explorer indexed is completely refreshed every few weeks, and currently consists of 105 billion links. While that's quite a large data set and mostly comparable with other indexes in the space, it's still only a fraction of the links on the web and only a portion of the links Google undoubtedly knows about.
The way OSE finds links is that it prioritizes pages with lots of inbound links and links it found during previous crawls. Because BBB pages are often found deep-deep-deep under layers of navigation and usually don't have a lot of external links pointing to them, they aren't easy pages for crawlers to find on a regular basis. BBB has an especially difficult site architecture, and sadly this issue comes up a lot with that particular site
We're working on including more historical data in the index, so that even if we didn't crawl the link in the past two weeks, we'll still include it if we are reasonably confident it still exist. This will help alleviate some of these concerns.
The good news (or bad depending on how you look at it) is that while OSE doesn't show every link, it tends to show the majority of the most valuable links.
Regardless, thanks for the feedback and best of luck with your SEO.
-
I share your frustration in wanting to be able to track links in OSE. Someone at Moz told me a while back that OSE only reaches 20-25% of links out there. I've just come to tell myself that if I know the link exists then I don't need to see it in OSE.
-
I wonder if Moz understands and follows the Robots.txt file? I know at one point they never, back in the day when I used Moz most of my errors came from URL's which were hidden by robots which would not of caused errors for Google.
& what your saying Oren makes perfect sense if it's correct. Why would they want to crawl aggregated content.
-
Perhaps a Mozzer can verify this, but I remember someone saying they Moz doesn't crawl some big sites that aggregate lots of data (twitter, BBB, yelp, domain listings, etc. etc.) because of the size of their servers, and the likelihood that the user put it there themselves. They have smaller servers than say, Google or Bing.
-
Hi,
Firstly it could be that Open Site Explorer has not had time to pick them up, if it's years old then let's not forget the OSE manually crawls the internet and is not Google's crawler meaning there will be sections of the web which won't be crawled by Moz.
If it's old then there may be no links pointing to these pages which would not give OSE a reason to crawl them, certainly if it's years. Don't worry about it, a missing link on OSE is one less your competitors can find I had a similar issue with a few links from BOTW but in time it updated and works fine but Open Site Explorer is not flawless.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block Moz (or any other robot) from crawling pages with specific URLs
Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
Moz Pro | | Blacktie
Disallow: /*numberOfStars=0 User-agent: rogerbot
Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!0 -
Why is Link Count smaller than Internal Links in Crawl Test report?
We recently ran the crawl test report and for most of our pages we are getting 1150 internal links but 40-50 as the link count. Why is there such a big disparity?
Moz Pro | | usdmseo0 -
Google and Open Site Explorer not showing as many links
I've noticed this past week that when you search for the links pointing to a given site, by using the "link:" operator, that Google not showing as many links as they use to. I noticed this also with Open Site Explorer, it is not showing the detail link information as much as it did before. Is Google trying to mask what we can view now on competitors backlinks? If so, how can we see the backlink building that our competitors are doing?
Moz Pro | | tdawson090 -
Is it possible to track and rank internal links?
Trying to create my cornerstone pages and I'd like to see which pages have the most internal links already to get started. Is there a tool for this?
Moz Pro | | hyperthalamus0 -
MozRank per Link: Valuable or Worthless?
Since the MozRank and MozTrust metrics are on a logarithmic scale, I'm wondering if it's still valuable to get a general idea of the quality of a site's link profile by calculating: MozRank / Total Links MozTrust / Total Links Would this be useful, or is it always going to be higher for sites with lower MozRank & MozTrust since it's "easier" to achieve the lower values?
Moz Pro | | conquerapathy-644120 -
What is the difference between followed/no followed/ internal/ external links? And how do i improve them?
Am on a beginner level of SEO and as a pro member when i ran a report for my site i found some basic terms used. Can anyone please let me know what exactly is the difference between these terms and how important are they and how do i improve them?. or please let me know a source to find out.
Moz Pro | | NikhilM0 -
Open site explorer - Analysing competitors links
When using open site explorer to analyse my competitors links, when going through them, I would click them to have a look at them. Almost all of the ones I clicked on something started to download on my computer and would be a blank page or the tab would disappear when the download begun. What is this? Why have they done this? Is this bad practice? Thanks
Moz Pro | | CompleteOffice0 -
Link Count Per Page Including JavaScript Links - Should We Worry About Them?
With large ecommerce sites, we usually have more than 100 links per page and many times have more than 200 links on each page due to links and images in the header, footer, guided navigation and then the body product grid and content. When I use most on-page link counting tools like SEO x-ray and the SEO Moz Pro crawl report, I notice that every visible link on the page gets counted. This includes and javascript based links that expand the product grid to 30, 60 or view all, javascript sorting links, javascript links to view customer reviews for each product. etc. There was a QA post here http://www.seomoz.org/q/should-i-nofollow-the-main-navigation-on-certain-pages about nofollowing and page rank sculpting and it seems pretty unanimous that most don't think that page rank sculpting is very valuable. So my question is, are the javascript links on pages that don't link to another page viewed differently by search engines? If so, shouldn't there be a way to see on-page link count minus javascript call links that don't actually link to another page? To expand a bit on my question, we also use nofollow attributes on the text links in the left navigation that are meant for refining products just as the javascript links in the product grid are meant to refine the products, sort them, allow for product comparison, allow for viewing customer reviews, etc. So should it be ok to have 300 links on a page if the unimportant ones that you don't want crawled like the left navigation refinements and product grid javascript links all have rel="nofollow" applied to them? I know that would basicly be PageRank sculting, but it seems like the best options for shopping sites that have a lot of navigation links.
Moz Pro | | abernhardt0