Mozscape Index
-
Hello:
There was a Mozscape Index scheduled 9/8/2015 and now it go pushed back October 8,2015.
There seems to be a lot of delays with the Mozscape Index. Is this something we should expect? Updates every 2 months instead of every month?
Thanks!
-
Thank you for your question and concerns. Moz is committed to providing high quality metrics. I wrote a piece about it prior to my joining Moz a month or two ago comparing the proportional representativeness of Moz's index to Google vs. the other major providers, which showed that Moz's crawl was more representative of Google than any other. An unfortunate side effect of monitoring index and metric quality is actually doing something about it when things don't look quite right. We would rather know we delayed an unsatisfactory release than not know if we delivered a satisfactory one.
That being said, I think you will be pleasantly surprised over the next year or so with the improvements you see coming out of the Big Data team in everything from quality to quantity and stability. They are doing some truly amazing things.
-
Yeah, you guys have pretty much nailed it. So sorry, again, for the delay!
-
Moz finally added an update on the Mozscape API Updates page:
https://moz.com/products/api/updates
TL:DR; This month's update has been cancelled.
-
Just adding to what Dmitrii said...
Rand offered an explanation in August on this Q&A - https://moz.com/community/q/august-3rd-mozscape-index-update-our-largest-index-but-nearly-a-monthly-late.
Like Dmitrii, while I can understand why the delays are happening, I hate learning after the fact that the update is delayed. I've concluded that every two months is the new normal, at least for now, and am setting my own and client expectations accordingly.
-
Hello there.
If you go here - https://moz.com/products/api/updates You can see that it's far the first delay and that it's somewhat "normal". So, I'm kind of used to it. Now, also if you go here - https://moz.com/about/jobs#listings
You can see that there are lots of open positions for engineering staff. So, if you are willing to help MOZ improve - there is a good chance They are growing and their amount of clientele is growing even faster. I'm not trying to justify the delay, but at the same time i do understand why it's happening.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Mozscape API subscription
We have questions regarding our subscription and the plan we are on. We are more interested in Mozscape API and not the features that we have access to currently. Will you let us know how we can change. Is there some one that we can chat with. Thanks,
API | | PatientPop
Naveen
naveen.sarabu@patientpop.com0 -
MOZscape API Signup "An unknown error has occurred"
Hello, I am not able to signup for MOZscape API, I am getting error while signing up for MOZScape API under free trial. https://moz.com/checkout/api --> Getting error here, please help. Thanks.
API | | rahul2k11in0 -
Do you fetch website titles from paid api https://moz.com/help/guides/moz-api/mozscape/api-reference/url-metrics?
We are using one of your API: https://moz.com/help/guides/moz-api/mozscape/api-reference/url-metrics on our website and it does not show up the title for each website.
API | | SOSCreatives
But when I see the title of the same website through your extension Moz Bar, then it does show the title from the extension of that website. Can you tell me what is missing here?0 -
Spring is here and so is our May Index Update!
Happy Index Release Day! For the second month in a row, our hard-working, supremely dedicated Big Data team has delivered our Index Update EARLY! Beyond being punctual, the May Index is one of our most comprehensive and largest update of the year for Moz. Let’s dig into the details: 162,225,495,455 (162 billion) URLs. 1,135,327,420 (1.1 billion) subdomains. 194,346,505 (194 million) root domains. 1,168,465,575,815 (1.1 Trillion) links. Followed vs nofollowed links 2.84% of all links found were nofollowed 65.80% of nofollowed links are internal 34.20% are external Rel canonical: 28.89% of all pages employ the rel=canonical tag The average page has 92 links on it 76 internal links on average. 16 external links on average.. Go have fun with your new data! PS - For any questions about DA/PA fluctuations (or non-fluctuations) check out this Q&A thread from Rand: https://moz.com/community/q/da-pa-fluctuations-how-to-interpret-apply-understand-these-ml-based-scores
API | | IanWatson5 -
Mozscape API - Keyword Rankings?
Hi, I'm using the free access to the Mozscape API and while I'm still a novice about what data the Mozscape API can pull I can't seem to figure how to pull the keyword rankings for my campaigns. I reviewed https://moz.com/help/guides/moz-api/mozscape/api-reference/url-metrics but can't seem to find what value I need to call. Can someone guide me in the right direction or is this something only the paid version has access to?
API | | FPK0 -
What happens if I go over my Mozscape api free limits?
Hello,
API | | FPK
I just started using the free version of Mozscape and I fully understand there are limits and charges under this category. However to avoid any costly surprises, I like to know:
What happens when I get near my usage limit?
What happens when I just hit the limit?
What happens when I past the limit? Along with my questions, Is there any alert systems to let me know when I get to the range of said limit, like an email?0 -
10/14 Mozscape Index Update Details
Howdy gang, As you might have seen, we've finally been able to update the Mozscape index after many challenging technical problems in the last 40 days. However, this index has some unique qualities (most of them not ideal) that I should describe. First, this index still contains data crawled up to 100 days ago. We try to make sure that what we've crawled recently is stuff that we believe has been updated/changed, but there may be sites and pages that have changed significantly in that period that we didn't update (due to issues I've described here previously with our crawlers & schedulers). Second, many PA/DA and other metric scores will look very similar to the last index because we lost and had problems with some metrics in processing (and believe that much of what we calculated may have been erroneous). We're using metrics from the prior index (which had good correlations with Google, etc) until we can feel confident that the new ones we're calculating are correct. That should be finished by the next index, which, also, should be out much faster than this one (more on that below). Long story short on this one - if your link counts went up and you're seeing much better/new links pointing to you, but DA/PA remain unchanged, don't panic - that's due to problems on our end with calculations and will be remedied in the next index. Third - the good news is that we've found and fixed a vast array of issues (many of them hiding behind false problems we thought we had), and we now believe we'll be able to ship the next index with greater quality, greater speed, and better coverage. One thing we're now doing is taking every URL we've ever seen in Google's SERPs (via all our rank tracking, SERPscape, the corpus for the upcoming KW Explorer product, etc) and prioritizing them in Mozscape's crawl, so we expect to be matching what Google sees a bit more closely in future indices. My apologies for the delay in getting this post up - I was on a plane to London for Searchlove - should have got it up before I left.
API | | randfish4 -
August 3rd Mozscape Index Update (our largest index, but nearly a monthly late)
Update 5:27pm 8/4 - the data in Open Site Explorer is up-to-date, as is the API and Mozbar. Moz Analytics campaigns are currently loading in the new data, and all campaigns should be fully up-to-date by 4-10pm tomorrow (8/5). However, your campaign may have the new data much earlier as it depends on where that campaign falls in the update ordering. Hey gang, I wanted to provide some transparency into the latest index update, as well as give some information about our plans going forward with future indices. The Good News: This index, now that it's delivered, is pretty impressive. Mozscape's August index is 407 Billion URLs in size, nearly 100 Billion (~25%) bigger than our last record index size. We indexed 2.18 trillion links for the first time ever (prior record was 1.54 trillion). Correlations for Page Authority have gone up from 0.319 to 0.333 in the latest index, suggesting that we're getting a slightly more accurate representation of Google's use of links in rankings from this data (DA correlations remain constant at 0.185) Our hit ratio for URLs in Google's SERPs has gone up considerably, from 69.97% in our previous index to 78.66% in the August update. This indicates we are crawling and indexing more of what Google shows in the search results (a good benchmark for us). Note that a large portion of what's missing will be things published in the last 30-60 days while we were processing the index (after crawling had stopped). The Bad News: August's index was late by ~25 days. We know that reliable, consistent, on-time Mozscape updates are critically important to everyone who uses Moz's products. We've been working hard for years to get these to a better place, but have struggled mightily. Our latest string of failures was completely new to the team - a bunch of problems and issues we've never seen before (some due to the index size, but many due to odd things like a massive group of what appear to be spam domains using the Palau TLD extension clogging up crawl/processing, large chunks of pages we crawled with 10s of thousands of links which slow down the MozRank calculations, etc). While there's no excuse for delays, and we don't want to pass these off as such, we do want to be transparent about why we were so late. Our future plans include scaling back the index sizes a bit, dealing with the issues around spam domains, large link-list pages, some of the odd patterns we see in .pl and .cn domains, and taking one extra person from the Big Data team off of work on the new index system (which will be much larger and real-time rather than updated every 30 days) to help with Mozscape indices. We believe these efforts, and the new monitoring systems we've got will help us get better at producing high quality, consistent indices. Question everyone always asks: Why did my PA/DA change?! There are tons of reasons why these can change, and they don't necessarily mean anything bad about your site, your SEO efforts, or whether your links are helping you rank. PA and DA are predictive, correlated metrics that say nothing about how you're actually performing. They merely map better than most metrics to Google's global rankings across large SERP sets (but not necessarily your SERPs, which is what you should care about). That said, here's some of the reasons PA/DA do shift: The domains/pages with the highest PA/DA scores gain even faster than most of the domains below them, making it harder each index to get higher scores (since PA/DA are on a logarithmic scale, this is smoothed out somewhat - it would be much worse on a conventional scale, e.g. Facebook.com 100, everyone else 0.0003). Google's ranking algorithm introduces new elements, changes, modifies what they care about, etc. Moz crawls a set of the web that does or doesn't include the pages that are more likely to point to a given domain than another. Although our crawl tends to be representative, if you've got lots of links from deep pages on less popular domains in a part of the web far from the mainstream, we may not consistently crawl those well (or, we could overcrawl your sector because it recently received powerful links from the center of the web). My advice, as always, is to use PA/DA as relative scores. If your scores are falling, but your competitors' are falling more, that's not a bad thing. If your scores are rising, but your competitors' are rising faster, they're probably gaining ground on you. And, if you're talking about score changes in the 1-4 points range, that's not necessarily anything but noise. PA/DA scores often shift 1-4 points up or down in a new index so don't sweat it! Let me know if you've got more questions and I'll do my best to answer. You can also refer to the API update page here: https://moz.com/products/api/updates
API | | randfish8