Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
301 Redirects: How long for Google to recognize? How long for Moz/OSE to recognize?
-
I redirected a few pages that had funky URL issues (capitals and underscores and useless words) to the same page renamed with proper URLs. The sitemaps were changed also, and re-submitted.
Like this: mydomain.com/Not_Pertinent_Words.html >>> mydomain.com/good-words.html
Google seems to have found them and changed the search engine results listing in about 8 days.
But it's been about a month now and Moz and OSE still have not transferred all the strength and link data from the old URLs to the new ones.
Question 1: How long does it usually take Google to transfer all the link and strength data for a 301?
Question 2: How long does it take Moz and OSE to do the same? Is there something I need to do to tell them about the changes?
-
I had a similar issue with usedcars.com. We had several hundred thousand URLs that were redirecting from parameter driven URLs, to clean hierarchical URLs with a huge initiative to clean them all up in a few months. It took well over a year and still didn't see them getting recognized.
I blogged the process - but the part about what I did to speed it up:
I grabbed one of my many impulsive purchased domains and quickly set up hosting and an old school html site that consisted of one page. I then exported all of the links on the Google "site:" search through a Firefox plugin called SEOquake that exports the results into a csv file. It's not the prettiest, and there was a lot of work still needed to get to just the URLs, but it was the best solution I could find (note: if any SEO reading this knows of an easier way to do this - please add to the comments for prosperity). I then parsed out the parameters in the URLs in a separate document and used those as the anchor text for each URL. Finally, using excel I then concatenated the URLs and parameters (that were now anchor text) into an html href string.
Then copying and pasting the "string" column into the html code, the page looked like:
The page wasn't the prettiest, and it had thousands of links (the above is just an example) so it was bad all around, but the point was to get those links crawled by Google.Of course every SEO knows that you can't just build a website and expect it to immediately get crawled - right?
So I set it up in Google Webmaster Tools and submitted the page to the the index:
I even got more fancy to ensure Google would see the page and crawl all of those old URLs and +1'd it on Google.Did it work?
I checked the URLs this evening to see how many Google is seeing and the number has dropped from 550,000 to now only 175.
I took the domain off of the server, and now have it parked elsewhere (back where it belongs) and removed the webmaster tools account. All traces of it ever existing are now gone, and the small moment of my attempt to get those URLs removed has passed.
It's very black hat - and could potentially get you in trouble if not approached carefully. But - it worked...
-
Hi Gregory,
Great suggestion. It might be a little challenging to mix the private crawl (Your campaign) with the public one (OSE) but it's certainly possible to cross-pollinate data.
In the meantime, we're working on better incorporating our "just discovered links" in OSE to be included in the main index. This means that anytime a link is tweeted (our source of discovery for Just Discovered Links) it will be included for crawling in the general index.
Regardless, thanks for your input into this matter, and I apologize for it being a frustrating experience.
Cheers,
Cyrus
-
Hi Greg
So our new index went up today. Looking at the last crawl date for www.easydigging.com we crawled the domain on 9/27 but did not crawl deep enough to find the original page containing the 301. So it may be a couple of more indexes before we crawl deeper to find it. What you would need to do help www.easydigging.com increase it's visibility/domain authority which will result in OSE finding more pages which will eventually find the link we need. The current page is no longer in our index as it not longer has any crawl authority and pretty soon it will not show any data in OSE unless it is re-crawled.
Once we can include "Just Discovered Links" into the main index, as Cyrus mentioned above, all you would need to do then is have any link tweeted from a high ranking account which will basically guarantee a crawl to that page.
-
I don't have documentation, but the new page it redirects to has this tag:
So I would guess I did the 301 redirect the to this new page at the same time I launched the new page (there was no major design changes - just shortening the URL and getting rid of the capitals and underscores)
-
Hi Greg
Would you be able to tell me when the 301 was added for http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html?
-
Hi Greg
We do set the expectations about how long it would take for us to process pages from our help hub:
Here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 65,000,000,000 pages (which is about 25% the amount in Google's index).
- Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important.
So it's very possible that it may be re-crawled with an index we are possibly updating today ahead of schedule. Like I mentioned earlier, it can take a while for metrics to transfer over as it isn't guaranteed that we will crawl the same page for each index update.
-
Thanks David. That does explain the issue better.
However, the occasional really long lag time for OSE to re-crawled pages that have been 301 redirected is still a problem for those of use who subscribe to Moz and are trying to use it to monitor and improve our SEO. But I think there is an easy solution...
Give Moz subscribers a new tool like GWT's "Fetch as Google" - but call it "Fetch as Moz" and have it trigger BOTH the Moz spider and the OSE spider to go re-crawl that page. This would let us quickly and positively know that our data for 301'd pages has been transferred to the new page.
Can this be done?
-
Hi Greg
It can take a while for all of your pages to be picked up in new indexes as I mentioned earlier. OSE will actually check for re-directs immediately but for the example page you provided, that page was last crawled on September 8th which did not have the re-direct at that time. It is not guaranteed that the same pages will be re-crawled during the next index updates which is why our metrics from seomoz.org moved to moz.com took a few months. OSE does not crawl pages in a hierarchal manner unlike our other crawler for diagnostics which follows links within the source code.
OSE needs a reason to re-crawl new and old pages so it can see the re-direct. We take the last 40 days of crawl data from the last Index if a page hasn't been re-crawled. We actually are about to release a new index today ahead of schedule so it's possible we may be able to re-crawl the old page. After I confirm a new index is complete I will re-check the old URL to see if OSE picked it up.
-
Hi David,
Sorry to be blunt here, but no it doesn't make sense - and I also think it is incorrect...
Websites do internal 301 redirects all the time (redirecting from old page to new page within the same site) BUT your explanation above appears to say that OSE does not follow those redirects and move the data from old to new url ? That would make for a pretty useless SEO program...
Your message also appears to say that if we have to go contact everybody that links to www.mysite.com/old-page.html and have them change their link to www.mysite.com/new-page.html ? That's just silly...
I think your explanation is incorrect because SOME of the 301 re-directs I have done on my site have passed the data perfectly in OSE and Moz. Run OSE on these two URLs and you will see: Old URL = www.bestdryingrack.com/clothes_drying_rack.html New URL it was 301'ed to = www.bestdryingrack.com/clothes-drying-rack-main.html
So it appears that inconsistency is the a big problem here. That should be some firm rule like: "Moz will transfer all data across 301 redirects on the same domain within 1 month" This would let us analyze or sites without trying to keep track of silly situations of which new pages have the proper data and which ones we still need to fudge between old and new page data.
I hope this all makes sense...
-
Hi Greg
Our last index update was on the 23rd and the next one is not scheduled until Nov 20th. Keep in mind that not all metrics will carry over, we will still index the site based on strong backlinks. For external links you would need to make sure the linking root domains linking the old pages update their links with the new ones as our crawler does not follow links and only sees what they are as is.
So what our index will eventually update/bring over:
Internal Followed Links
Total Internal LinksUpdating the linking root domains with the new links will help with the external links.
Let me know if that makes sense!
-
Hi David,
I waited another week and just now checked the new and redirected URL again. The data STILL has not transferred over to the new URL.
It's been a long time. This excessive lag time in moving data across re-directs really needs to be fixed.
Personally I can mentally keep track of it, but what about the other SEO programs out there that are using Moz data? They have to experiencing "garbage in - garbage out" problems on any analysis involving re-directed URL's.
Can you check into this and let us know what's up?
-
Hello David, Thanks for jumping in!
The old URL was http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html and the new one it 301's to is http://www.easydigging.com/wheel-hoe-hoss.html Please reply back here so I know what the situation is.
Since it appears that Index updates only happen monthly, and it takes a few months for data to fully migrate over to redirected URLs, it would be really nice to change the little message that shows on OSE when an analyzed page is redirected (currently something like "Do you want to see the data for page this redirects to") to something that describes the lag, like.. "It takes a few months for Moz data to transfer to redirected URLs, so be sure to check both the old and the new URL since the one with higher statistics is the accurate one"
-
Hi Greg
When setting up your re-direct not all metrics will transfer over to the new domain. It took us a few index updates to see most of our metrics to appear after we moved from seomoz.org to moz.com.
Would you be able to provide us with the URLs of the old site and the new site so we can make sure our crawler is picking up the 301s?
If you do not wish to share them here you can submit your information through our contact form:
-
Hi again Chris. Mostly I was wanting to use Moz to evaluate my site for SEO reasons and frustrated that I kept having to use "old" page URLs to get data.
Also I have just started to experiment with the Penguin Analysis tool from Virante, and it uses Moz data mixed with data from other SEO programs - so I get corrupt results there whether I input the old URL or the new URL. http://www.penguinanalysis.com/
If they really updated the Mozscape index on the 18th, it seems it should show by now...
-
I wouldn't worry about the Mozscape index not showing your changes. I put up a client site on the 8th and, despite thousands of visitors and some very strong backlinks, there is no data for it on the OSE. Like I said, try Majestic, their tool does show this recent client site and the link data is accurate, so it may display more accurate citation and trust flow (Majestic authority metrics) for your site after the redirect than OSE.
-
Thanks Chris. But unfortunately that makes the situation even worse. I did these 301 redirects weeks ago, and your quote say that there was a Mozscape Index update was a few days ago on October 18th.
I just checked again in OSE and the old page data and strength still has not transferred over to the new page. I have attached a screenshot from OSE showing this.
Any ideas?
-
Question 1: If Google has correctly updated the index, I would imagine that it has redirected the link profile as well.
Question 2:
"Just-Discovered Links index updated continuously. Last Mozscape index update: October 18, 2013. Next Mozscape index update: November 15, 2013" From OSE.
If you're looking for results on a more recent site, try Majestic's Site Explorer. It is in my experience more accurate with newer sites and fresh redirects.
Hope this helps.
Chris
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is redirect notice?
I am newbie.while using moz,I put my competitor url and found some backlink.which included redirect notice .can someone explain me about it? image also adde.thanks view?usp=sharing
Link Explorer | | fikip463650 -
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
How is Moz DA affected by spam links? Disavow file?
So it does not appear that moz let's you upload your disavow file. So when moz calculates your DA how do spammy links factor in? After digging through our GA it appears our site was hit with the 2016 penguin update and never recovered. Our weekly visitors were 2k, then dropped to 500 and have stayed close to that level for a while. We've used the disavow tool, without success over the past 3 years. During that time we have done link out reach and built around 10 legit good quality DA links since. But we have not recovered. At this point i'm thinking I should just remove the disavow file. Moz says our spam score for our domain is 5%.
Link Explorer | | jessicapremier0 -
Moz's new Link Explorer, including our revamped index and DA/PA scores is now open to everyone!
Hey Moz Community, Link Explorer is now open to the public! Everyone can access it via a subscription or a free Moz ‘Community’ account. As you may know by now, the brand-new Link Explorer tool is primed to replace Open Site Explorer as Moz’s link building and analysis tool. The Link Explorer project is the result of an incredible amount of perseverance and hard work by the team, and we’re proud to be able to finally share it with you — we know it’s going to revolutionize how you approach link building and make your job easier. You can read more about the tool here in Sarah Bird’s announcement post. Because Link Explorer improves on almost every aspect of Open Site Explorer, the metrics have improved, too. That means you’re likely going to see some Domain Authority and Page Authority discrepancies between OSE’s index and Link Explorer’s index. We definitely suggest you use the new DA/PA from Link Explorer, as they’re more accurate and refresh daily rather than monthly, as was the case with OSE’s index. However, we also realize that many of you use these metrics to report to your clients and colleagues, and a sudden change or fluctuation could potentially make your job harder. Which DA is the real DA? The new DA is based on a much larger index that has many improvements, several of which are designed to make the index more like Google’s than ever before. You should consider moving towards the new DA (and the old DA won’t be updated after April 26th 2018, so the sooner the better). While there will be fluctuations as we improve the model and add features to the index, we expect it to remain largely stable and to be a far more accurate picture of a site’s authority according to how it’s seen by Google. Why is Link Explorer’s DA/PA considered better than OSE’s, and which should I trust? The larger link index with improved crawl selection allows us to produce a stronger model that includes a much larger proportion of the web. That being said, DA and PA should always be considered in the context of your competitors. A drop in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Is Domain Authority/Page Authority an absolute score or a relative one? Both DA and PA are relative to the Internet as a whole. If Facebook acquired a billion new links, everyone’s PA and DA would drop relative to Facebook. Because of this, it’s always best to look at PA and DA in comparison to your competitors. What does a drop/raise in DA mean in Link Explorer vs OSE? How can I explain this to my clients when I’m reporting it? DA and PA should always be considered in the context of your competitors. A drop or raise in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Reporting that your site has moved from a DA of 45 to a DA of 42 doesn’t tell the whole story, but reporting that your site has a DA of 42 while your main competitor moved from a 43 to a 37 shows that, relative to the sites you’re competing against in the SERPs, your site has significantly more authority and ranking power. What’s happening to MozTrust and MozRank and why, and what should I replace those with? The improvements to our DA/PA and Spam Score metrics now now account for more important nuances in helping you determine one site’s ability to rank higher than another. Because they no longer correlate with Google’s ranking model as well as they used to, MozRank and MozTrust are being deprecated for better metrics. Users should rely on Page Authority, Domain Authority, and Spam Score to determine the importance and quality of pages, domains, and links. I have historical data I use to help my clients benchmark their progress. What do I do now that DA is calculated differently? You should annotate any KPI changes referencing the change in DA and PA. However, most importantly, you should compare those changes to your competitors, as this will best show how strong your site’s authority is relative to the sites you’re competing against in the SERPs. We take updating our metrics very seriously, and our last major update to the model was 7 years ago. Users of Domain Authority and Page Authority can expect us to continue to produce steady, reliable metrics for the long haul, and only make changes to these metrics when we believe the benefits dramatically outweigh the stability of the metric. Do you have any questions about the new metrics? Anticipating a tough time reporting changes to clients or bosses? Metrics, features or functionality missing that you would want to see? Let us know in the thread, and we’ll work to find a good answer for you. Hope you enjoy the new Link Explorer product and the amazing new link index powering it. We are very excited to provide this valuable data to our community and customers.
Link Explorer | | IanWatson9 -
DA/PA Fluctuations: How to Interpret, Apply, & Understand These ML-Based Scores
Howdy folks, Every time we do an index update here at Moz, we get a tremendous number of questions about Domain Authority (DA) and Page Authority (PA) scores fluctuating. Typically, each index (which release approximately monthly), many billions of sites will see their scores go up, while others will go down. If your score has gone up or down, there are many potential influencing factors: You've earned relatively more or less links over the course of the last 30-90 days.
Link Explorer | | randfish
Remember that, because Mozscape indices take 3-4 weeks to process, the data collected in an index is between ~21-90 days old. Even on the day of release, the newest link data you'll see was crawled ~21 days ago, and can go as far back as 90 days (the oldest crawlsets we include in processing). If you've done very recent link growth (or shrinkage) that won't be seen by our index until we've crawled and processed the next index. You've earned more links, but the highest authority sites have grown their link profile even more
Since Domain and Page Authority are on a 100-page scale, the very top of that represents the most link-rich sites and pages, and nearly every index, it's harder and harder to get these high scores and sites, on average, that aren't growing their link profiles substantively will see PA/DA drops. This is because of the scaling process - if Facebook.com (currently with a DA of 100) grows its link profile massively, that becomes the new DA 100, and it will be harder for other sites that aren't growing quality links as fast to get from 99 to 100 or even from 89 to 90. This is true across the scale of DA/PA, and makes it critical to measure a site's DA and a page's PA against the competition, not just trended against itself. You could earn loads of great links, and still see a DA drop due to these scaling types of features. Always compare against similar sites and pages to get the best sense of relative performance, since DA/PA are relative, not absolute scores. The links you've earned are from places that we haven't seen correlate well with higher Google rankings
PA/DA are created using a machine-learning algorithm whose training set is search results in Google. Over time, as Google gets pickier about which types of links it counts, and as Mozscape picks up on those changes, PA/DA scores will change to reflect it. Thus, lots of low quality links or links from domains that don't seem to influence Google's rankings are likely to not have a positive effect on PA/DA. On the flip side, you could do no link growth whatsoever and see rising PA/DA scores if the links from the sites/pages you already have appear to be growing in importance in influencing Google's rankings. We've done a better or worse job crawling sites/pages that have links to you (or don't)
Moz is constantly working to improve the shape of our index - choosing which pages to crawl and which to ignore. Our goal is to build the most "Google-shaped" index we can, representative of what Google keeps in their main index and counts as valuable/important links that influence rankings. We make tweaks aimed at this goal each index cycle, but not always perfectly (you can see that in 2015, we crawled a ton more domains, but found that many of those were, in fact, low quality and not valuable, thus we stopped). Moz's crawlers can crawl the web extremely fast and efficiently, but our processing time prevents us from building as large an index as we'd like and as large as our competitors (you will see more links represented in both Ahrefs and Majestic, two competitors to Mozscape that I recommend). Moz calculates valuable metrics that these others do not (like PA/DA, MozRank, MozTrust, Spam Score, etc), but these metrics require hundreds of hours of processing and that time scales linearly with the size of the index, which means we have to stay smaller in order to calculate them. Long term, we are building a new indexing system that can process in real time and scale much larger, but this is a massive undertaking and is still a long time away. In the meantime, as our crawl shape changes to imitate Google, we may miss links that point to a site or page, and/or overindex a section of the web that points to sites/pages, causing fluctuations in link metrics. If you'd like to insure that a URL will be crawled, you can visit that page with the Mozbar or search for it in OSE, and during the next index cycle (or, possibly 2 index cycles depending on where we are in the process), we'll crawl that page and include it. We've found this does not bias our index since these requests represent tiny fractions of a percent of the overall index (<0.1% in total). My strongest suggestion if you ever have the concern/question "Why did my PA/DA drop?!" is to always compare against a set of competing sites/pages. If most of your competitors fell as well, it's more likely related to relative scaling or crawl biasing issues, not to anything you've done. Remember that DA/PA are relative metrics, not absolute! That means you can be improving links and rankings and STILL see a falling DA score, but, due to how DA is scaled, the score in aggregate may be better predictive of Google's rankings. You can also pay attention to our coverage of Google metrics, which we report with each index, and to our correlations with rankings metrics. If these fall, it means Mozscape has gotten less Google-shaped and less representative of what influences rankings. If they rise, it means Mozscape has gotten better. Obviously, our goal is to consistently improve, but we can't be sure that every variation we attempt will have universally positive impacts until we measure them. Thanks for reading through, and if you have any questions, please leave them for us below. I'll do my best to follow up quickly.13 -
OSE results vs. Google
Is OSE still a worthy analyzer of the factors breaking down Google rank of a page or website? More so over the past six months, I've seems to noticed how the page comparison in OSE seems to have less and less correspondence to Google rank. For instance, I'm analyzing sites where the top result has a Page Authority rank of 11, Subdomain MozRank of 3.53, outranking a page coming in in spot #14 with Page Authority of 36, Subdomain MozRank of 3.94. I see this frequently, that by the factors measured in OSE, a certain page with favorable scores is being outranked by a page with poor OSE scores. Of course, I realize how sophisticated Google's algorithm is and the OSE can't reverse engineer the whole thing. Also, there are lots of other factors, such as personalized search results, banklink velocity, and who's in my G+ circles, that isn't taken into account in OSE. Yet, I see this more-and-more, where websites kicking ass in OSE measurements rank poor, and poor websites ranked in OSE come out near the top in Google. So, given this, where OSE is apparently a less effective tool to help analyze the factors building Google rank as Google evolves and values brand, what should a chap do? I already know the predictable answers of do RCS, build your community, and so on. But, I want to know what's driving the juice behind these top ranking websites, and OSE doesn't seem to be cutting it anymore... Your thoughts?
Link Explorer | | ExploreConsulting0
