Google crawl rate almost zero since re-launch, organic search up 50% though!
-
We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png
-
Hi David,
First, my preference would always be to use 301 redirect to send pages that no longer exist to a new URL.
Using a 301 passes link juice and signals search engines that the old page should be de-indexed and the new page indexed in its place.
Barring some issue with your rel=canonical tags, which would be the first place to look, I would check other things that can send signals to the search engines about freshness:
- meta refresh tags
- update frequency reported in your sitemap
- custom crawl rate setting in Webmaster Tools
Then of course, there is an analysis of the differences between your old design and the new...have you made a significant change to your menu(s) and/or internal linking structure which has made it harder for crawlers to follow links to all pages in your site?
Hope that helps,
Sha
-
It's the same domain but we've rebuilt the website and improved the url and content structure.
We've consolidated urls (e.g. redirecting to canonical url) and published the update xml sitemap.
Pages are crawled correctly, we just can't figure out why the graph changed like that.
-
Hi David,
Just a few questions to help understand your situation:
- When you say "new site" do you mean that you have changed to a new domain?
If it is a new domain:
- Did you 301 individual pages to relevant pages on the new domain, or do you have a catch-all sending all traffic to home?
- Is the chart you uploaded from the new Webmaster Tools account (for the new domain)?
- Have you created and uploaded xml sitemap(s) for the new site?
If it is not a new domain:
- Have you created and uploaded xml sitemap(s) for the new version of the site?
- Have you checked the crawl rate setting in the Webmaster Tools account in case it has been manually altered?
These answers will give us a clearer idea of where to start.
Thanks,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving from http to https - what do I need to do in Google Search Console?
Hi all, I have moved my site from http to https. I current have two profiles in Google Search Console: http://mysite.com
Technical SEO | | Bee159
http://www.mysite.com Do I need to set up the same but with https and if so, what do I then do with the http profiles? Do I delete them? Or just remove the sitemaps? Confused.0 -
PDF in search results?
Hello community! I am not an SEO professional, though I am a practitioner, I would say. I am seeking a solution on behalf of a friend. If you search the term "Peter Blatt" you will discover a "black eye" on the first page, towards the bottom of SERPs. It's a PDF published on the Florida Department of Financial Services website regarding the final order for a settlement he and his company ("Blatt Financial Group") reached with the state as it related to professional conduct allegations. Does anyone have any advice on how to address this? I don't want "game" the search engines, but at the same time, this document looks really scary and much worse than it actually is to people, and I would love for it do drop below page one. Any advice or suggestions from the community? Thanks! Tom
Technical SEO | | 800GoldLaw0 -
Local Google vs. default Google search
Hello Moz community, I have a question: what is the difference between a local version of Google vs. the default Google in regards to search results? I have a Mexican site that I'm trying to rank in www.google.com.mx, but my rankings are actually better if I check my keywords on www.google.com The domain is a .mx site, so wouldn't it make more sense that this page would rank higher on google.com.mx instead of the default Google site, which in theory would mean a "broader" scope? Also, what determines whether a user gets automatically directed to a local Google version vs. staying on the default one? Thanks for your valuable input!
Technical SEO | | EduardoRuiz0 -
Google bot notification
Hi there! I've just made some changes in my website in order to optimize it but I don't know if there's a way to notify the googlebot that some aspects of the configuration (metas) have changed and must be "taken into account". The spider visited my site two days ago and obviously processed the sitemap file. I've heard that it's possible to do a ping to certain websites. Is this the way to proceed? I must say that there're not many updates in the site (just one way information) as the social media activity is still low. Thanks in advanced.
Technical SEO | | juanmiguelcr0 -
HELP: Wrong domain showing up in Google Search
So i have this domain (1)devicelock.com and i also had this other domain (2)ntutility.com, the 2nd domain was an old domain and it is not in use anymore. But when i search for devicelock on Google, the homepage devicelock.com does not exist. Only ntutility.com comes up. I asked one of the developer how the redirect is happening from the old domain to the new one and he told me its through a DNS forward. And there is no way to have an .htacess file to set up a 301 instead. Please help!
Technical SEO | | Devicelock0 -
How long will Google take to stop crawling an old URL once it has been 301 redirected
I need to do a clean-up old urls that have been redirected in sitemap and was wondering about this.
Technical SEO | | Ant-8080 -
New page titles not updating in Google results even though recrawlled
I have updated the page title of a website - it's a top product page and very high authority (64 PA on a 63 DA). The page has been recrawlled and the cache updated - but the page tiltle in the Google snippet is still the same. Any ideas?
Technical SEO | | OddDog0 -
Is it possibly to use anything besides a 302 re-direct when your doing a re-direct for someone to login?
Hopefully this makes sense. So I am working on a site that uses a 302 re-direct for logins. As in it goes from a profile page to the login via a re-direct, most of the time I see sites use this as a meta refresh, but in this case I wasn't sure. Obviously when I run a crawl diagnostic I'm getting a lot of errors as in over 100. Now I know there is no link juice with this, but I was just wondering what other people thought on using 302's for logins? Thanks
Technical SEO | | kateG12980