Google crawl rate almost zero since re-launch, organic search up 50% though!
-
We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png
-
Hi David,
First, my preference would always be to use 301 redirect to send pages that no longer exist to a new URL.
Using a 301 passes link juice and signals search engines that the old page should be de-indexed and the new page indexed in its place.
Barring some issue with your rel=canonical tags, which would be the first place to look, I would check other things that can send signals to the search engines about freshness:
- meta refresh tags
- update frequency reported in your sitemap
- custom crawl rate setting in Webmaster Tools
Then of course, there is an analysis of the differences between your old design and the new...have you made a significant change to your menu(s) and/or internal linking structure which has made it harder for crawlers to follow links to all pages in your site?
Hope that helps,
Sha
-
It's the same domain but we've rebuilt the website and improved the url and content structure.
We've consolidated urls (e.g. redirecting to canonical url) and published the update xml sitemap.
Pages are crawled correctly, we just can't figure out why the graph changed like that.
-
Hi David,
Just a few questions to help understand your situation:
- When you say "new site" do you mean that you have changed to a new domain?
If it is a new domain:
- Did you 301 individual pages to relevant pages on the new domain, or do you have a catch-all sending all traffic to home?
- Is the chart you uploaded from the new Webmaster Tools account (for the new domain)?
- Have you created and uploaded xml sitemap(s) for the new site?
If it is not a new domain:
- Have you created and uploaded xml sitemap(s) for the new version of the site?
- Have you checked the crawl rate setting in the Webmaster Tools account in case it has been manually altered?
These answers will give us a clearer idea of where to start.
Thanks,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Website disappeared from Google Search Results overnight
Hello there, I'm the owner of the Website https://cours-toujours.com/, dedicated to reviewing running shoes. My Website is pretty young and I'm currently focused on building new reviews (so I keep adding new articles, week after week, I did not really focus on the rest of the website for now).Until a few days ago, I saw growing traffic on my Website, everything seemed good and I kept adding new reviews to my website.And then suddently traffic dropped and went to 0 in 2 days (I went from 550 impressions/day to 49 impressions/day in 2 days :/)When I look in the Google Search Console, I don't see any issue: my sitemaps are submitted and the correct number of URLs are reported I don't have any Manual Action or Security Issue I don't have any Removal Request Everything seems fine... But I can barely find my website in Google Search Results.When I do a site search (site:cours-toujours.com), I find only 2 pages of results, mostly non-important pages (categories, etc.).I asked in Google Community Forums, and i got this reply about my pages being too similar to one another (https://support.google.com/webmasters/thread/44880689?hl=en). But I'm not really happy with this answer, as all my pages have ~1000 words of unique content (even if of course they have the same structure as they are all dedicatd to reviewing a running shoe...)Any idea where this might come from/how I can fix the issue?
Technical SEO | | SimonCoursToujours0 -
I added a WP Customer Reviews plugin but nothing seems to appear on Google search
Hi, I've added the wordpress Wp Customer Reviews plugin to a my client's website and we brought some past clients to put on reviews in order to empower the hReview factor. Google as scraped the website several times since but we don't see any change in the organic serp. Can you please tell me if I've done something wrong or I forgot something? That's the website - Capital Garage Door Thanks!
Technical SEO | | captainjoe0 -
I have a client whose shop will be down a few days. What would provide less impact to organic search program?
I have a client who is moving their warehouse, and their shop will be down for four days. I have been doing some research on the best ways to handle this and I wanted to get the communities feedback on this. One thought is to have the pages live, but people can't place an order - but this does not provide the best customer experience. Another thought is to just do temporary redirects for the shop pages, to land on the "sorry we are moving" page for customers. Another thought was to do 503 HTTP status codes on the pages and then do a temporary redirect to the landing page. Have any of you experienced this issue? If so, what did you do to minimize the impact to the organic search programs? NOTE: All of their static content will remain in tact. Only the shop/store will be down.
Technical SEO | | smulto0 -
Fixing Crawl Errors
Hi! I moved my Wordpress blog back in August, and lost much of my site traffic. I recently found over 1000 crawl errors in Webmaster Tools because some of my redirects weren't transferred, so we are working on fixing the errors and letting Google know. I'm wondering how long I should expect for Google to recognize that the errors have been fixed and for the traffic to start returning? Thanks! Jodi - momsfavoritestuff.com
Technical SEO | | JodiFTM0 -
Is Google caching date same as crawling/indexing date?
If a site is cached on say 9 oct 2012 doesn't that also mean that Google crawled it on same date ? And indexed it on same date?
Technical SEO | | Personnel_Concept0 -
My organic search results are down 16% since the Penguin update 4/24
Penguin has affected my search results down 16% When I look at my SEOmoz scan the only problem I see is "too many on page links" The problem is that my blog for each month is considered one page-eg august 2007 I wrote many blogs-the total on page links was 106-but that included all the blogs that were written in a month. The other problem area is duplicate content. I thought Penguin was after "link farming" which I do not do. Any advice how I can correct this? Brooke
Technical SEO | | wianno1680 -
Google not visiting my site
Hi my site www.in2town.co.uk which is a lifestyle magazine has gone under a major refit. I am still working on it but it should be ready by the end of this week or sooner but one problem i have is, google is not visiting the site. I took a huge gamble to redo the site, even though before the refit i was getting a few thousand visitors a day, i wanted to make the site better as i was getting google webmaster errors. But now it seems google is not visiting the site. for example i am using sh404sef and i have put friendly url in the site and on the home page it has its name and meta tag but when you look at google it is not giving the site a name. Also it has not visited the site since october 13th Can anyone advise how to encourage google to visit the site please.
Technical SEO | | ClaireH-1848860 -
Search Engine Blocked by Robot Txt warnings for Filter Search result pages--Why?
Hi, We're getting 'Yellow' Search Engine Blocked by Robot Txt warnings for URLS that are in effect product search filter result pages (see link below) on our Magento ecommerce shop. Our Robot txt file to my mind is correctly set up i.e. we would not want Google to index these pages. So why does SeoMoz flag this type of page as a warning? Is there any implication for our ranking? Is there anything we need to do about this? Thanks. Here is an example url that SEOMOZ thinks that the search engines can't see. http://www.site.com/audio-books/audio-books-in-english?audiobook_genre=132 Below are the current entries for the robot.txt file. User-agent: Googlebot
Technical SEO | | languedoc
Disallow: /index.php/
Disallow: /?
Disallow: /.js$
Disallow: /.css$
Disallow: /checkout/
Disallow: /tag/
Disallow: /catalogsearch/
Disallow: /review/
Disallow: /app/
Disallow: /downloader/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /utm
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Sitemap:0