New server update + wrong robots.txt = lost SERP rankings
-
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).
When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.
The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
-
Dr. Pete,
I just ran across one of your webinars yesterday and you brought up some great ideas. Earned a few points in my book
Too often SEOs see changes in the rankings and react to counter-act the change. Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them. If your changes were positive you should see positive results. I have rarely found an issue where a user made a positive change and got a negative result from Google. Patience is a virtue.
-
Thanks everyone for the help! Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back. I think the sitemap and fixed robots.txt were the most important factors.
-
I agree, let Google re-index first and then re evaluate the situation.
-
I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.
A few minor cues that are safe:
(1) Re-submit your XML sitemap
(2) Build a few new links (authoritative ones, especially)
(3) Hit social media with your new URLs
All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.
-
Remain calm. You should be just fine. It just takes time for Google to digest the new robots.txt. I would be concerned if things didn't change in 3-4 weeks. Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve. Sometimes Google moves things around and this is natural.
If you want Google to crawl your site faster, build some links and do some social media. That will encourage Google to speed it up.
-
If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.
-
That does not sound fun at all.... So you just changed the server, complete copy?
My first question would be other than the server did anything else change? Copy or URL's
My second question would be is the other server still up and live to the internet?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to rank for a keyword
Hi guys, I'm looking to rank a new blog for a search term which we currently already rank #1 for. I want to create a blog which provides a better solution to a searchers query and knock our current #1 page for this new one. Is there a way to do this simply without losing the real-estate the currently ranked #1 page has already accumulated? Or is just a matter of working on this new blog to find it's own way to the top? Thanks in advance, James
Intermediate & Advanced SEO | | jameseacott0 -
Phasing in new website - new content on www2
Hi Mozzers, I'm working on a large website redesign / redevelopment project. New sections of the website will be phased in over the next 12 months. The plan is to launch all new content on a subdomain (www2.domain.com) while the old site remains on www.domain.com. There will be no duplicate content across the www and www2 sites, as old content will be removed on www as it is replaced with new content on www2. 301 redirects will also be setup from old content on www to new content on www2. Once the new site on www2 is complete, everything will be moved to www, with a robust 301 redirect setup in place. Is this approach logical, and can you see any SEO implication for managing the migration in this way? Thanks!
Intermediate & Advanced SEO | | RWesley0 -
Robots.txt gone wild
Hi guys, a site we manage, http://hhhhappy.com received an alert through web master tools yesterday that it can't be crawled. No changes were made to the site. Don't know a huge amount about the robots.txt configuration expect that using Yoast by default it sets it not to crawl wp admin folder and nothing else. I checked this against all other sites and the settings are the same. And yet 12 hours later after the issue Happy is still not being crawled and meta data is not showing in search results. Any ideas what may have triggered this?
Intermediate & Advanced SEO | | wearehappymedia0 -
Robots.txt - blocking JavaScript and CSS, best practice for Magento
Hi Mozzers, I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento. I'm concerned we are blocking bots from crawling essential information for page rank. My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not? You can view our robots.txt file here Thanks, Blake
Intermediate & Advanced SEO | | LeapOfBelief0 -
Huge Dip in Traffic Last Week - New Algo Update?
Hi Mozzers, We experienced a huge dip in traffic on Thursday, 8/14, across our entire site. It was not a specific set of pages, it was sitewide. Google Webmaster Tools notes our impressions are down as well. The traffic has not recovered. It appears our pages are still indexed in Google, just not ranking well. Here are some questions I have to help isolate the cause: We recently completed a major redesign of our entire website on 7/26. We did not notice any dip in traffic after the new design launch - in fact, it actually increased a bit. Is it possible that only now Google sees our new site design and this is the reason for our dip? Is there a way to see Google's past cache dates? Did anyone else experience a similar dip in traffic since Thursday? Was there a recent Google update? It would be much appreciated if someone takes a look at our site - www.consumerbase.com for any glaring SEO errors (missing necessary meta tags, etc.). What steps do you guys suggest I take to isolate the cause in this dip in traffic? Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
Not ranking in Google - why???
This will be a bit long, so please bare with me. I have a client in the auto parts industry who wants to rank their homepage for 13 different keywords. We are ranked first page for all keywords in Yahoo! Mexico and Bing Mexico, but not ranking first page at all in Google Mexico. My client's competitor, however, is clearly outranking my client in Google. When comparing both pages, my client's, while not 100% optimized, looks better optimized than their competitor's. Looking at all metrics using Moz, SEMRush, ahrefs, etc... my client's site looks MUCH better on all fronts. I know ranking a single homepage for more than 10 keywords is a difficult task. Our competitor is however, ranking for them, so it's not impossible. The keywords are not even that competitive according to Moz's analysis. I decided to create an optimized page for each keyword to try to rank these pages, but still my client wants the homepage to rank (again, if the competitor is ranking, then it's possible to do this) and I am afraid these pages I created could result in keyword cannibalization ultimately affecting the homepage's possibility to rank. My client had a previous SEO agency working for them and basically all they did was create fake blogs and have lots of keyword rich links directed to the site's homepage. I got the complete link profile from several tools and submitted a disavow requests for as many fishy links I could find, but that hasn't shown any results so far. Note: when looking at the competitor link profile, they have basically just a few links and no external links of real value whatsoever. My client is obviously very frustrated, and so am I. In my SEO experience, it shouldn't be such a difficult task to accomplish, however nothing seems to work even though everything seems to point that my client should rank higher. So now I'm running out of ideas regarding what to do with this site. Any insight you could provide would be SO helpful to me and my client. If needed I can provide my client's homepage URL and also their competitors homepage for you to review. i can also give you any extra information you need. Thanks a lot!
Intermediate & Advanced SEO | | EduardoRuiz0 -
Stupid Question?? Is [painter new york] the same keyword as [painter in new york]?
Hi, This may be a stupid question but... Google ignores short/common words like 'in', so if I optimize a page for 'painter in new york' will it rank just as well for 'painter new york'? In Google's keyword tool, exact match gives [painter new york] 140 searches per month but [painter in new york] gets < 10. However, it is much more difficult to write 'painter new york' naturally into body copy than it is 'painter in new york'. So what do I do? Thanks 🙂
Intermediate & Advanced SEO | | StrayCat0 -
Mozpoints not updating
Added 2 comments on 2 questions and my points are still at the same level. Anyone knows why? I don't.
Intermediate & Advanced SEO | | mosaicpro0