Recovering from robots.txt error
-
Hello,
A client of mine is going through a bit of a crisis. A developer (at their end) added Disallow: / to the robots.txt file. Luckily the SEOMoz crawl ran a couple of days after this happened and alerted me to the error. The robots.txt file was quickly updated but the client has found the vast majority of their rankings have gone.
It took a further 5 days for GWMT to file that the robots.txt file had been updated and since then we have "Fetched as Google" and "Submitted URL and linked pages" in GWMT.
In GWMT it is still showing that that vast majority of pages are blocked in the "Blocked URLs" section, although the robots.txt file below it is now ok.
I guess what I want to ask is:
- What else is there that we can do to recover these rankings quickly?
- What time scales can we expect for recovery?
- More importantly has anyone had any experience with this sort of situation and is full recovery normal?
Thanks in advance!
-
Great info Rikki
thats goid news!
-
Hi Antonio,
I would take a look at your entire site using
One of my very favorite tools this tool will crawl your site and tell you if you have no follow's or other issues that would cause Google bot have trouble indexing your site.
Simply put your sites URL in the box presented in the tool you can find in the link here
http://www.feedthebot.com/tools/spider/
Then use link 2
Displays amount of links (internal, external, nofollow, image, etc.) found on webpage.
http://www.feedthebot.com/tools/linkcount/
You can then see if there is a no follow that might be creating a real problem inside of a page using the two URLs you should be a will to get about of this.
Check this much of your site is you possibly can with this as it will show you A lot of information that would be very relevant as to if your site can be crawled correctly or not
This third tool Will show you if your robots.txt file is still blocking all or part of your website the nice thing about this tool is is is built to make her about star text files however if you simply put your URL in the top and hit the upload button it will pull your robots.txt file this is very helpful when making comparisons between changes that have been made or you wish to make
http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/
Two check out your robot.txt file against what could be something blocking it I think that will
http://moz.com/blog/interactive-guide-to-robots-txt
http://moz.com/learn/seo/robotstxt
http://tools.seobook.com/robots-txt/
http://yoast.com/x-robots-tag-play/
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?hl=de
http://www.searchenginejournal.com/x-robots-tag-simple-alternate-robots-txt-meta-tag/67138/
A citation that I hope will help you is the not too noticeable difference between allowing everything and not allowing everything simply having a / after disallow: Will tell Google that you do not want to be showing up in their search engine results
Simply put I have the information below websites by default are set up with
Allow: /
Example Robots.txt Format
Allow indexing of everything
User-agent: *
Disallow:
or
User-agent: *
Allow: /
Disallow indexing of everything
User-agent: *
Disallow: /Disallow indexing of a specific folder
User-agent: *
Disallow: /folder/Please remember there are multiple ways to block a website for instance
PHP-based websites are extremely popular and if you're using a WordPress or agenda for many other
header("X-Robots-Tag: noindex", true);
I want to remind you what Tom Roberts said in the first response about using Twitter I have quoted him here however you can read it at the top of the Page below the first question
The most frequently crawled domain on the web is Twitter. If you could legitimately get your key URLs tweeted, either by yourselves or others, this may encourage the Google crawler to revisit the URLs, and consequently re index them. There won't be any harm SEO wise in sending tweets with your URLs, it's a quick and free method and so may be worth giving it a shot
Hope This Helps,
Thomas
-
Hi Antonio,
Sorry to hear you have had the same problem, due to our clients nature this error by the developer cost them a load of lost revenue.
In answer to your questions:
-
It took 19 days in total to recover
-
We took everyone's advice and implemented them but I am unsure what actually helped. I think working work GWMT is the best thing for it. Make sure you submit for a re-crawl as soon as possible and see what is still blocked
I know how scary the situation is but things will go back to normal. Its just a matter of playing the waiting game really, sorry I couldn't be of more help.
Rikki
-
-
Hi Rikki,
I know it's been some time since your post, however I just found it because a couple of weeks ago my developer did exactly the same.
It's been 2 weeks now and our traffic is still divided by 4 compared with what it used to be. My questions are:
1/ How long it finally took you to completely recover your previous traffic levels (if you finally did)
2/ Did you apply any of the advices from other bloggers? What would you recommend to do from your experience?
Thanks in advance. I am really worried at this moment, since we've got a peak campaign coming on very soon.
Regards,
Antonio (Citricamente)
-
Hi Rikki,
I really want to say great job though with those numbers. It's always good to see somebody pulling positive ROI. Good work!If I may ask what type of development do specialize in if you have a specialty?
My reason for asking is there are some excellent hosts that will allow you to run a staging server that changes everything like robots.txt back to follow and index when you hit the production button. Other hosts have similar methods.
In fact, that might be an idea that's worth a little bit of money. A nice WordPress plug-in that gives you a constant reminder here in the development phase and does the swap then deletes itself?
Or use a managed WordPress host if it's WordPress.
You can do so many cool things would git these days.
I am extremely happy you have found out there's nothing to worry about if it is simply the tags you will have your rank back before you know it.you can also use Webmaster tools on the manual setting and put it to Max I have done it on test sites, and the site was indexed just as well I would simply make sure I had a reminder telling me to return it to normal after.
You should set the rel="canonical as well/
Glad I was able to help,
Thomas
-
Hi guys,
Thanks very much for the responses. I guess my gut feeling was right that everything would come back to normal but just needed some reassurance.
I have made real progress with this client going from an online brought in revenue of £15k per month at the start of the year to £105k last month but it is all phone based so at the moment his call centre is like a ghost town - its a shame that can happen when a developer is trying to block his own dev sub domain and ends up blocking the whole thing. Just hope it doesn't take too long.
We will certainly try the social media route to see if that speeds things along.
-
please look and see that I updated my response I did I copied from a dictation software's writing pad and only copied a part of it when I meant to copy all of it
please read and let me know if I can be of help
sincerely,
Thomas
-
Please forgive my 1st comment I the button too early and use the dictation software so I save it to one page then paste to another I am sincerely sorry I got this part on there without the entire thing.
Send me the domain either privately if you can or through this chat I would be more than happy to look into it for you. I can tell you I have made the no follow no index mistake myself showing a intern something on our own site and talk about it here below.
However if you are still getting problems you may want to download
screaming frog SEO spider
it only will check for 500 links for free however it gives you invaluable insight
It is a download and works on Mac, Windows and Linux
http://www.screamingfrog.co.uk/seo-spider/
if you want to try something web-based
http://www.internetmarketingninjas.com/tools/
http://www.internetmarketingninjas.com/broken-links-tool/
http://www.internetmarketingninjas.com/seo-tools/robots-txt-generator/
http://www.internetmarketingninjas.com/seo-tools/google-sitemap-generator/
I would also not hesitate to use their DNS tool to check that everything there is okay
Another tool or tools I would strongly recommend and you can access for free are the excellent Internet marketing ninjas
The words used in the metadata tags, in body text and in anchor text in external and internal links all play important roles in on page search engine optimization (SEO). The On-Page Optimization Analysis Free SEO Tool lets you quickly see the important SEO content on your webpage URL the same way a search engine spider views your data. This free SEO onpage optimization tool is multiple onpage SEO tools in one, helpful for reviewing the following onpage optimization information in the source code on the page:
- Metadata tool: Displays text in title tags and meta elements
- Keyword density tool: Reveals onpage SEO keyword statistics for linked and unlinked content
- Keyword optimization tool: Analyzes on page optimization by showing the number of words used in the content, including anchor text of internal and external links
- Link Accounting tool: Displays the number and types of links used
- Header check tool: Shows HTTP Status Response codes for links
- Source code tool: Provides quick access to on-page HTML source code
if you are talking about just the index and no follow
I can now happily say I have done this identical thing.
I have done the exact same thing. I can tell you I was showing somebody how to use the WordPress SEO plug-in when I got distracted and simply did not change the settings back to follow and index. So approximately 2 to 3 days later I noticed a huge loss in ranking year for the company brand name.
(Luckily this was mine not a clients)
It took approximately two days after I changed the settings back to normal follow and index them submitted my entire website to Google's Webmaster tools even clicking yes when asked the index all large change
before I knew it all the rankings had returned back to normal literally the keywords I was tracking returned within the normal fluctuation I see as they were in many cases sometimes better & sometimes little bit worse what I had feared they never would come back at all.Sincerely,
Thomas
Believe me when I say I was extremely thankful for this and don't see why you will not get the same results with your site.
I hope this is a simple a mistake of just that one problem like mine that's the only thing I can give you a testimony of. I would say you have nothing to worry about. But remember to tell Google Webmaster tools I also did tell Bing but that's up to you
-
Should be as quick as google re-crawls the robots.txt.
Best thing you can do is add a couple of links to sites that are crawled daily, to encourage google to visit your clients site as soon as possible
Could be:
- newspaper sites - comments
- and the like
-
Hey there
I've seen this before and in almost all cases the rankings were returned to their previous state, give or take maybe 1 or 2 places (which would be normal SERP flux).
Unfortunately, I've found that this can often take weeks and there's no real sure-fire way of getting Google to update it quicker. Theoretically, to speed things up you want to get the crawler revisiting the URLs more and more often. Fresh backlinks would do this, but obviously you can't game that sort of thing for web spam reasons. You could also try pinging devices, such as GooglePing, but I'm not convinced by their effectiveness.
The most frequently crawled domain on the web is Twitter. If you could legitimately get your key URLs tweeted, either by yourselves or others, this may encourage the Google crawler to revisit the URLs, and consequently reindex them. There won't be any harm SEO wise in sending tweets with your URLs, it's a quick and free method and so may be worth giving it a shot.
Hope this helps you - I've often found you can't control these things but hopefully some of these theories might work. In the long-run, however, the rankings will return and so for normal SEO purposes, create content and links as per usual.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Positions dropped and do not recover
Hi all, Site: https://www.fascinators.com.au k/w: fascinators (this is the main one I am looking at) About 2 years ago, I redesigned the site. It used to be on first positions in Google Australia. After redesign, positions dropped and never recovered. No matter what I do, I can not get them back. I even involved professional SEO agency for the last 6 month to help build links, but nothing works. (old site used really old non mobile-friendly engine) As far as I know site is technically sound with no issues. In the last few month position for my main keyword 'fascinators' jumps really wildly. From position 9 to not being in Google first 100. And jump can happen overnight. As of today it disappeared from first 100 again. I am really tearing my hair out on what can be wrong. Link building is underway. This is one area I am aware of. Thinking of keyword spamming, i removed lots of 'fascinators' from URL, but that did not seem to help much. Any ideas will be greatly appreciated! Thanks, Rudolf
Intermediate & Advanced SEO | | rudolfl1 -
I'm noticing that URL that were once indexed by Google are suddenly getting dropped without any error messages in Webmasters Tools, has anyone seen issues like this before?
I'm noticing that URLs that were once indexed by Google are suddenly getting dropped without any error messages in Webmasters Tools, has anyone seen issues like this before? Here's an example:
Intermediate & Advanced SEO | | nystromandy
http://www.thefader.com/2017/01/11/the-carter-documentary-lil-wayne-black-lives-matter0 -
How to recover google rank after changing the domain name?
I just started doing SEO for a new client. The case is a bit unique as they build a new website and for some reason lunched in under another domain name. Old name is foodstepsinasia.com and new one is foodstepsinasiatravel.com OLD one is a respected webites with 35 in MOZ page authority and with +15000 incomming link (104 root domains) NEW one is curently on 0 The programmer has just that build the new website has set it up so that when people write or find the old domain name it redirect to the front page of the new website with the new domain name. this caused that my friends lost a lot of their rankings was so I believ it was a very bad solution. But I also think I can get most of the old rankings back, but my question is what to do now to get as much back of the rankings as fast as possible?? A) I believe I must change the domain name back to foodstepsinasia.com on the new website ? O B) Should I on the old website try finding the url of the pages with most page authority and recreate these urls on the new website or should i redict them to a page with related content? Looking forward to feedback from someone who have experience with similar cases. Thanks!
Intermediate & Advanced SEO | | nm19770 -
Manual Webspam Error by Google!
Hi,back in June 2013, our company received a notice of unnatural links which resulted in 'a manual spam action' from Google.A reconsideration request was filed a week later which received the following response from Google:_'We reviewed your site and found no manual actions by the webspam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team.'_Naturally we are confused by what seems to be an error in Google's communication.We are also left questioning whether it was necessary to remove the links Google stated were unnatural.Since the notice was received, we have struggled to recover traffic even after implementing Google best practices. Some clarity on the issue would be greatly appreciated.My URL is: www.homefurnitureland.co.uk
Intermediate & Advanced SEO | | users_engaged0 -
Whole site blocked by robots in webmaster tools
My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions. Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help. Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file. The suppliers have the descriptions from the files in their own sites.Help!!
Intermediate & Advanced SEO | | rrogers0 -
Squarespace Errors
We have a website hosted by SquareSpace. We are happy with SS, but have done some crawl diagnostics and noticed several errors. These are primarily: Duplicate Page Title Duplicate Page Content Client Error (4xx) We dont really understand why these errors are taking place, and wonder if someone in the Seomoz forum has a firm understanding of SS who is able to assist us with this? rainforestcruises.com thanks.
Intermediate & Advanced SEO | | RainforestCruises0 -
Duplicate page content and Duplicate page title errors
Hi, I'm new to SeoMoz and to this forum. I've started a new campaign on my site and got back loads of error. Most of them are Duplicate page content and Duplicate page title errors. I know I have some duplicate titles but I don't have any duplicate content. I'm not a web developer and not so expert but I have the impression that the crawler is following all my internal links (Infact I have also plenty of warnings saying "Too many on-page links". Do you think this is the cause of my errors? Should I implement the nofollow on all internal links? I'm working with Joomla. Thanks a lot for your help Marco
Intermediate & Advanced SEO | | marcodublin0 -
Managing 404 errors
What is the best way to manage 404 errors for pages that are no longer on the server. For example, client deletes old site from server and replaces it with new site. Webmaster tools is reporting 100 + 404 errors form the old site. I've blocked the 404 pages with robot.text. Requested removal in google webmaster tools. And created a custom 404 page - http://www.tvsphoto.com/missingurlexample Is there anything else I can do?
Intermediate & Advanced SEO | | SEOProPhoto0