Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
US and UK Websites of Same Business with Same Content
Hello Community, I need your help to understand, whether I can use the US website's content on my UK website or not? US Website's domain: https://www.fortresssecuritystore.com UK Website's domain: https://www.fortresssecuritystore.co.uk Both websites are having same content on all the pages, including testimonials/reviews. I am trying to gain business from Adwords and Organic SEO marketing. Thanks.
Technical SEO | | CommercePundit1 -
How to point a framework to your website
Hi, my client use the NJOYN framework to insert all of the new job post on his website. The problem is that this framework is not hosted at the same place that his website is hosted. This create 2 domain name. First: Example.com second: example.njoyn.com How can I tell google that example.noyn.com link with the site example.com? Can I do this by simply 301 redirect the domain from example.njoyn.com to example.com? I know that this is a framework, is there a way to have access to their htaccess file? Finally, is there a way to rewrite example.njoyn.com/job1 to example.com/job1? Thanks alot
Technical SEO | | bigrat950 -
Sub Domains and Robot.txt files...
This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left. I have a sub level domain on which a website sits. The Main domain has a robots.txt file that disallows all robots. It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website. My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain? I wouldn't have thought so but I can find nothing else. Thanks in advance.
Technical SEO | | Vizergy0 -
Big Mess - Multiple Websites
I have a customer, a Psychologist, who put up +/-20 websites many years ago. He has 1 main site (with his name as the domain) with hundreds of pages of quality content. The other sites are all exact match domains - anxiety counseling, couples counseling, etc. Some are single page sites, others have a good amount of quality content. Many of the EMD sites were getting ranked on the first page, as was the main site. The money site was ranking on the first page for the best keywords All of the EMD pages linked back to the main site, many with site wide footer links. The main site did not link back. All of the sites are on the same IP address. These sites have been in place for years. I don't believe that he has a duplicate content problem. About 8 weeks ago, the rankings for the main site crashed, moving 10 or more SERP pages deep. The EMD sites are still ranking. He has not gotten any nasty-grams from Google in Webmaster Tools. The Psychologist relies exclusively on organic for his business, and it has taken a significant hit. 1. Has anyone else seen this happen? 2. Is it safe to assume that Google finally nailed him for using a linking scheme? 3. How can we unwind this? The other sites are still generating business, and if those go away, he is really screwed. 4. Will taking down all of the links from the other sites be enough? Would moving the money site to another hosting company on a different IP make a difference? Ideas? I think the white hat answer would be to take down the EMD sites, and 301 redirect to the main site. The problem is that the loss of business from this process could be catastrophic.
Technical SEO | | CsmBill0 -
Removing thousands of shady backlinks?
Hey guys, We've been hired to redesign a website that has thousands of backlinks created by a (possibly) shady offshore company, and I'm wondering if anyone out there has experience dealing with a deletion of this size and type. Is it as simple as just disavowing the whole lot? Thanks, Jason
Technical SEO | | JKorolenko0 -
Mobile site - allow robot traffic
Hi, If a user comes to our site from a mobile device, we redirect to our mobile site. That is www.mysite/mypage redirects to m.mysite/mypage. Right now we are blocking robots from crawling our m. site. Previously there were concerns the m. site could rank for normal browser searches. To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site). From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing. Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)
Technical SEO | | NicB10 -
Subdomain Robots.txt
If I have a subdomain (a blog) that is having tags and categories indexed when they should not be, because they are creating duplicate content. Can I block them using a robots.txt file? Can I/do I need to have a separate robots file for my subdomain? If so, how would I format it? Do I need to specify that it is a subdomain robots file, or will the search engines automatically pick this up? Thanks!
Technical SEO | | JohnECF0