How can I make Google Webmaster Tools see the robots.txt file when I am doing a .htacces redirec?
-
We are moving a site to a new domain. I have setup an .htaccess file and it is working fine. My problem is that Google Webmaster tools now says it cannot access the robots.txt file on the old site. How can I make it still see the robots.txt file when the .htaccess is doing a full site redirect?
.htaccess currently has:
Options +FollowSymLinks -MultiViews
RewriteEngine on
RewriteCond %{HTTP_HOST} ^(www.)?michaelswilderhr.com$ [NC]
RewriteRule ^ http://www.s2esolutions.com/ [R=301,L]Google webmaster tools is reporting:
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.
-
Possible Solitions for your problem:
.htaccess authentication blocking robots.txt
301 redirect. How to make an exception for the robots.txt
http://forum.cs-cart.com/topic/23747-301-redirect-how-to-make-an-exception-for-the-robotstxt/
1. Canonical robots.txt
http://digwp.com/2011/03/htaccess-wordpress-seo-security/
General .htaccess tutorials: http://httpd.apache.org/docs/2.0/howto/htaccess.htmlhttp://httpd.apache.org/docs/2.0/misc/rewriteguide.html
-
Thank you that seems to be working.
-
You could add an exception to the htaccess to allow the robots to be loaded. You would do this with by adding another condition. I'd use something like:
<code>Options +FollowSymLinks -MultiViews RewriteEngine on RewriteCond %{REQUEST_URI} !/robots.txt RewriteCond %{HTTP_HOST} ^(www\.)?michaelswilderhr\.com$ [NC] RewriteRule ^ [http://www.s2esolutions.com/](http://www.s2esolutions.com/) [R=301,L]</code>
Disclaimer: I am lucky enough to have people at work who check these things. This hasn't been checked! Use at your own discretion
However I'll admit that I've never used this. I just stick the 301 in and it all seems to work out fine. Probably done it on hundreds of domains over the years.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can we re rank our Penalyzed website in Google?
Hello This is Maqbul, from India. I have a jobs portal blog [ bharatrecruit.com]. It was getting around 50K to 100K Views a Day and made me $100 a day. But after a few months, my competitor made negative SEO with 12,000 Spammy backlinks. Suddenly my site was hit by Google and now it is getting 200 to 300 Pageviews a day. So the question is I did not disavow bad links for a long time like 3 to 4 months. Now I disavow all the bad links but the website is not ranking. Can we re-rank this site or create another website. Please reply must. None of the bloggers can answer this. Thanks, Regards Maqbul
Technical SEO | | vinaso960 -
How can I make it so that robots.txt is not ignored due to a URL re-direct?
Recently a site moved from blog.site.com to site.com/blog with an instruction like this one: /etc/httpd/conf.d/site_com.conf:94: ProxyPass /blog http://blog.site.com
Technical SEO | | rodelmo4
/etc/httpd/conf.d/site_com.conf:95: ProxyPassReverse /blog http://blog.site.com It's a Wordpress.org blog that was set as a subdomain, and now is being redirected to look like a directory. That said, the robots.txt file seems to be ignored by Google bot. There is a Disallow: /tag/ on that file to avoid "duplicate content" on the site. I have tried this before with other Wordpress subdomains and works like a charm, except for this time, in which the blog is rendered as a subdirectory. Any ideas why? Thanks!0 -
Duplicate Meta Titles and Descriptions Issue in Google Webmaster Tool
Hello All, We have one site named http://www.bargains-online.com.au/ & have some categories along with filter option on left side like filter by price & by brand, ect. We have already set rel canonical tags on all filtered pages, but still those all pages showing duplicate page titles and description warning in HTML Improvements section in Google Webmaster Tool. For Example: http://www.bargains-online.com.au/pressure-cleaners.html We've set rel canonical tag on below pages. http://www.bargains-online.com.au/pressure-cleaners/l/manufacturer:black-eagle.html http://www.bargains-online.com.au/pressure-cleaners/l/price:2,100.html http://www.bargains-online.com.au/pressure-cleaners/l/price:3,100.html Kindly request if anybody has any solutions for the same, please share with us. Thanks, Akshay
Technical SEO | | akshaydesai0 -
Webmaster Tools Data Discrepancies/Anomalies
Hi Im looking in GWT account for a client of mine and see that in the index status area 400+ pages are indexed so all seems ok there! But then in the sitemap area 111 pages have been submitted but only 1 indexed ! Any ideas whats going on here ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Link Webmaster Tools to Tag Manager
I know it is now possible to verify google webmaster tools accounts using google tag manager, but is it also possible to create a link from webmaster tools to tag manager? I would like to connect the two. I have analytics already linked to tag manager, but can't find a clear answer regarding webmaster tools. Anybody? tx!
Technical SEO | | susancompass0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
What are the pros and cons of changing my domain from .com to .us in Google webmaster tools?
Hi, I'm migrating my site from a .com domain to local country domains. I'm wondering what to consider if i chose to move the .com to .us domain. What should I consider before deciding? BR
Technical SEO | | Quru0 -
Does google "see through" php/asp redirects?
A lot of the time I see companies employing a technique like this: <a target="_blank" href="/external/wcpages/referral.aspx?URL=http%253a%252f%252fwww.xxxx.ca&ReferralType=W&ProfileID=22&ListingID=96&CategoryID=219">xxxxxa> Or similarly with php. In an attempt to log all the clicks that exit their site from certain locations. When google bot comes along and crawls this page, does it still understand that this page links to www.xxxx.ca?
Technical SEO | | adriandg0