Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Removing robots.txt on WordPress site problem
-
Hi..am a little confused since I ticked the box in WordPress to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap.
Checked source code and the robots instruction has gone so a little lost. Any ideas please?
-
Hi,
I edited the robots.txt file for my website http://debtfreefrombankruptcy.com yesterday to allow search engines to crawl my site. However, Google isn't recognizing the new file and is still saying that my sitemap is blocked from search. Here is a link to the file itself:
http://www.debtfreefrombankruptcy.com/robots.txt
The Blocked URLs tester said that the file allows Google to crawl the site, but in actuality it still isn't recognizing the new file. Any advice would be appreciated. Thanks!
-
I can help you out as this issue DROVE ME NUTS.
1. I didnt have a Robots.txt (yet)
2. I had Yoast installed
3. Im pretty sure it created a Robots.txt even though it doesnt exist in my root (.com/here)
4. My Google webmaster tools shows this
User-agent: Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category//* Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /? Disallow: /?Allow: /wp-content/uploadsAllow: /assets Create a Robots.txt
1. login to wordpress 2. Click SEO in your side toolbar (Yoast WordPress Plugin settings) 3. Go to edit files under SEO (in the side toolbar)
And now you have the option to edit your Robots.txt file.
-
Hi Sophia
I just checked and see your homepage indexed in google.co.uk with a cache date of April 26th. You should be all set!
-Dan
-
Quick update - by amending the robots text file and switching sitemap plugin over to Yoast I finally got the sitemap to index without robots.txt warnings although the Home page of site was not indexed, 'oh dear'. 5 out of the 7 pages in the sitemap were indexed by Google so It's a start but some more investigating to be done on my side.
-
Dan,
Cant thank you enough! The sitemap request is still pending in Google - maybe I sent too many requests But it's time to sit back and wait for the good news hopefully. Thanks again.
-
Hi Sofia
I just ran the same validator on your sitemap and it went through fine - see screenshot
I intended to mean that you should just be sure Google Webmaster Tools accepts the sitemap as valid - if so, there's no need to run through a 3rd party validator. Apologies if I didn't state it clearly!
Let me know, but from what I can see it looks good!
-Dan
EDIT - Looking more closely, it looks like your ran the homepage through the validator - you would actually enter the sitemap address its self in the validator - http://containerforsale.co.uk/sitemap.xml
-
Hi Dan,
I followed the above advice and switched to the Yoast generated sitemap but after testing on http://www.xml-sitemaps.com/validate-xml-sitemap.html I got the following result - no idea what it means but it looks nasty...
Schema validating with XSV 3.1-1 of 2007/12/11 16:20:05Schema validator crashed
The maintainers of XSV will be notified, you don't need to
send mail about this unless you have extra information to provide.
If there are Schema errors reported below, try correcting
them and re-running the validation.Target: http://containerforsale.co.uk
(Real name: http://containerforsale.co.uk
Server: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4)The target was not assessedLow-level XML well-formedness and/or validity processing output
Warning: Undefined entity raquo
in unnamed entity at line 16 char 83 of http://containerforsale.co.uk
Warning: Undefined entity nbsp
in unnamed entity at line 160 char 10 of http://containerforsale.co.uk
Error: Expected ; after entity name, but got =
in unnamed entity at line 274 char 631 of http://containerforsale.co.u -
Sofia
You are using Yoast SEO plugin for WordPress, so use the XML sitemap within Yoast. You don't need a separate plugin for the XML sitemap. And yes, within Yoast turn the sitemap on.
Hope that helps!
-Dan
-
Indeed, thanks everyone - it's really appreciated!
I have updated the robots.txt as indicated and re submitted site map but looks like Google still has problems with my site since the error warning for robots is there after the processing is done.
Quick question - I am using a plugin called Google XML Sitemaps which has the following tick box option.
'Add sitemap URL to the virtual robots.txt file'.
The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!'Should this box be ticked or un-ticked please? Fyi I currently don't have the box ticked.
-
Thanks guys for all the responses and helping!
Three Things to try
1.Fix Robots.txt
Sofia - I just checked your robots.txt now and it reads;
User-agent: * Disallow: Sitemap: http://containerforsale.co.uk/sitemap.xml.gz- with the sitemap on the same line as disallow - I'd check on that and make sure its on a separate line.
- ALSO, you don't need the .gz on the sitemap file just sitemap.xml
2. Re-submit Sitemap
- RESUBMIT your sitemap to webmaster tools and make sure its valid.
3. Submit URL to Webmaster Tools (only last resort)
this is only last case scenario, shouldn't have to do this on the homepage if everything is correct.
- go to fetch as googlebot ->run the fetch ->then submit URL
- do this for the homepage
- see article on google blog for reference
Let us know if you're all set, thanks!
-Dan
-
Ok thanks Brent, I changed to
User-agent: *
Disallow:
Sitemap: http://containerforsale.co.uk/sitemap.xml.gz
Guess I will just have to wait for Google to refresh now...
-
yes, the urls being blocked are includes from your Wordpress program.
-
Thanks for the heads up.
The warning just says 7 Url''s blocked by robots.txt. - have seen this issue posted on the WordPress boards by others but no real insight into solutions.
Perhaps I should try your idea of
Change the robots.txt file to this:
User-agent *
Disallow:
-
Well there is a robots.txt file. You can view it here: http://containerforsale.co.uk/robots.txt
What warnings are you getting in your sitemap submission area? It appears to look alright: http://containerforsale.co.uk/sitemap.xml But I tried to validate it and got a 504 Gateway Time-out error. http://www.xml-sitemaps.com/index.php?op=validate-xml-sitemap&go=1&sitemapurl=http%3A%2F%2Fcontainerforsale.co.uk%2Fsitemap.xml&submit=Validate
-
Its weird, the front page warning on Google webmaster for robots has disappeared now, but still got the warnings in the sitemap submission area. My host suggests I just wait a bit longer for Google to update because he said same as you - that there doesn't seem to be any robot.txt file.
-
Doesn't appear to be blocked, so maybe it has something to do with your /wp-includes/ directory.
Change the robots.txt file to this:
User-agent *
Disallow:
-
Hey Guys,
Thanks for your replies...the domain is http://containerforsale.co.uk ,My host told me to look in the Public HTML file folder for the robots.txt file and just delete it but can't see it in there?
My host said he found a tester site and it doesn't report any issues:
http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php
This is the display I get from http://containerforsale.co.uk/robots.txt
User-agent *
Disallow: /wp-admin/
Disallow: /wp-includes/ -
Hi Sofia,
Two things you need to consider when troubleshooting this:
The actual robots.txt file (located in the root directory of your site) and the meta-robots tags in the section of your HTML. When you say you checked the source code and the robots instructions were missing, I think you were talking about the meta-robots tags in the actual HTML of your site.
Webmaster Tools is probably referring to the actual robots.txt file in your domain's root path, which would differ entirely and not be visible by checking the HTML on your site. Like Nakul and Brent said, if you'll let us know your site's URL and paste the content of your robots.txt file here, I'm sure one of us can help you resolve the problem fairly quickly.
Thanks!
Anthony
-
copy whatever you have in your robots.txt file here and we will tell you the issue.
SEOmoz has a great article about Robots.txt files here: http://www.seomoz.org/learn-seo/robotstxt
-
The robots.txt would probably not be a part of the Wordpress Configuration. Allow indexing is controlled via Meta Data by the Wordpress Architecture.
I would look for something like this in yourdomain.com/robots.txt
disallow /
or something like that. If that does not help, PM me your site URL and I would be glad to look it up for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Move a Wordpress Site to HTTPS with Bluehost
HI Guys, do you think that the following guide is enoght to move a bluehost wordpress site to https in a seo best practive way? https://www.shoutmeloud.com/free-ssl-certificate-bluehost-hosting.html Basically their steps are: Install SSL on Bluehost panel Install Really Simple SSL Wp Plugin Edit Your .htacess File & Add The Code For HTTP To HTTPS Redirection Update All HTTP URLs In Database To HTTPS Using Search and Replace Plugin Use Broken Link Checker plugin & use its redirection module to find links to 3rd party sites with HTTP that should now be HTTPS. Last thing to do Submit your new HTTPS site to Google Search Console & submit your sitemap. Update your profile link on Google Analytics. Update your website links on social media profiles & anywhere else they exist. This step you can do in pieces in the coming days. Read this guide to learn more about HTTP to HTTPS migration & fixing mixed content. If you disabled Who.Is guard for your domain name, you can enable it now. Do you know a better practical guide for wordrpess? in term of usefull plugins to handle the migration? Tx to everyone!
Technical SEO | Jul 30, 2018, 8:03 AM | Dreamrealemedia0 -
Robots.txt & meta noindex--site still shows up on Google Search
I have set up my robots.txt like this: User-agent: *
Technical SEO | Sep 23, 2016, 4:03 PM | RoxBrock
Disallow: / and I have this meta tag in my on a Wordpress site, set up with SEO Yoast name="robots" content="noindex,follow"/> I did "Fetch as Google" on my Google Search Console My website is still showing up in the search results and it says this: "A description for this result is not available because of this site's robots.txt" This site has not shown up for years and now it is ranking above my site that I want to rank for this keyword. How do I get Google to ignore this site? This seems really weird and I'm confused how a site with little content, that has not been updated for years can rank higher than a site that is constantly updated and improved.1 -
How to force Wordpress to remove trailing slashes?
I've searched around quite a bit for a solution here, but I can't find anything. I apologize if this is too technical for the forum. I have a Wordpress site hosted on Nginx by WP Engine. Currently it resolves requests to URLs either with or without a trailing slash. So, both of these URLs are functional: <code>mysite.com/single-post</code> and <code>mysite.com/single-post/</code> I would like to remove the trailing slash from all posts, forcing mysite.com/single-post/ to redirect to mysite.com/single-post. I created a redirect rule on the server: ^/(.*)/$ -> /$1 and this worked well for end-users, but rendered the admin panel inaccessible. Somewhere, Wordpress is adding a trailing slash back on to the URL mysite.com/wp-admin, resulting in a redirect loop. I can't see anything obvious in .htaccess. Where is this rule adding a trailing slash to 'wp-admin' established? Thanks very much
Technical SEO | Nov 13, 2018, 1:49 PM | james-tb0 -
Trailing Slash Problems
Link juice being split between trailing slash and non versions. ie. ldnwicklesscandles.com/scentsy-uk and ldnwicklesscandles.com/scentsy-uk/ Initially asked in here and was told to do a rewrite in the htaccess file. I don't have access to this with squarespace, nor can I add canonical tags on a page by page basis. 301 redirect from scentsy-uk to scentsy-uk/ didn't work either...said that the redirect wasn't completing in an error message on the browser. Squarespace hasn't been very helpful at all. My question is....is there another way to fix this? or should I just call it a day with squarespace and move to wordpress?
Technical SEO | Feb 19, 2016, 3:28 PM | cmjolley0 -
Internal search : rel=canonical vs noindex vs robots.txt
Hi everyone, I have a website with a lot of internal search results pages indexed. I'm not asking if they should be indexed or not, I know they should not according to Google's guidelines. And they make a bunch of duplicated pages so I want to solve this problem. The thing is, if I noindex them, the site is gonna lose a non-negligible chunk of traffic : nearly 13% according to google analytics !!! I thought of blocking them in robots.txt. This solution would not keep them out of the index. But the pages appearing in GG SERPS would then look empty (no title, no description), thus their CTR would plummet and I would lose a bit of traffic too... The last idea I had was to use a rel=canonical tag pointing to the original search page (that is empty, without results), but it would probably have the same effect as noindexing them, wouldn't it ? (never tried so I'm not sure of this) Of course I did some research on the subject, but each of my finding recommanded one of the 3 methods only ! One even recommanded noindex+robots.txt block which is stupid because the noindex would then be useless... Is there somebody who can tell me which option is the best to keep this traffic ? Thanks a million
Technical SEO | Apr 13, 2012, 7:13 PM | JohannCR0 -
Is blocking RSS Feeds with robots.txt necessary?
Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | Jul 9, 2011, 12:15 PM | nicole.healthline0 -
Delete old site but redirect domain to a new domain and site
I just have a quick query and I have a feeling about what the answer is so just wanted to see what you guys thought... Basically I am working on a client site. This client has a few other websites that are divisions of their company. However these divisions/websites are no longer used. They are wanting to delete the websites but redirect the domains to their name main website. They believe this will pass on SEO benefits as these old division sites are old and have a good PR and history. I'm unsure for DEFINITE, which way is correct?
Technical SEO | Jun 10, 2011, 9:24 AM | Weerdboil0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | May 6, 2011, 3:55 PM | JordanJudson0