What can I do if Google Webmaster Tools doesn't recognize the robots.txt file?
-
I'm working on a recently hacked site for a client and and in trying to identify how exactly the hack is running I need to use the fetch as Google bot feature in GWT.
I'd love to use this but it thinks the robots.txt is blocking it's acces but the only thing in the robots.txt file is a link to the sitemap.
Unde the Blocked URLs section of the GWT it shows that the robots.txt was last downloaded yesterday but it's incorrect information. Is there a way to force Google to look again?
-
No, but they might write to it, modify it, or do all sorts of other nasty stuff I've seen hackers do when they get a hold of any writeable file on a system.
-
lol it's a robots text file. what are they going to do. Steal it? I should have clarified do a 777 to make sure that is not your problem, then yes change the permission to be tighter
-
Eesh I don't recommend 777. 644 or, if you're going to change it right back, 755 at most.
-
File permission maybe? Change it to 777 and try it again
-
If you have shell access on Linux you can use wget or GET or run lynx.
If google is getting the wrong robots file then your web server must be sending out something other than what you think is the robots file.
What happens if you do this in your browser:
-
Looking in my log files, Google hits robots.txt just about every time it crawls our site.
What are you trying to accomplish using fetch as Googlebot? Any chance CURL could do the job for you, or another tool that ignores robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SERP result (URL) doesn't change after a 301
A couple of months ago there was a result in Google for our branded search term which wasn't the 'official' URL, actually the result shown in the SERP was www.mycompany-ip.nl. We've applied a 301 redirect of this URL to the 'official' URL which is a subdomain: department.mycompany.nl. From Google the redirect is obviously working, but up until now, I don't see Google replacing the incorrect URL by the correct URL. I am wondering what to do to make the result correct. André
Technical SEO | | ConclusionDigital0 -
Will it be possible to point diff sitemap to same robots.txt file.
Will it be possible to point diff sitemap to same robots.txt file.
Technical SEO | | nlogix
Please advice.0 -
Webmaster Tools Verification Problem
Hello, Somehow a website I'm working on has lost it's verification in Webmaster Tools. I have absolutely no clue why... The Google Analytics code is still working, the verification meta tag is in place, both are not working. I get an error message about Google not being able to connect to the server. I asked about any possible changes to server settings or stuff about that, but apparently nothing has changed there. The URL in question is Bivolino.com Does someone has any other ideas what I could be looking for. Thanks, Kind regards, Erik
Technical SEO | | buiserik0 -
Why can't I rank for my brand name?
We are soon to launch a new company in New Zealand called Zing. I have been tasked with the challenge of ranking as highly as possible for anything to do with Zing before launch in February. Zing is in the financial industry so my colleagues thought that it would be a good idea to make a small blog (very small with literally one post) that reviewed other financial lenders. This sight stayed online for a couple of months before it was replaced. The official website is still yet to launch, so as an in between, I asked that we make a splash page with a small competition on it (see here at zing.co.nz). I would have preferred there were more keywords on the website but this was not achieved. I am still pushing for this and am hoping to get a few pages on there in the near future. Instead of getting the keywords on the splash page, I was given permission to start a subdomain, (blog.zing.co.nz). This contains many more common search terms and although its not quite doing the job I would like, the rankings for Zing have started to increase. At the moment, we are ranking number 1 for a few brand related keywords such as zing loans. This is why I feel something is wrong, because we rank number 1 for over 10 similar terms but yet we DO NOT EVEN APPEAR on the search engines at all for Zing. Have we been penalized? Do you have any suggestions at all? Do you think we could have been penalized for the first average blog? Maybe I messed up the swap over? Any help would be hugely appreciated!
Technical SEO | | Startupfactory0 -
Link Webmaster Tools to Tag Manager
I know it is now possible to verify google webmaster tools accounts using google tag manager, but is it also possible to create a link from webmaster tools to tag manager? I would like to connect the two. I have analytics already linked to tag manager, but can't find a clear answer regarding webmaster tools. Anybody? tx!
Technical SEO | | susancompass0 -
Google webmaster showing 0 indexed, yet I can see them all them Google search?
I can see them all the pages showing up in Google when i search for my site. But in webmaster tools under the sitemaps section in the indexed pages - the red bar is showing 0 indexed pages, even though they seem to be indexed. Any idea why is this showing like this? I don’t really think it’s that important as the pages are still indexed, but it just seems odd. Please see in the image.
Technical SEO | | Perfect0070 -
OpenSite Explorer doesn't show Twitter
When I type competitors sites into opensiteexplorer it shows their Twitter page as one of the tops links back. However, my site, our Twitter page isn't even on the links back. Does twitter even though no follow hold any value? Why can't opensite explorer see it? If OSE can't see it then do you think Google isn't?
Technical SEO | | PhotoGazza0 -
Sitemap for pages that aren't on menus
I have a site that has pages that has a large number, about 3,000, pages that have static URLs, but no internal links and are not connected to the menu. The pages are pulled up through a user-initiated selection process that builds the URL as they make their selections, but,as I said, the pages already exist with static URLs. The question: should the sitemap for this site include these 3,000 static URLs? There is very little opportunity to optimize the pages in any serious kind of way, if you feel that makes a difference. There is also no chance that a crawler is going to find its way to these pages through the natural flow of the site. There isn't a single link to any of these pages anywhere on the site. Help?
Technical SEO | | RockitSEO0