How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Getting keywords to rank on new landing pages
I've built new landing pages for a website, and have loaded them with researched keywords in the content, alt image attributes and metas etc - but, after a number of crawls, the keywords are currently being matched to other existing web pages on the website. Does anyone have any advice on 'unlatching' these keywords from these pages, and instead getting them to match with pages that have been optimised for them? Many thanks!
Moz Pro | | Darkstarr6660 -
Search Volume Discepancies
I was doing research on keywords and found significant discrepancies between the search volumes on Moz and sites such as Spyfu and Semrush. What could be the reason behind these differences?
Moz Pro | | arjun97970 -
My localized domains are not getting any love or juice it seems?
Not sure what best practices are, but I have my main domain, a .com and my localized domains, .com.sg and .co.id. As you can see from the screenshot from opensite explorer, i am not getting any linkjuice from my .com domain to my other local domains - what am i doing wrong here? j3LWnOJ
Moz Pro | | venkatraman0 -
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location?
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location? What tool/service have you found that gives accurate rank results? To me it seems like there is no guaranteed global #1position any more, or is this the situation for only certain types of searches? Likewise how can anyone tool give a global accurate rank score?
Moz Pro | | greengroup0 -
Crawl results on keyword rankings differ from actual search rankings
For the second week in a row, the weekly keyword rankings in my MOZ account show "not in top 50" for most keywords. However, when I check some of my rankings I still broadly have the rankings that I used to have. It seems that the MOZ crawl is experiencing some difficulties. Does anyone have suggestions how to find out what's going on? If I can't rely on the crawl outcomes, then the tool is not very useful anymore!
Moz Pro | | matteis0 -
Metric "Total Links" can somebody explain this metric to me?
Dear colleagues, Who can explain the following to me? Subdomain metrics - Total links The total links is huge compared to the sum of internal and external links. I do not understand this metric. Can somebody help me to explain this the metrich "total links" I have to present these metrics to my customer and do not want to have "don't know" as an answer 😉 Thanks, Alain Nijholt BMC Internet Marketing
Moz Pro | | bmcinternetmarketing0 -
SEOMoz toolbar - Anyone else have problems with Search Profiles?
(Using Firefox 7.0.1) I just downloaded the toolbar and the Custom Search Profiles do not work--clicking on any of them adds "%" and numbers to the search query. I've created a couple of specific locations and I'd really like to get this figured out. Does this function work correctly for anyone? Am I doing something wrong?
Moz Pro | | Court_LOQUA0