How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Way To Write H1 to Get Max CTR?
Can anyone please share some guidelines regarding how to write the H1 subject of the post which can generate the maximum CTR.?. I noticed even when i able to rank on the featured snippet with good enough search volume the CTR is too low. Anyone share some expert advice please?
Moz Pro | | ndhsne450 -
My homepage is not getting indexed by google for some reason
My homepage http://www.truebluelifeinsurance.com is not indexed by google. The rest of my site is indexed. The hompage is indexed by bing. I looked in the webmaster tools and there is no indication why. I believe the issue started when I did a site re-design in August. Any ideas?
Moz Pro | | Brian_Greenberg0 -
Did I get paid for Jan or Feb ?
I started pro account in 18 jan. And I got an invoice in 18 feb. So am I supposed to get first month free ? Is this payment belongs to Jan or Feb ?
Moz Pro | | zizigo0 -
What is the correlation between the SEOMOZ PR number and where my site is showing up in an organic search?
For years, our company placed 3 or 4 in an organic search on Google for the keyword I am inquiring about. A couple of months ago, we went down to number 15. I then subscribed to SEOMOZ and began a concerted effort to improve our ranking for this keyword. Slowly, our listing came up higher in searches and on our reports- back onto the first page #8. Our weekly SEOMOZ reports reflected this improvement, but then it dropped down again to page 2, #11. My most recent report, this week, gave us a PR 8 for this keyword, but our listing is showing upon page 2, #11. I do have my personal search and browser history disabled. I am unclear as to why there is this discrepancy.
Moz Pro | | gfiedel0 -
Does Rogerbot respect the robots.txt file for wildcards?
Hi All, Our robots.txt file has wildcards in it, which Googlebot recognizes. Can anyone tell me whether or not Rogerbot recognizes wildcards in the robots.txt file? We've done a Rogerbot site crawl since updating the robots.txt file and the pages that are set to disallow using the wildcards are still showing. BTW, Googlebot is not crawling these pages according to Webmaster Tools. Thanks in advance, Robert
Moz Pro | | AC_Pro0 -
"Wed" flag in the ranking repport
Hi seomoz, Sorry to bother you with that "noobie" question but i was wondering what can the wed flag" mean in the repport ? A crawl just ran this morning so i guess it's not some kind of calendar "issue" unless the crawl don't run on every keywords ?? Thanks for the enlightenment 😉
Moz Pro | | mywebshop0 -
On the Crawl Diagnostics Summary, its reporting over 100 "Title Missing or Empty" issues, but they all check out fine?
Wondering if there Is a bug with the crawler or known timeout issues? Site speed is fast, but we do run a couple of large cron jobs out of hours, which may be the cause of any timeouts, but shouldn't the crawler report that, rather saying no title tags on 100 pages, when there are? SEOmoz newbie, so still finding my feet 🙂
Moz Pro | | sjr4x40 -
What's name of SEOmoz and Open Site Explorer robots?!
I would like to exclude in robots.txt SEOmoz and Open Site Explorer bots to don't let them index my sites… what's their names?
Moz Pro | | cezarylech0