Our crawler was not able to access the robots.txt file on your site.
-
Good morning,
Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.
Any advice or guidance would be much appreciated.
https://www.agrisupply.com/robots.txt
Thank you for your time.
-Danny
-
Hi,
Aren't you just allowing Googlebot to scan /images folder?
-
Try changing the user agent to googlebot and see if it works
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links to Your Site: No Data Available in Google Search Console
The site I am working on did not have their site submitted to Google Search Console (formerly Google Webmaster Tools). I submitted the site and a sitemap that auto updates. Google is crawling the site daily (about 30 pages a day). Under Search Traffic > Links to Your Site it shows no data is availible. I thought it was because it was a newly submitted site, but it has been two months now. Moz seems to have the same issue. Moz does show inbound links, but their are some that we think should really help us that are not shown. For instance, the Dallas Morning News wrote this article. They have a high DA and PA. Also, iliveindallas.com has an article about us that is still on the front page. That was a few weeks ago but also does not show up on Moz or Google SC. We are trying to be selective about the links we are getting. That they are follow links from reputable sites. Worried that both Google and Moz are not showing them.
Moz Pro | | TapGoods1 -
Compare sites?
I'm frustrated, so want to ask a stupid question....My site.. www.seadwellers.com outranks my biggest competitor in most Moz catagories... www.rainbowreef.us ...EXCEPT Facebook likes...(he has a ton) **And yet, rainbowreef.us outranks me in most keywords on
Moz Pro | | sdwellers
Google?! I know it's not simple...but Can anyone take a quick peek and give me any insight as to why??? ** Example "Dive Key Largo" keyword...he is #1 and I am #5...typical in the most important keywords!0 -
Issues with Moz producing 404 Errors from sitemap.xml files recently.
My last campaign crawl produced over 4k 404 errors resulting from Moz not being able to read some of the URLs in our sitemap.xml file. This is the first time we've seen this error and we've been running campaigns for almost 2 months now -- no changes were made to the sitemap.xml file. The file isn't UTF-8 encoded, but rather Content-Type:text/xml; charset=iso-8859-1 (which is what Moveable Type uses). Just wondering if anyone has had a similar issue?
Moz Pro | | BriceSMG0 -
Open Site explorer
I checked some of the results with OSE that ranked top on Goolge SERPS that have high Domain Authority,Page Authority 1 and content on the website's particular page is thin. How come they rank high when PA is very low?
Moz Pro | | Frost0 -
Factoring in OBL to Open Site Explorer
Morning all - I've been using Open Site Explorer a fair bit since I signed up to SEOMoz a couple of months ago and found it to be a little different to other tools in the past, so would welcome some thoughts on what I should be looking for when I run reports. For example, I've always thought that Outbound links should be important when deciding whether to place a link on a a particular site, but in OSE I'm only given a Page Ranking and a Domain Ranking. Am I therefore to understand that these two factors outrank OBL and that the number of links on a page don't matter? Or have these been factored into the overall Page rank already? Thanks a lot for your help,
Moz Pro | | theshortstack
Simon0 -
Is there any alternatives to Yahoo Site Explorer?
Hi, Is there any alternatives to Yahoo Site Explorer? I want to find backlinks of some unavailable pages and YSE was a great tool to deal with. But when I used tools like Open Site Explorer, ahrefs.com etc., I couldn't find one same like YSE. Because when I tried, it shows'No data available for this URL' where on YSE I found inbound links. What Alternatives Are There? thanks in advance!
Moz Pro | | VipinLouka780 -
Why is the crawler following form action links?
I have an issue with one of my sites where the SEOMoz crawler is following some form action links. It is my understanding that the crawler will ignore these links. Why would it not be ignoring them in certain cases. If you need more detail, please ask. Thanks.
Moz Pro | | AmberHanson0 -
Any tools for scraping blogroll URLs from sites?
This question is entirely in the whitehat realm... Let's say you've encountered a great blog - with a strong blogroll of 40 sites. The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy. Are there any good tools that will a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.) b) same, but export as OPML so you can subscribe. Thanks! Scott
Moz Pro | | scottclark0