The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help recover lost traffic (70%) from robots.txt error.
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change. Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated. On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL? Thanks,
On-Page Optimization | | akin671 -
Web designer doesn't see duplicate pages or issues??? Help!
I have a website that has duplicate pages, and I want to fix them. I told my web guy and his response was: "we do not see any issues on our end. It is not hurting any ranking in the 3 major searches. We have no duplicate pages" Here's the site http://www.wilkersoninsuranceagency.com/ Here's the home page again http://www.wilkersoninsuranceagency.com/index.php Here's what MOZ say: Please see attached image. I'm not sure what to tell him, as i'm not a web person, but MOZ is telling me we have issues on the site and I feel like they should be fixed?? 7QWkVU0 tYCjV
On-Page Optimization | | MissThumann0 -
URL structure of the page: Does this one need to contain the most important keyword for better SEO?
Hi everyone, I’m trying to get "air-conditioner-repair.html" to rank higher for the keyword "air conditioner los angeles". I am wondering whether or not I should change URL to "air-conditioner-los-angeles-repair.html" to get better results? Will be thankful very much for any advise you can offer!
On-Page Optimization | | kirupa0 -
Help with the indexation of my page
Hi all, I have a problem with my website. When writing site:www.pinesapiensa.com there're no pages indexed although the webmaster tools tells me that the sitemap file has been processed in 13 May and the number of indexed paged are 21. ¿What could be happening? I have to mention that there are two domains "www.piensapiensa.es" and "www.piensapiensa.com" addressing the same website and there's a redirection from piensapiensa.com to piensapiensa.com but it doesn't work properly. Thanks
On-Page Optimization | | juanmiguelcr0 -
Site Maps / Robots.txt etc
Hi everyone I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/ Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both? Many thanks, Abi
On-Page Optimization | | LockCity0 -
Website pages missing from seomoz crawl
Hi! I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing? Thanks!
On-Page Optimization | | ovieira0 -
Can you help with quality and interesting content ideas?
Hi I'm ranking for "online biology degree" and "online wildlife biology degree" but I have bad content and I receive almost 100% bounce rate. Can you help me with ideas on good content and interesting information I can provide for people looking for "online wildlife biology degree" and "online biology courses" "biology degree online" Every idea would be appreciated. Yoseph
On-Page Optimization | | Joseph-Green-SEO0 -
Can't canonical, but need pages to show in Google News
We are a news media site in which much of our content is third-party, and already published by several other sources. Our current version of our CMS doesn't expose head tags, so I can't canonical to the original and avoid a duplicate content penalty. Is it ok for news sites NOT to use canonical, or do I have to NOINDEX until our CMS is fixed?
On-Page Optimization | | Aggie0