The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi i have a few pages with duplicate content but we've added canonical urls to them, but i need help understanding what going on
hi google is seeing many of our pages and dupliates but they have canonical url on there https://www.hijabgem.com/index.php/maxi-shirt-dress.html has tags https://www.hijabgem.com/maxi-shirt-dress.html
On-Page Optimization | | hijabgem
has tagshttps://www.hijabgem.com/index.php/quickview/index/view/id/4693
has tags
my question is which page takes authority?and are they setup correct, can you have more than one link rel="canonical" on one page?0 -
Do you need Metadata on Password Protect Pages?
Hey guys and gals, I was wondering if it matters to have metadata on password protected pages or if they can be ignored. Thanks,
On-Page Optimization | | brightvessel0 -
Help optimizing website for speed
Hello, My website is www.likechimp.com and is a University project. I need to optimise the website for speed as the bounce rate is fairly quick - I feel this could be due to how long it takes web site to load? Any tips in increasing internet speed. I am willing to higher someone if they feel they can help! Thanks, L
On-Page Optimization | | xlucax0 -
Robots.txt Question for E-Commerce Sites
Hi All, I have a couple of e-commerce clients and have a question about URLs. When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout I'm not sure that I want these indexed. Could I be causing any harm/danger if I add this to the robots.txt file? /*? Any suggestions welcome! Gavin
On-Page Optimization | | IcanAgency0 -
SeoMOZ meta descriptions
Anyone notice that seoMOZ doesn't use meta descriptions? Any idea why not?
On-Page Optimization | | SoulSurfer80 -
Why doe SEOmoz include my categories and subcategories when counting on page links?
Hi everyone, I am new to SEOMOZ, but it looks like really good program. I ran a report last week, and SEOMOZ says that out of 10001 pages, I hve a total of 9995 warnings. When I take a look at the breakdown, it says I have too many on page links. I removed the links from the bottom of each page, but my on page links are still over 100. Does this mean that I will have to get rid of categories and sub-categories in the navigation bar? Thanks, Don
On-Page Optimization | | ge01734000 -
Why is SEOMoz telling me that meta descriptions are missing?
We have descriptions using an Ektron CMS on a .NET platform. IT is wondering if using "meta ID" vs "meta name" is the issue? Here is an example of the HTML code: <meta http-equiv="<a class="attribute-value">Content-Language</a>" name="<a class="attribute-value">Language</a>" content="<a class="attribute-value">en</a>" /> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaKeywords</a>" name="<a class="attribute-value">Keywords</a>" content="<a class="attribute-value">hunting, fishing, outdoor, sportshooting, rodeo, off-road, hunt, fish, outdoors, community, shotgun, rifle, bass, trout, sportfishing, tarpon, ATV, television</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaDesc</a>" name="<a class="attribute-value">Description</a>" content="<a class="attribute-value">Deep-cranking technique lands Starks his second Elite Series victory.</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaSubject</a>" content="" name="<a class="attribute-value">subject</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaRating</a>" content="" name="<a class="attribute-value">rating</a>">meta>
On-Page Optimization | | cmoise0 -
Do I need canonical link on target page?
I've placed in my head tag on duplicate content pages, but do i need to place it on the target page such as http://www.example.com/index.html too?
On-Page Optimization | | CaliB0