The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do a bunch of footer internal links help or hurt?
We are an ecommerce site... In days gone by, having a bunch of footer links with your top products / categories was a good idea - as it created a ton of internal links to these products. Now, I am hearing that those links "dilute" the value of our other links on a page - and essentially, there is more harm than good from these. Does anyone know what I am talking about (the olds days) and should we still be doing this? Thanks
On-Page Optimization | | Ted_Cullen0 -
Static or Dynamic Home Page? Please Help!
Hi again 🙂 I ask for ur input again my friends I finally decided to go for the shorter .COM domain, and I put my domain online. Since it will be an online games site, every post will be in fact a new game. And of course, every game, will be in fact a post. I put my home page as "latest posts" this way, all my posts are shown in home page, and posts being for me games, all my games appear on my home page. So my home page always has the latest games and is always fresh. The theme I use has the option to ad some "static content" at the end of the posts, on home page, for SEO. I my home page I have also about 300 words of "article" content, that never changes, and is for seo, since my post content, is in fact java script, and other html codes ( flash games). So my home pages is that article plus excerpts of latest post. But the problem is, that I cannot control the seo, on page this way, and I cannot control my keywords very well, or the density of them, since the home page isnt a page. Now the question is, to let my Home page as it is, or change it and set it up as a page. I can put games on pages to, that is not a big problem. What is best, for the future of the site, google ranking, and seo, in ur opinion. I ad that I recived "A" Grade on moz for Page Seo Grade! Thank You!
On-Page Optimization | | Catinas970 -
Can somebody help me with a "Grade F" report
My Seomoz account tells me i've got a Grade F for my on-page optimalisation. The report said there's no single "on page keyword" usage at the whole page. Can somebody tell me what went wrong? If you take a look at my website: www.oceandrivers.nl, you'll see that i've used the keyword "prive chauffeur huren" everywere. In the URL, the H1 etc. (See image)
On-Page Optimization | | OceanDrivers
So i don't get it?! Thanks in advance! [](<a href=)" target="_blank">a> visWA visWA
0 -
Large block of several thousand words on homepage of Ecommerce site - opinions?
I work for a local SEO company who are continually adding content to the home page of clients websites. While i do agree that it is a good idea to add content to the home page and to link to inner pages, the home page of several clients exceeds 6,000 or more words. Every month, an article based on a brand or category (which already has its own page which is optimised with an article) is added to the home page with links to inner pages. I have stressed that it is not a good idea to have so much content on the home page, and even more so that it shouldn't target all the same keywords. I have pointed out that many of the brand and category pages do not rank well for their keywords, whereas in most cases it is the home page which is ranking instead, which i have suggested this is because the home page is too well optimised for those keywords. What are your opinions on this? Are you for or against continually adding content (which already has its own designated page) to the home page of an Ecommerce website? I should also add that this content is within a drop down div box at the footer on some sites and just above the footer on other sites. The category and brand pages also have drop down divs with an article just below the header.
On-Page Optimization | | Kinsel0 -
SEOmoz's On-page Checker upto date?
Helllo Mozzers, Just wondering if SEOmoz's on-page optimisation checker is upto date with google recent updates? If not... what do you suggest?
On-Page Optimization | | Prestige-SEO0 -
Duplicate content issue in SEOmoz campaign.
Hi, We are running a campaign for a website in SEOmoz. We get a dup content issue warning: http://www.oursite.com and http://www.oursite.com/ are being seen as 2 different urls. Only difference among 2 urls is the trailing slash at the end of the second url. Why is this happening? I was aware of www vs non www but never heard of an issue related to the slash. Thanks for your help!
On-Page Optimization | | gerardoH1 -
Is there a tool out there I could use to help me compose unique meta tags in bulk?
We have a website that has hundreds of crawl errors due to duplicate meta tags. I could do with a tool to help compose unique ones in bulk so we don't exceed the recommended character limit and follow any other best practices.
On-Page Optimization | | WebDesignBirmingham0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0