The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hreflang - Is it needed even if the site is only one language
This topic came up in a discussion I had with a fellow SEO colleague, I don't believe it makes sense to have Hreflang if you don't have a second language but according to my friend they mentioned that it is great if your only targeting one country. Any opinions out in the Moz community? It seems like overkill to me
On-Page Optimization | | JonAlonsoCNC0 -
Does name of town in title tag help if queries don't include the town name?
Hi. Wanted to know if targeting local traffic online and the search volume of KWs in the area do not include the local names (according to KW planner) does it still help to keep the town names in the title tag? does google deliver local results based on location names in title tag if query didn't mention it?
On-Page Optimization | | Morris770 -
Neglected Blog SEO help
I am wedding & sports photographer in San Antonio. I've been busy with photographing sports (lots of them) & totally neglected SEO work for nearly 2 years. I know it's a broad question but what is the best way to get back on track? Should I create a strong landing page with keyword wedding photographer? Should i make separate websites - one for weddings, one for sports? Is PPC good way to jump start? Do I have website structure problems etc? If anyone can comment, that would be great. I am willing to hire SEO experts so if anyone knows anybody, please let me know My website is www.soobumimphotography.com Thank you in advance
On-Page Optimization | | soobumim0 -
Need SEO Advice for different languages and How to optimize the page
Hello. Currently we have an ecommerce site that sells shoes online in France. The site is in English. Say the keyword is shoes and our page is example.com/shoes it;s perfectly fine to optimize that page for keyword of "shoes". Now, what about french keyword of shoes "chaussures" ? How do i optimize it? Should i create a different page or just edit the same page with keyword of "chaussures" in title tag etc? What is the best way to go about doing this? Thanks
On-Page Optimization | | WayneRooney0 -
Why is SEOMOZ Crawl Diagnostics not in sync with Webmaster Tools
Currently, my Website, according to the Crawl Diagnostics Summary, has 401 'Duplicate Page Title Errors'. But in Google Webmaster Tools, under Óptimization on the Left hand Side Toolbar, if you look up HTML Improvements, there are only 4 'Duplicate Title Tags'. I have two questions re this: A) Do 'Duplicate Page Title Errors' and 'Duplicate Title Tags' have the same meaning' ? , and B) why are there 401 errors located by the former, and just 4 by the latter?
On-Page Optimization | | ABCPS0 -
Followed seomoz advice for on page tweaks but dropped
Hi all We followed seomoz advice regarding on-page tweaks for this page: http://www.compactlaw.co.uk/agency-agreement.html Dropped 10 places by Google UK. Obviously could be a coincidence, but any on-page critique very gladly received. (But also seem to have been dropped by Google across the board in the past week or so. Usual trouble of trying to find causes. But specific page advice would be a start and much appreciated.) Regards P
On-Page Optimization | | dexm100 -
Canonical URL tags help I am not sure what this is
I am trying to get an A grade on my webpage and this is one of the critical steps canonical URL tags I cant find much information as to what this even is never mind fixing it. Thanks I am a total newbe at this any advice is appreciated
On-Page Optimization | | gemfirez0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0