The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical question - Would this create some sort of crawler redirection loop?
What happens if a canonical link, links to the url with / but the main url does not have the / For example: rel="canonical" href="https://www.exampleURL.co.uk/"> Main URL - https://www.exampleURL.co.uk (without the /) 301 Redirect https://www.exampleURL.co.uk/ to https://www.exampleURL.co.uk Would this create some sort of crawler redirection loop?
On-Page Optimization | | Evosite10 -
Help with Temporary Redirects on Pages
Hi Guys, My latest crawl shows this: "44% of site pages served 302 redirects during the last crawl" When I click this to investigate the issue I see: URL:
On-Page Optimization | | jeeyer
https://www.....
https://www.....
https://www.....
https://www.....
https://www..... And under Redirect URL:
http://www....
http://www....
http://www.... I've recently read an article from Yoast that this is a https:// to http:// redirect (ofcourse 😉 ).
but why is this an issue that needs to be solved and how do I solved it? Thanks again for your help and thoughts. Joost0 -
Magento Canonical & Default Robots Settings
Hello! I'm working with Magento 1.9 for an eCommerce site with several hundred products. Currently I understand it is best practices to use the Canonical tag, however I also have my default robots set to "Index, Follow". Will this cause an issue having product pages set to index, follow but also having a canonical tag included? What are some best practices regarding Magento default robots & canonical tags? Any help is appreciated.
On-Page Optimization | | BretDarby0 -
Google crawler showing cache of another page
For the page http://www.thinkdigit.com/top-products/Laptops-and-PCs/top-10-laptops-124.php google is showing another page in cache (http://www.thinkdigit.com/top-products/Ultrabooks/top-10-ultrabooks-153.php). Please let me know how this happened and how to correct it.
On-Page Optimization | | 9dot90 -
SeoMoz Pro - Keyword stuffing but its a forum
hiya guys Just going through the Seomoz Pro on page thing One of the flagged items is keyword stuffing, I have 22 keywords (stuffed) but they want it down to 15, the thing is, my website is a forum. and the keyword stuffing is from the title name of each categories eg its a nightlife forum so Birmingham nightlife forum Brighton nightlife forum Leeds nightlife forum Manchester nightlife forum u get the picture To avoid this keyword stuffing, should I change some of the forum names to eg leeds nightlife forum > leeds nightlife forums? just by adding the 's that will half the keywords, but its not solving the problem in the long run, if i was to add more cities and areas with the XXXX nightlife foru. What you reckon guys? Cheers guys Luke
On-Page Optimization | | Lukescotty0 -
On page Analysis... how does seomoz choose to match a keyword to a url?
On my "On Page" analysis report it show me a couple pages that have D ratings... one key phrase is "my destination lodging" and it is matching it up with a url like mydomain.com/my-destination-hotels.html. It says I'm ranking 37th for this term.... Is it just suggesting that I optimize this url because it is already ranking for that term? So should I create a different page with a different url to better match this phrase? Thanks for any ideas.
On-Page Optimization | | PillarMarketing0 -
What's the best practice for implementing a "content disclaimer" that doesn't block search robots?
Our client needs a content disclaimer on their site. This is a simple "If you agree to these rules then click YES if not click NO" and you're pushed back to the home page. I have this gut feeling that this may cause an upset with the search robots. Any advice? R/ John
On-Page Optimization | | TheNorthernOffice790