The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best On page SEO guides needed.
I know my question looks so basic but let me explain what I want. I want advanced on-page SEO guides, should include something new exce[t the below listed things: title, description keyword in article content length video images with alt tag headings external authority links internal links please share something new....
On-Page Optimization | | hammadrafique0 -
Does my site just need a tweak, or should it be rebuilt from the ground up?
My wedding photography website has grown organically over the last 4 years or so.
On-Page Optimization | | chuckstar_za
Each year it's been fiddled with, changed, tweaked, prodded - in short it's been mucked about with a lot.
To the point now where I think it's become a hodge potch of mucky code (theme installed on theme in stalled on theme and numerous plugins) and no real 'structure' to it. At the moment it hovers around #12 for my main area of focus - people looking for a wedding photographer in Essex, England. So 'wedding photographer Essex'. I try to write blog posts regarding venues in Essex where I've photographed weddings. Despite trying to follow best practice, and with weak/low competition, these posts very rarely crack top 10. Last year when I was top 2 for wedding photographer Essex I was victim of a spam link attack. At the time I asked for advice and was told not to worry as Google didn't count these links. Now I'm feeling that along with the bloated website, lack of proper site architecture and other little issues, these links are contributing to restricting my pages from ranking in top 10 spots. Backlinking, while always helpful, doesn't seem to play a major role when trying to rank for venue related searches according to backlink reports on competitors. I'm thinking that it might be an idea to rebuild the site and content from the ground up. This time with a proper site structure in place and conforming to best practice. Obviously this is probably the most extreme course of action. Or does my site just need a little TLC to get it ranking better?
The site in question is:
www.alexkilbeephotography.co.uk Thanks1 -
Avoid Keyword Self-Cannibalization. Please Help
This has been addressed plenty of times but I cannot find an example that addresses my issue so that is why I am posting this. I am getting the following Self-Cannibalization error for my Homepage and I am trying to fix it but I just don't see it or maybe I do not understand this correctly. http://fake-diploma.com Keyword Fake Diploma Cannibalizing link"How to make a fake diploma", "How to get a fake diploma", "Making a Fake High School Diploma", "Fake Diploma Template", and "Framing your fake diploma" My understanding is that for Self-Cannibalization to occur I would have to have a link on this page pointing to another page using "Fake Diploma" as my anchor text since I want this page to rank for Fake Diploma. I do have the left sidebar which contains my most recent post and my of my titles do include Fake Diploma but I thought that since they were not an Exact Match and are actually longtail keywords that they did not matter or cause Self-Cannibalization. Am I wrong? How do I fix this.
On-Page Optimization | | diplomajim0 -
Hi can anyone help me with navigating through my business admin page? having a little trouble. Many Thanks
It looks like this... The person that created it has put the term "Hand painted ceramics from spain - inspired ceramica" on every page title and i cant find where it is to delete it. If anyone can help that will be a real help. Many Thanks, Robbie oOU4YDY
On-Page Optimization | | allenr120 -
Help, a certain directory is not being indexed
Before I start, dont expect this to be too easy. This really has me puzzled and am surprised I am still yet to find a solution for it. Get ready. We have a wordpress website, launched over 6 months ago and have never had an issue getting content such as pages and post pages and categories indexed. However, I some what recently (about 2 months ago) installed a directory plugin (Business Directory Plugin) which lists businesses via unique urls that are accesible from a sub folder. Its these business listings that I absolutely cannot get indexed. The index page to the directory which links to the business pages is indexed, however for some reason google is not indexing all the listing pages which are linked to from this page. Its not an issue of the content being uncrawlable or at least dont think so as when I run crawlers on my site such as xml sitemap crawlers it finds all the pages including the directory pages so I am sure its not an issue of the search engines not finding the content. I have created xml sitemaps and uploaded to webmaster tools, tools recongises that there are many pages in the xml sitemap but google continues to only index a small percentage (everything but my business listings). The directory has been there for about 8 weeks now so I know there is a issue as it should of been indexed by now. See our main website at www.smashrepairbid.com.au and the business directory index page at www.smashrepairbid.com.au/our-shops/ To throw in a curve ball, in looking into this issue and setting up tools we noticed a lot of 404 error pages (nearly 4,000). We were very confused where these were coming from as they were only being generated from search engines - humans could not access the 404s and so we are guessing se's were firing some javascript code to generate them or something else weird. We could see the 404s in the logs so we know they were legit but again feel it was only search engines, this was validated when we added some rules to robots.txt and we saw the errors in the logs stop. We put the rules in robots txt file to try and stop google from indexing the 404 pages as we could not find anyway to fix the site / code (no idea what is causing them). If you do a site search in google you will see all the pages that are omitted in the results. Since adding the rules to robots, our impressions shown through tools have jumped right up (increased by 5 times) so thought this was a good indication of improvement but still not getting the results we want. Does anyone have any clue whats going on or why google and other se's are not indexing this content? Any help would be greatly appreciated and if you need any other information to assist just ask me. Really appreciate anyone who can spare their time to help me, I sure do need it. Thanks.
On-Page Optimization | | ziller0 -
Can somebody help me with a "Grade F" report
My Seomoz account tells me i've got a Grade F for my on-page optimalisation. The report said there's no single "on page keyword" usage at the whole page. Can somebody tell me what went wrong? If you take a look at my website: www.oceandrivers.nl, you'll see that i've used the keyword "prive chauffeur huren" everywere. In the URL, the H1 etc. (See image)
On-Page Optimization | | OceanDrivers
So i don't get it?! Thanks in advance! [](<a href=)" target="_blank">a> visWA visWA0 -
Google Results Title Tag HELP
Can anybody tell us why Google changes your title tag in the SERP? If you check out the below link or type in 'days inn', you will see the 2nd result for www.daysinnrc.co.uk just says 'Days Inn' but on the actual site the title tag for this page is 'Days Inn UK | Days Inn | Daysinnrc.co.uk' http://www.google.co.uk/#hl=en&sclient=psy-ab&q=days+inn&oq=days+inn&gs_l=hp.3...4110.4110.4.4297.1.1.0.0.0.0.0.0..0.0...0.0...1c.1.kWVC24EnCHE&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&fp=7680231318a44bb0&bpcl=35466521&biw=1920&bih=934 This has happened with another site too, does anybody know why? Thanks
On-Page Optimization | | SEOwins0 -
Sitemap Help!
Hi Guys, Quick question regarding sitemaps. I am currently working on a huge site that has masses of pages. I am looking to create a site map. How would you guys do this? i have looked at some tools but it say it will only do up to 30,000 pages roughly. It is so large it would be impossible to do this myself....any suggestions? Also, how do i find out how many pages my site actually has indexed and not indexed?? Thank You all Wayne
On-Page Optimization | | wazza19850