Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: / -
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Not Ranking for Key Term
Question for my fellow Mozers I have a ranking question that I cannot put my finger on. I have a site (visitplano.com) where the client wants to rank for the keyword "Plano". I can't say if the site was previously ranking for this keyword, but I looked into the basic SEO practices and found that the keyword is incorporated in: Domain Title Content There is a lack of internal linking and anchor text within the content External links - 1,558 DA - 46 PA - 55 Currently, the website does not rank for the keyword "Plano". Could someone shed some light on why they aren't ranking or what I may be missing? I would greatly appreciate your help.
On-Page Optimization | | flcity150 -
To change or not to change site URL structure?
I am learning my way around SEO, having always used professionals for it in the past on previous businesses i have decided to do it myself and learn more about it. Now the dilemma i am up against is i recently changed some of my permalinks on quite a few main pages throughout the site. The site launched in April this year so we're quite new. The problem is since my last change i have not seen any increase...a decrease which in fact hasn't recovered at all. Having now analysed them in more depth and read up more on the whole subject of SEO, (which is endless) i have put together a complete new strategy; with this increased understanding of what i am doing (but by no means conclusive) i want to complete a full overhaul on all SEO (via Wordpress which i use along with YOAST SEO tools), ensuring i have all my keywords, permalinks and descriptions spot on throughout every page, post and picture. I spent a lot of time mapping these out, ensuring there is no Focus keyword duplication, and that the site is relatively flat in terms of its layout. What i am unsure about now is whether changing my permalinks again is a bad thing to do?
On-Page Optimization | | MrCostello
Could it permanently damage my rep going forward?
Should i just focus on my content and keywords/descriptions? I am at a loss as i don't want to do irreparable damage to our reputation. The site is still reasonable easy to manage so changing now is the best time to do it, but if changing the URLs is a waste of time then i may just forget that and just work on the keywords, descriptions and content. Advice is 'oh so welcome' 🙂0 -
Site structure suggestions/feedback
I asked this on Reddit and got some some decent answers. I'm curious to see what the pro's of SEOmoz think. I've got a lead generation site for forklift parts--liftxparts.com. You can think of it similar to car parts, where we have sections for specific brands (e.x. Toyota forklift parts) and sections for specific categories (e.x. forklift filters). Right now, the site is structured in two main levels: the top level is a dozen or so brands (separate pages for Toyota forklift parts, Clark forklift parts, etc), and then the second level is the categories (separate pages for a dozen or so different categories like forklift filters, forklift engine parts, etc.).If you check out one of the pages, like Clark forklift parts for example (our top landing page)--liftxparts.com/clark-forklift-parts.html, you'll see that on the brand pages (they're all structured the same), we list all the different categories (with links to the same second level category pages) and "search" buttons. All pages point to the same lead capture form.This has been working pretty well--about 90% of visitors end up on our lead capture form, and a high percentage of those convert. We're working on increasing organic traffic now and I'm thinking our structure could use some improvement.Looking at the analytics, there are a lot more impressions for keywords like "clark forklift" than "clark forklift parts". One gap I've uncovered is while our average position, and by extension CTR and traffic, for phrases like "clark forklift parts" is quite good, it's not so good for broader and higher searched terms like "clark forklift". Should we add another level of hierarchy targeted to just general brands? So now we have content for clark forklift parts, but should we add a page for terms like "clark forklift"? Or should we just add some broader content to the existing brand pages? The pages are quite long already, I'm afraid adding more content to the bottom of the page isn't very functional. Our thinking is that we can increase average position for higher searched terms by adding content targeted to those terms. The question is how exactly to go about it and how to work it into our current site structure? Any feedback related to our site structure or even just related ideas about other ways to approach our goal of increasing organic traffic would be very much appreciated! Thanks!
On-Page Optimization | | wisamabdulla0 -
What are the best eCommerce sites from an SEO perspective?
We're working hard on improving our website right now, and would love to get the community's examples of the best eCommerce sites out there, from an SEO and a general customer-centric design perspective...
On-Page Optimization | | reddogmusic0 -
Sudden Site Rankings Drop
Good day guys, We have been following strict SEO strategies for the past 6 months, all sites have been improving incredibly well, all except one. The site in question is http://bit.ly/IH4pkM . The site is regarding automotive spray booth equipment. We were ranking on the first page for the keyword "spray booth" (which is the most important one), at place #4 for weeks on end. However since half-way last week, the site has been dropped to half-way the second page (#17). There are barely any crawler errors listed for our campaign on SEOMoz. There were several pages of which the meta description was missing, but that has been fixed earlier this week. When it comes to link building, I looked at what the top competitors were doing, and was looking for unique link building opportunities myself. We have received 0 webmaster tools warnings as well. I do not believe we are penalized due to the "penguin" update. After all, if you search for for the company's name in Google, it is still listed on there (# 2). Nor have we been part of dodgy link networks at all. So my question is, what do you guys believe made us drop the rankings? Is there some on-page issues I am overlooking? Any recommendations to restore out previous rankings? Kind Regards, Roderic
On-Page Optimization | | Michael-Goode0 -
WordPress (.com) and SEO
I am in my 30 day trial and very interested in my results. I think I am probably in a small minority in having the same web site up and running for approaching 17 years (registered in January 1995 :)) but only now am I looking at SEO seriously (to the extent that I want to learn more myself, as opposed to having others promise great fortune!)). Anyway, before committing to SEOMoz on an ongoing basis I want to understand just how actionable the information on my dashboard is. With that in mind, here's the first of what is (hopefully) a series questions that about low-hanging fruit I might be able to check off quickly. I recently brought up a new blog on WordPress.com (note - hosted by WordPress, not a self-hosted implementation). I have had this blog running for less than a month and have just 18 posts. And I am being overwhelmed with thoudands of errors/warnings from SEOMoz. These fall into a few categories: Duplicate content: As I understand it, each TAG I associate with a single blog post creates a unique URL. For example, if I have a single post with tags for "flowers", "wine" and "cakes", I get URLs generated such as <blog url="">/flowers, <blog url="">/wines and <blog url="">/cakes. Obviously, tagging posts is a common scenario. Must I just accept these duplicate content warnings?</blog></blog></blog> Title element too long: These are self generated by WordPress.com and the default format includes the date the post was submitted (which takes a bunch of characters followed by the title used). Many of the posts are well over 70 and this seems really easy to do. Missing meta-description: As far as I can tell, Wordpress.com doesn't give me an option to specify these. So, must I just accept these issues if I use WordPress.com (which, again, seems like a very common scenario) and how negative is this to me? Thanks. Mark
On-Page Optimization | | MarkWill0 -
Advice needed about site
I would like to get advice about the site http://www.wb-3d.com regarding what we should do to get it highly on Google. This seems to be a tough one as there is not enough content on the site. I would immensly appeciate any suggestion to improve the rankings. Specifically, what we should tweet and update on facebook. Thanks
On-Page Optimization | | seoug_20050 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0