Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rel-nofollow for price comparison site?
I run a price comparison site, so we have TONS of outbound links. Should my outbound links be marked with 'rel=nofollow'?
On-Page Optimization | | lancerpanz0 -
Moving Site from HTTP to HTTPS
Hi, So the news is that Google has started giving more importance to sites with HTTPS i.e. it is now a new ranking signal. It says that as of now it affects fewer than 1% of global queries, and carrying less weight than other signals such as high quality content but it may decide to strengthen it as they would like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web. In that case, what should we do? Switching from http:// to https:// means change in urls and low traffic. How to cope with it? Do we have to implement 'n' number of redirects? Regards,
On-Page Optimization | | IM_Learner3 -
Linking Out To External Sites
Hi, All If I have created a (website, logo, email campaign) for a client and written an article about it with screen shots on my website and link to them with a (do-follow) link how does Google see the (do-follow) link? Regards to the sites they have one link in the footer on the home page, which is a (do-follow) back to our site. Also, the websites are not in my Niche.
On-Page Optimization | | deskjet0 -
H1 tag in wordpress
Hi there! I'm having problems to locate the h1 tag in wordpress (in each page) in order to optimize the pages for SEO. When introducing text it starts with h2. Where can I locate it? Thanks in advanced.
On-Page Optimization | | juanmiguelcr0 -
How To Change Wordpress Category Title
My categories are indexed and I want to change the category page title. At present it just defaults to the category name but I want to set a different page title. For example I want the category to be 'Motor Cars' but I want the category page title to be 'Buy Motor Cars - New And Used'. How can I do this?
On-Page Optimization | | SamCUK0 -
What precaution should we take to change the default page of the site
For some reason we wanted to change the default page of my site from example.com to example.com/default.aspx. We will be using 301 redirection to get the back link benefits. Do we need to make any changes in webmaster tool and sitemap too??
On-Page Optimization | | CyrilWilson0 -
SEO Site Planning Tool?
Does anyone know of a good SEO Site planning tool? I see that SEOBOOK has something that looked interesting but they want $300/mo! Thanks in advance! Andy
On-Page Optimization | | MaxOtto0 -
Setting up a domain for a future site
Hi there, That may be a bit of a silly question to ask, but we've setup a new domain for an existing site. While the site is in the making, the site owners wants to already start promoting the new URL on stationeries etc. Hence, we need to setup the new URL so that it forwards to the site, but so that Google doesn't give it the history of a secondary (less important) domain. What is the best way to do this? Currently we've put in a 301 redirect, but will that bear no future consequences on the SEO of the site, when the site is moved to this new domain, and the old domain is 301 redirected. Thanks, SEOeclipse
On-Page Optimization | | Bozboz0