Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Website title not showing correctly
Hi Im not sure what triggered this - Page title is showing up as 'Isagenix Australia' while it should be <title>Buy Isagenix with 30 Day money back guarantee & Free Shipping in June</title> Previous title was 'Isagenix Australia - Read Isagenix reviews & buy isagenix products' but now it seems to have a mind of its own. Any suggestions would be helpful
On-Page Optimization | | IsaCleanse0 -
WordPress image urls - need a WP maven
We were having a conversation re urls that are indexed for images that are stored in various media plugins in WP. My question for anyone who is an uberWP person is: What is your opinion re best media storage plugins and how these URLs affect pages on a site for ranking, etc. I realize this is broad, but it is driven out of my concern that I cannot touch everything. When I see a url like this: http://www.drumbeatmarketing.net/wp-content/themes/drumbeat2/img/DB-LOGO-White.png I know there is no way with all the sites and clients we handle that I can get it perfect but this just bugs me for some reason. Should I just chill since it (seemingly) affects so little....?
On-Page Optimization | | RobertFisher1 -
Wordpress sitemap url problem causing WMT errors
The following types of links are appearing in my webmaster tools crawl errors report under 'other'. I've noticed they are in my sitemaps ( I run wordpress and use a plugin called Google XML sitemaps). How do I get rid of this error? http://www.musicliveuk.com/bands/postname%/
On-Page Optimization | | SamCUK0 -
WordPress and category/subcategory landing pages
Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?
On-Page Optimization | | mihaiaperghis0 -
Duplicate Content from WordPress Category Base?
I recently changed my category base in WordPress and instead of redirecting or deleting the old base, WordPress kept the content up. So I now have duplicate content on two different urls - one on the old category base, one on the new category base. How should I handle this situation? The site is only a couple weeks old, if that makes any difference.
On-Page Optimization | | JABacchetta0 -
AD on wordpress
I notice advertisement hyperlinks on my site such as hyperlinks on my test gift, fill out etc. I never put hose links but somehow it shows up on firefox. Does anyone experience this? It's showing up this url http://www.soobumimphotography.com/portfolio/wedding/
On-Page Optimization | | BistosAmerica0 -
Impact of website redesign when changing to wordpress
Agency designed website and not sure what platform was used. Are now working minor redesign of site but wants to do so in word press. I don't have any problem with word press but don't want to start over ranking for key phrases. I don't want to change URL from mysite.com for the home page. What is the right way to make sure this is done correctly. I could not find a resource on the site. I don't mind researching this but am not sure what to search for to find info. Thanks
On-Page Optimization | | arainey0070 -
Optimizing Wordpress Categories
I’m trying to optimize the category links in the wordpress blog http://blog.reputationforward.com. I’m in the process of installing the Yeost SEO plugin and am going to revise the Category Title Tags, Meta Descriptions and slugs. I’m also in the process of using Tags so that I can get the posts that appear in a category to be specific to that category (currently the publisher is listing posts in multiple categories: See “Business Management Category”). But other than that, is there anything else I can do to optimize each category?
On-Page Optimization | | EricVallee341