Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wordpress: Why do the URLs of my posts keep changing to match the posts titles?
I'll try to keep this brief. The URLs of my posts keep snapping back to exactly match their post titles, no matter how often I try to change them. e.g. title: How to Tie Your Shoelaces desired URL: tie-shoelaces BUT actual URL: how-to-tie-your-shoelaces Anyone come across/ resolved this issue before?
On-Page Optimization | | GerardAdlum0 -
Wordpress #comment links?
We just started our trial account and have the results from our first Moz Pro Site Crawl. It's showing that we have a large portion of our pages have 'Too Many Links' and I'm trying to determine exactly what this means and how to fix it. The article referenced is from 2011 and doesn't fully address what I'm looking for. Here are a few questions: 1. Can we lower the 'link count' by adding a 'no follow' or does too many links, count links regardless? The question being, is the only way to solve this by removing links or are their no follow or no index options that will prevent us from having this issue moving forward? 2. Comment Links: Our site is in Wordpress and I just recognized that each of the comment links are followed: https://screencast.com/t/b0CIKVafWw. These aren't links from our users, rather these are links within Wordpress and are structured like this: https://mysite.com/blog-post/#comment-6970. From my screenshot you can see they are highlighted as 'followed links'. Is there a setting within Wordpress to turn this off or is there another option I should consider? Should we just make these no index, no follow links? Will that solve the 'Too many links' problem? I searched through the Q&A's and couldn't find an answer directly to my question. Most were around people leaving links in the comments section, which isn't what I'm looking for. Thank you for any help you can provide.
On-Page Optimization | | FabulesslyFrugal0 -
Wordpress & SEO Markup
Buon Giorno from 6 degrees C wetherby UK 🙂 I'm close to commissioning a designer to create a wordpress driven site but I'm anxious around the standard of the markup. Specifically I need pages to have the classic unique title tags, one H1 tag per page and contnet in p tags.
On-Page Optimization | | Nightwing
Now I know theres a wordpress plugin claiming to empower the administartor to have control but are there any icebergs ahead I need navigate to ensure my sites meets Google optimization guildlines? Any advice welcome 🙂1 -
Wordpress SEO. How to add static content above home page posts.
I think I many have some duplicate content issues as have been adding unque content above posts in categories using the all category SEO. How can I add static content to the posts on the home page though? Any help appreciated!
On-Page Optimization | | SamCUK0 -
WordPress and category/subcategory landing pages
Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?
On-Page Optimization | | mihaiaperghis0 -
Site Maps / Robots.txt etc
Hi everyone I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/ Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both? Many thanks, Abi
On-Page Optimization | | LockCity0 -
Home Page Optimization For Wordpress Blog
Hi, I'm developing a wordpress site and currently confused over home page optimization. I'm planning to use excerpts of posts on home page so that users can glance over all recent posts and at same time avoid content duplication. But I'm not sure how to optimize the home page for couple of targeted keywords . How can i make sure that there is enough content, proper H1, title tag etc on home page for target keywords even by using excerpts of different posts on home page with Read more links to complete posts? Should i use some sticky post on home page to take care of all onpage requirements for required keywords or there are some other better ways to do it? Regards, shaz
On-Page Optimization | | shaz_lhr0 -
How do Maximize WordPress with 2 SEO Plugins
I have 2 WordPress SEO Plugins, Yoast and All-in-One SEO. I have tried like heck to make them work together, but every time I crawl my site here, I get multiple error messages. My question is, how can I tweak the title settings to avoid having multiple meta desctiptions, titles etc.
On-Page Optimization | | TheSportsDaddy0