Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Need Wordpress Front-end Plugin With Moz API
Hi Guys,
On-Page Optimization | Nov 5, 2024, 11:59 AM | mrezair
I'm looking for Moz SEO Front-end Wordpress Plugin To Audit My Visitors Website And Show Results in my site. Like This plugin for Moz DA Checker: https://www.724ws.net/domain-authority-checker/ It's not important to be a free plugin or Premium one. I need to increase leads and traffic by it. Any suggestion will be appreciated.0 -
Best way to separate blogs, media coverage, and press releases on WordPress?
I'm curious what some of your thoughts are on the best way to handle the separation of blog posts, from press releases stories, from media coverage. With 1 WordPress installation, we're obviously utilizing the Posts for these types of content. It seems obvious to put press releases into a "press release" category and media coverage into a "media coverage" category.... but then what about blog posts? We could put blog posts into a "blog" category, but I hate that. And what about actual blog categories? I tried making sub-categories for the blog category which seemed like it was going to work, until the breadcrumbs looked all crazy. Example: Homepage > Blog > Blog > Sub-Category Homepage = http://www.example.com First 'Blog' = http://www.example.com/blog Second 'Blog' = http://www.example.com/category/blog Sub-Category = http://www.example.com/category/blog/sub-category This just doesn't seem very clean and I feel like there has to be a better solution to this. What about post types? I've never really worked with them. Is that the solution to my woes? All suggestions are welcome! EDIT: I should add that we would like the URL to contain /blog/ for blog posts /media-coverage/ for media coverage, and /press-releases/ for press releases. For blog posts, we don't want the sub-category to be in the URL.
On-Page Optimization | Apr 19, 2017, 5:17 PM | Philip-DiPatrizio0 -
Post url not matching with post title ( wordpress)
I have this site called searchoflife.com on which I have noticed the post urls are not matching with the post title. For Example:Post Title - A Dialogue With NaturePost URL - http://searchoflife.com/dialogue-nature-2013-09-12 Words like 'A' and 'with' are not present in the post URL. This has been the trend since a few days. After investigating I found out that it was due to my plugin SEO ultimate which is actually creating post slugs automatically for the post urls. So my question is whether it is advisable to use post slugs instead of the full post url. Does it affect the SERPS for my site?
On-Page Optimization | Sep 15, 2013, 6:07 AM | toxicpls0 -
Solve duplicate content issues by using robots.txt
Hi, I have a primary website and beside that I also have some secondary websites with have same contents with primary website. This lead to duplicate content errors. Because of having many URL duplicate contents, so I want to use the robots.txt file to prevent google index the secondary websites to fix the duplicate content issue. Is it ok? Thank for any help!
On-Page Optimization | Jul 16, 2013, 11:02 PM | JohnHuynh0 -
Disabling archives in wordpress
Hi! Do you think it better to: a) have all archives crawled (date, time, category etc.) even though they create duplicate content,. or; b) disable crawling of all archives? Ideally, I'd like to set up just excerpts in the archives and have a unique intro for each, but I'm having trouble doing this just now so was wondering which is the smarter option in the meantime? Thanks very much!
On-Page Optimization | Apr 21, 2013, 5:06 AM | LeahHutcheon0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | Mar 28, 2013, 5:29 PM | bricerhodes0 -
How to optimize a wordpress blog
I’m helping a client optimize a word press blog, and I’m not that familiar with Wordpress. The site is www.athleticfoodie.com. At first I was treating it like a normal website, where the categories would be optimized like pages on a website. However, I now realize that categories don’t have any content on them, so I can’t really optimize anything other than the names. Are the following things the best way to handle on-page optimization for a blog? Optimizing the homepage & domain: Find ways to incorporate the most important keywords into the elements on the main frame of the site: Navigation menu, Widgets, Category names, Alt Images. Optimizing the categories: For the posts within the categories (i.e., photos), work to make sure the category keywords are worked into the post titles (but not too much to seem spammy) Optimizing specific posts. Work keywords into the text and images. Any other suggestions would be greatly appreciated.
On-Page Optimization | May 10, 2011, 1:53 PM | EricVallee340 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | May 11, 2012, 5:56 AM | DisMedia0