Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wordpress #comment links?
We just started our trial account and have the results from our first Moz Pro Site Crawl. It's showing that we have a large portion of our pages have 'Too Many Links' and I'm trying to determine exactly what this means and how to fix it. The article referenced is from 2011 and doesn't fully address what I'm looking for. Here are a few questions: 1. Can we lower the 'link count' by adding a 'no follow' or does too many links, count links regardless? The question being, is the only way to solve this by removing links or are their no follow or no index options that will prevent us from having this issue moving forward? 2. Comment Links: Our site is in Wordpress and I just recognized that each of the comment links are followed: https://screencast.com/t/b0CIKVafWw. These aren't links from our users, rather these are links within Wordpress and are structured like this: https://mysite.com/blog-post/#comment-6970. From my screenshot you can see they are highlighted as 'followed links'. Is there a setting within Wordpress to turn this off or is there another option I should consider? Should we just make these no index, no follow links? Will that solve the 'Too many links' problem? I searched through the Q&A's and couldn't find an answer directly to my question. Most were around people leaving links in the comments section, which isn't what I'm looking for. Thank you for any help you can provide.
On-Page Optimization | | FabulesslyFrugal0 -
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
Best way to separate blogs, media coverage, and press releases on WordPress?
I'm curious what some of your thoughts are on the best way to handle the separation of blog posts, from press releases stories, from media coverage. With 1 WordPress installation, we're obviously utilizing the Posts for these types of content. It seems obvious to put press releases into a "press release" category and media coverage into a "media coverage" category.... but then what about blog posts? We could put blog posts into a "blog" category, but I hate that. And what about actual blog categories? I tried making sub-categories for the blog category which seemed like it was going to work, until the breadcrumbs looked all crazy. Example: Homepage > Blog > Blog > Sub-Category Homepage = http://www.example.com First 'Blog' = http://www.example.com/blog Second 'Blog' = http://www.example.com/category/blog Sub-Category = http://www.example.com/category/blog/sub-category This just doesn't seem very clean and I feel like there has to be a better solution to this. What about post types? I've never really worked with them. Is that the solution to my woes? All suggestions are welcome! EDIT: I should add that we would like the URL to contain /blog/ for blog posts /media-coverage/ for media coverage, and /press-releases/ for press releases. For blog posts, we don't want the sub-category to be in the URL.
On-Page Optimization | | Philip-DiPatrizio0 -
H1 tag in wordpress
Hi there! I'm having problems to locate the h1 tag in wordpress (in each page) in order to optimize the pages for SEO. When introducing text it starts with h2. Where can I locate it? Thanks in advanced.
On-Page Optimization | | juanmiguelcr0 -
1st attempt at schema - does it look correct?
My name is Ken Jansen, Here is my homepage: [www.kansascityrealestate411.com](http://www.kansascityrealestate411.com). I work in Overland Park, KS and work as an real estate broker at Keller Williams Realty Partners, Inc.. Trying to convey the following information Name: Ken Jansen Work Location: Overland Park, KS Occupation: Real Estate Broker Homepage: www.kansascityrealestate411.com
On-Page Optimization | | Ken_Jansen0 -
Duplicat contents on wordpress
I ran a crawl error and found that I have many pages with "tag" i.e. http://www.soobumimphotography.com/tag/70-200-2-8-is/ What's the best way to deal with this problems? Is it worth to visit all of them and fix? Delete? Could you give me some suggestions?
On-Page Optimization | | BistosAmerica0 -
Home Page Optimization For Wordpress Blog
Hi, I'm developing a wordpress site and currently confused over home page optimization. I'm planning to use excerpts of posts on home page so that users can glance over all recent posts and at same time avoid content duplication. But I'm not sure how to optimize the home page for couple of targeted keywords . How can i make sure that there is enough content, proper H1, title tag etc on home page for target keywords even by using excerpts of different posts on home page with Read more links to complete posts? Should i use some sticky post on home page to take care of all onpage requirements for required keywords or there are some other better ways to do it? Regards, shaz
On-Page Optimization | | shaz_lhr0 -
URL structure for a new WordPress site
Hi I'm building a new next big thing website from scratch (for a translation agency) and I encountered an issue with the URL structure. I need to chose the URL for important targeted keyword pages and I have a conflict between two tools I'm using. Please read below the situation: domain: mashtranslation.com target keyword: french translation services which URL you think is better from a SEO point of view (and possibly for users): mashtranslation.com/services/french/ OR mashtranslation.com/french-translation-services/ I'm asking this because one WordPress plugin (Wordpress SEO by Yoast) says the URL structure is not optimised while another tool (Market Samurai) says the URL is optimised.
On-Page Optimization | | flo20