Correct robots.txt for WordPress
-
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong?
I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right.
Thank you so much.
-
Me too, can you upload or screenshot the actual file that you are using
-
I have edited it down to
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /contact/ Disallow: /refer/ It didn’t help. I get a “Blocked by robots.txt” message after submitting the URL for indexing in google webmaster tools. I’m really puzzled.
-
Hi, in addition to the answer that effectdigital gave; another option,optimised for WordPress:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml -
Just seems overly complex and like there's way more in there than there needs to be
I'd go with something that 'just' does what you have stated that you want to achieve, and nothing else
User-Agent: *
Disallow: /wp-content/plugins/
Disallow: /comments
Disallow: /*?s=
Disallow: /*&s=
Disallow: /search
See if that helps
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Robots index & noindex Both Implemented on Website
I don't want few of the pages of website to get indexed by Google, thus I have implemented meta robots noindex code on those specific pages. Due to some complications I am not able to remove meta robots index from header of every page Now, on specific pages I have both codes 'index & noindex' implemented. Question is: Will Google crawl/index pages which have noindex code along with index code? Thanks!
On-Page Optimization | | Exa0 -
Wordpress / Full URL In Menu Box
I came across an article online (not Moz) that says adding the full url in a menu is a Google standard. So when you make a menu link you put "www.example.com/page" instead of "/page". What are your thoughts on this? Any real reason to? Y840pbN lrwZPDj
On-Page Optimization | | InfinityTechnologySolutions0 -
Title Tags not Displaying in SERPs Correctly
We have two websites that are very similar. One is .co.nz and the other is a .com.au. I've noticed some of our page titles are displaying differently in SERPs even though the title tags are formatted the same:e.g. The title: "PC Games | at Mighty Ape Australia" shows as: "PC Games | at Mighty Ape Australia - MightyApe.com.au"It's showing with extra text "- MightyApe.com.au". Whereas the NZ result for the title "PC Games | at Mighty Ape NZ" is fine i.e. displays as: "PC Games | at Mighty Ape NZ"Does anyone who why this might be happening?P.S. It's odd that in the Australian results the site links don't show up even though the sites are structured the same.
On-Page Optimization | | kevinliao0 -
How Do I Get Rel=Author to Work Via Wordpress SEO by Yoast?
I have read Yoast's tutorial and others on how to get rel=author to work properly on my site (http://injuryattorneyshouston.com). Here are the results I get from the Google Structured Data Testing Tool: "Authorship is working for this webpage." "rel=author markup has successfully established authorship for this webpage." "Publisher markup is verified for this page." All seems well except for this: hatom-feed | hatom-entry: | |
On-Page Optimization | | cbizzle
| Error: At least one field must be set for HatomEntry.Error: Missing required field "entry-title".Error: Missing required field "updated".Error: Missing required hCard "author". |
| Error: At least one field must be set for HatomEntry.Error: Missing required field "entry-title".Error: Missing required field "updated".Error: Missing required hCard "author". | Does anyone know how to fix this?0 -
How To Get GD Star Rating working in wordpress categories.
I want to use GD star rating so that stars show up in SERPS for my category pages. I've installed the plugin but can't see how to do it. Does anyone have any idea?
On-Page Optimization | | SamCUK0 -
How to ALT tag correctly many similar images?
scenario: I have a product page with some text about the product and 12-15 different images af the product: details, colours, different views... how should I name an alt tag them correctly, been them actually really similar? Is it keyword stuffing to use the same name containing a relevant keywords?
On-Page Optimization | | DavideM0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050 -
Wordpress when to use posts or pages
Hi Guys, I have a network of EMD sites that currently use a homepage and then we have a blog page which has 5-6 posts on. Is this the best way to do it with sites under 10-20 pages? Or should we create say 3-4 new pages/categories and drop the posts relevant to each page/category in there? Thank you Jon
On-Page Optimization | | imrubbish0