Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What do you add to your robots.txt on your ecommerce sites?
We're looking at expanding our robots.txt, we currently don't have the ability to noindex/nofollow. We're thinking about adding the following: Checkout Basket Then possibly: Price Theme Sortby other misc filters. What do you include?
Intermediate & Advanced SEO | | ThomasHarvey0 -
How to add dropdown menus in the Wordpress Cutline theme?
I've got a fan website for a cult US TV show (that also sells merchandise as an affiliate). The site's design is about five years old, and I should probably revamp it at some point: http://www.btvsonline.com/ Anyway, the site's Cutline theme (http://cutline.tubetorial.com/) does not support native headers, and the header menu shows only a text list of my top-level pages. There are many subpages below these pages, but it's hard for users to find them without going to different pages to try to find things. What I'm looking for: if someone hovers above "Store" in the menu, a dropdown menu showing all the product pages would appear. I've tried to find Wordpress plugins or PHP hacks to add dropdown menus and I have gone through Cutline's support site, but I have had no luck. Anyone in the Moz community have any advice? Perhaps I just need to revamp the site with a new and modern Wordpress theme? Thanks in advance for any thoughts!
Intermediate & Advanced SEO | | SamuelScott0 -
Is WordPress a Blog in the eyes of Google?
Hi, My online Shop is based on WordPress with the WooCommerce plugin. Now, I have met a SEO guy who told me that's bad in the eyes of Google: Because Google apparently sees my website as a blog and not as a E-Commerce site. Wow, this statement really confused my, since I am working so hard on content and good rankings. Any opinions on this would be appreciated. Best, Robin
Intermediate & Advanced SEO | | soralsokal0 -
Changing domains - best process to use?
I am about to move my Thailand-focused travel website into a new, broader Asia-focused travel website. The Thailand site has had a sad history with Google (algorithmic, not penalties) so I don't want that history to carry over into the new site. At the same time though, I want to capture the traffic that Google is sending me right now and I would like my search positions on Bing and Yahoo to carry through if possible. Is there a way to make all that happen? At the moment I have migrated all the posts over to the new domain but I have it blocked to search engines. I am about to start redirecting post for post using meta-refresh redirects with a no-follow for safety. But at the point where I open the new site up to indexing, should I at the same time block the old site from being indexed to prevent duplicate content penalties? Also, is there a method I can use to selectively 301 redirect posts only if the referrer is Bing or Yahoo, but not Google, before the meta-refresh fires? Or alternatively, a way to meta-refresh redirect if the referrer is Google but 301 redirect otherwise? Or is there a way to "noindex, nofollow" the redirect only if the referrer is Google? Is there a danger of being penalised for doing any of these things? Late Edit: It occurs to me that if my penalties are algorithmic (e.g. due to bad backlinks), does 301 redirection even carry that issue through to the new website? Or is it left behind on the old site?
Intermediate & Advanced SEO | | Gavin.Atkinson0 -
Site Penalty After Changing Hosting Companies?
In one week's time, we've dropped from #3 on Page 1 of Google to Page 7 (similar on Bing). It looks like our traffic started to drop on 9/5 to 9/7 and has been a steady, rapid decline ever since. 1000s of pages are indexed, just suddenly ranking poorly -- even for branded terms. History:
Intermediate & Advanced SEO | | ddwilliamson
--In January, we switched to a web redesign & new domain
--In August, our hosting server was slow & kept crashing so we migrated our site to a new hosting company. We're not currently using the old hosting server. All domains, redirects, .htaccess files should now be correct and site speeds are improved.
--In early September, our NEW hosting company had a DNS issue causing more slow speeds and downtime for about 1 wk. Originally they thought it was htaccess so they changed our htaccess file - no luck - then discovered it was DNS. DNS issue was finally resolved on September 6th -- one day before the penalty/traffic issue seemed to begin.
-- According to GWMT, it looks like there were crawls completed around 9/4-9/5 What we've tried:
--Webmaster Tools - Googlebot dropoff since 9/5 (see attached screenshot). Nothing flagged. No site health alerts. Fetch as Google works. No manual webspam actions found.
-- W3C link checker, screamingfrog SEO spider, Xenu Link Sleuth, OSE (found some 4xx errors so we've updated those links)
-- Majestic SEO - backlinks reviewed 9/3 to 9/8
-- spoke to two different Adwords salespeople; unable to help
-- Bing Webmaster Tools
-- not showing organic search traffic since 9/6
-- 15% fewer pages crawled this month
-- top keywords are very odd -- stuff like "mt1 google apis" and "aaremel"
-- there are 4xx crawl errors under Crawl Information. We've fixed those URLs but they still appear in Webmaster Tools
-- some missing h1's and meta's, and dup titles, which we're working to fix
-- spike in crawl errors 9/11-9/12 and again on 9/14-9/15 It's been one thing after another this year, but all issues are now resolved with the exception of this newly-discovered penalty. We also have sites on a separate hosting server (with a different hosting company) that rank just fine. googlebot-crawls.jpg0 -
Changing Server IP Addresses. Should I be concerned?
Hello Mozers Our site has been on a dedicated server for about four years now. (no other sites, just ours on the server) I have made the decision to move it to a much better and faster server than the current server we are on for more than one reason. My big fear is Google will lose trust for my site because of the IP change. Ip's stay with the server at 1and1 they do not follow the website. So, I have done my due diligence and copied over all code and databases and have tested it completely to insure there are no issues when I change the DNS to point to the new server. Made sure 1and1 is giving me an IP that has never been used, I am Keeping the old server on until cached DNS records expire for it. Is there anything else I need to do to make sure I do not lose current rankings in Google? I have heard nightmare stories about making these kinds of changes but at this point for our site there is no turning back this is a change that must take place. Any pointers and advice would be much appreciated! Thanks!
Intermediate & Advanced SEO | | Robbie82991 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0 -
Robots.txt disallow subdomain
Hi all, I have a development subdomain, which gets copied to the live domain. Because I don't want this dev domain to get crawled, I'd like to implement a robots.txt for this domain only. The problem is that I don't want this robots.txt to disallow the live domain. Is there a way to create a robots.txt for this development subdomain only? Thanks in advance!
Intermediate & Advanced SEO | | Partouter0