Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?
-
my site uses is set up at http://www.site.com I have my site redirected from non- www to the www in htacess file. My question is... what should my robots.txt file look like for the non-www site? Do you block robots from crawling the site like this? Or do you leave it blank?
User-agent: *
Disallow: /
Sitemap: http://www.morganlindsayphotography.com/sitemap.xml
Sitemap: http://www.morganlindsayphotography.com/video-sitemap.xml
-
Hi there
If you configured this properly, I wouldn't worry about this at all.
Check your internal links and sitemap to make sure that your URLs listed as a reflection of this www. version.
Beyond that, you're all good, no need to block non www.
Hope this helps! Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Updating 2013 Site Built with Custom Theme, Modify Existing Theme, Create New Custom Theme, Or Use Child Theme?
Our website was designed in 2013 using a custom theme. Some of the plugins are built from scratch. Ranking in our industry is hyper competitive. We are seeking a better interface and also to improve ranking. I have read that custom themes use lighter code and can rank better. Does this apply to a custom theme from 2013? Will we have an SEO advantage using a custom theme? If so, will that advantage be significant? We are using a discontinued plugin called "Firestorm" to display real estate listings. That plugin has been customized. Can we use that plugin on a new "custom" theme? How about on a "child" theme? In terms of the cost of future maintenance, will a "custom" theme require much more intervention (manual installation of updates) moving forward? Which of the following options is best: 1. Adapt our existing custom theme
Intermediate & Advanced SEO | | Kingalan1
2. Create a new custom theme
3. Create a child theme Thanks,
Alan0 -
Robots.txt advice
Hey Guys, Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you? user-agent: AhrefsBot User-agent: trovitBot
Intermediate & Advanced SEO | | eLab_London
User-agent: Nutch
User-agent: Baiduspider
Disallow: / User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ? I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers?0 -
Disavow both www. and non www. version of site?
I just submitted my disavow file to Google after several months of work. A few months ago a saw a partial match unnatural link penalty in the www. version of my site's Search Console account. The penalty has since expired. Should I also upload the file to the non www. side of the Search Console account? No penalty ever appeared there.
Intermediate & Advanced SEO | | pajamalady0 -
What To Do About Yahoo Slurp Bot Bogging My Site Down?
Hello, Our IT department has informed me that they have seen extremely heavy traffic from the Yahoo Slurp bot in recent days. They are claiming this bot has single-handedly caused one of our servers to crash. I am a bit skeptical of this, as I have not found these particular legitimate search engine bots to be aggressive resource hogs, especially for an enterprise-level web server. I have requested to examine the server logs myself, but have not had success with this. IT is requesting to block this particular bot, but I am apprehensive about doing this, as I don't want this to have any negative implications on our site showing in Yahoo News or other Yahoo properties. Does anyone else have experience with this bot being an overly-zealous resource drag, and if so, what is the best course of action to satisfy all parties?
Intermediate & Advanced SEO | | RobbieFoglia1 -
SEO question regarding rails app on www.site.com hosted on Heroku and www.site.com/blog at another host
Hi, I have a rails app hosted on Heroku (www.site.com) and would much prefer to set up a Wordpress blog using a different host pointing to www.site.com/blog, as opposed to using a gem within the actual app. Whats are peoples thoughts regarding there being any ranking implications for implementing the set up as noted in this post on Stackoverflow: "What I would do is serve your Wordpress blog along side your Rails app (so you've got a PHP and a Rails server running), and just have your /blog route point to a controller that redirects to your Wordpress app. Add something like this to your routes.rb: _`get '/blog', to:'blog#redirect'`_ and then have a redirect method in your BlogController that simply does this: _`classBlogController<applicationcontrollerdef redirect="" redirect_to="" "url_of_wordpress_blog"endend<="" code=""></applicationcontrollerdef>`_ _Now you can point at yourdomain.com/blog and it will take you to the Wordpress site._
Intermediate & Advanced SEO | | Anward0 -
Robots.txt
Hi all, Happy New Year! I want to block certain pages on our site as they are being flagged (according to my Moz Crawl Report) as duplicate content when in fact that isn't strictly true, it is more to do with the problems faced when using a CMS system... Here are some examples of the pages I want to block and underneath will be what I believe to be the correct robots.txt entry... http://www.XYZ.com/forum/index.php?app=core&module=search&do=viewNewContent&search_app=members&search_app_filters[forums][searchInKey]=&period=today&userMode=&followedItemsOnly= Disallow: /forum/index.php?app=core&module=search http://www.XYZ.com/forum/index.php?app=core&module=reports&rcom=gallery&imageId=980&ctyp=image Disallow: /forum/index.php?app=core&module=reports http://www.XYZ.com/forum/index.php?app=forums&module=post§ion=post&do=reply_post&f=146&t=741&qpid=13308 Disallow: /forum/index.php?app=forums&module=post http://www.XYZ.com/forum/gallery/sizes/182-promenade/small/ http://www.XYZ.com/forum/gallery/sizes/182-promenade/large/ Disallow: /forum/gallery/sizes/ Any help \ advice would be much appreciated. Many thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Questions about turning my wordpress site into an ecommerce site. Experience needed.
I have a wordpress site that is about a product that is now getting some great traffic. Right now It has affiliate stuff on it. I want to sell my own product so I will be turning this wordpress site into an ecommerce site. I want to redesign it so I am not looking for simple plugins to just add a cart. The part I am really confused about is what to do with my posts and categories? How does that work when turning this site into an ecommerce site? Lets say the site is "hats for adults" My post pages are things like "funny hats for adults", "hats for adult men" etc etc. Would I turn these posts pages into like category pages that have a category of products. Or should I create real categories and have my developer turn those into the ecommerce category pages and then redirect my posts to those categories? Maybe I don't even know what I am talking about. Is this even making sense? This is a small site (5posts and 1 category) and most of the traffic will come from the homepage keywords anyways.
Intermediate & Advanced SEO | | PEnterprises0 -
Would the use of
Hi, I am wondering on you through relevant to SEO in the following situation. I have a "travel" website and obvisouls as part of that I have a whole list of desitinations. So I have a drop down in my page navigation, which lists all my desitinations. At the moment I see have 2 main options to display the lists as follows: 1/. Perfect Anchors, but not good for usability - IE repeating the word "holiday in a list of 100 destinations, looks spammy for one, and when the headline says "Holiday Destinations", then from a use perspective its pretty pointless and takes away from navigation rather than improves it".
Intermediate & Advanced SEO | | James77
New York Holidays
Las Vegas Holidays 2/. Non Perfect Anchors - But better for usability
New York
Las Vegas So I am thinking - would the use of the title attribute provide a perfect solution?? Or am I wasting my time with this and it is just pointless considering it as an option. EG - what I had in mind was:
3/. Ideal Solution for both SEO and usability??
New York
Las Vegas Thanks for you help in advance.0