Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
IP Canonicalization for HTTPS site?
I received an unsolicited SEO report for one of my sites. My site was faulted for not having IP canonicalization set up. I reviewed this carefully. My site runs on Apache, is https and is on a dedicated IP. The mod rewrite rules for Apache all deal with the http version of the site. When I type my site's IP into a browser, I get the the https version, but with a unsecure cert warning as the certificate does not include the IP. Should I implement the http IP canonicalization rule. Another rewrite rule would then redirect the request to the https version?
On-Page Optimization | | FatRodent20130 -
A Hummingbird Question
So with Google SERPs looking for more customer intent and especially to direct questions eg, How much does a x cost? Where can I find x etc - is it good practice to create a bank of Q&A's on site or is it sufficient to have good quality content on the subject matter stating the costs of x and where you can find it written in a customer friendly manner? Will a bank of Q&As add any value if they directly answer the most likely search questions? Thanks Ash
On-Page Optimization | | AshShep10 -
Author snippet image in results, question!
Hi, Quick question really, I can see how a Google+ profile picture showing up in the results can increase CTR and even trust but would this be a no no for a normal website? I mean if you had a website which offered personal training, so a brand rather than an individual would you not use this snippet? I've seen some website, even ecommerce sites using this tag to just increase their CTR.
On-Page Optimization | | Bondara0 -
Couple of noob questions
Hi, couple of easy questions for you guys... Im a web designer and my #1 keyword is web design huddersfield. I own the domain huddersfield.com, and i will have a page titled web design. Now do i target the web design page or the index page as i don't want keyword cannibalization, but is it better for my index page to be the landing page? #2 Considering that huddersfield.com is my domain should i set the title to: web design huddersfield or just web design? please help.
On-Page Optimization | | klsdnflksdnvl0 -
E-commerce site product descriptions and duplicate content
Hi everyone. I'm developing an e-commerce site using Prestashop and concerned about the issue of duplicate content among product descriptions. My main concerns are: If there are 500 or more products and those product descriptions are obtained from a manufacturer or supplier's website hence running into external duplicate content issues. Internal duplicate content is also an issue, if there are multiple similar products and each product has the same description across several pages. What would be the best approach to eliminate the possibility of incurring a duplicate content penalty due to similar product descriptions? I've already considered the suggestion of noindex-ing the complete range of products to help protect from duplicate content penalties and having unique articles written in the site blog discussing products instead linking to certain products on the site. Another consideration I had was noindex-ing all product pages except pages for featured products in the store and rewriting descriptions for a set amount of those featured products regularly (this will still have the problem of internal duplicate content across pages if similar product descriptions are rewritten). The product range is intended to be very large so I'm really seeking an alternative solution from the insane task of rewriting many product descriptions. Any suggestions to make SEO work efficient are very much welcome and appreciated. Thank you!
On-Page Optimization | | valuepets0 -
Site Maps / Robots.txt etc
Hi everyone I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/ Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both? Many thanks, Abi
On-Page Optimization | | LockCity0 -
Which pages on my site should I back link to
The majority of the back links I have been creating link directly to our home page and to the store page. Is this the best approach or should I be trying to spread the links throughout our site to include product categories and subcategories etc?
On-Page Optimization | | Hardley0 -
What is wrong with this site
Hello, I need advice regarding design ( current design as well as any changes ) and seo advice as to what we should do to promote the site. The site in question is - Allkindofessays.com Thanks
On-Page Optimization | | seoug_20050