Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to analysis Responsive VS Dynamic site?
Hello Experts, Can you please throw your knowledge on How to analysis Responsive VS Dynamic site? Also please suggest best tool to optimize both type of sites plus if I have to optimize manually then what is the best way? It will help lots of people as currently google focusing more on mobile site so we can optimize our responsive and dynamic site. Pls suggest. Thanks!
On-Page Optimization | | adamjack0 -
Can the Lightboxes on My Site be Crawled?
I'm trying to optimize my site, but I have lightboxes and I don't know if they are visible to the search engines. If they aren't, could you suggest something that I could do? THANK YOU so much!!!!! My site is lymphexpo.com
On-Page Optimization | | bosleypalmer0 -
Moving Site from HTTP to HTTPS
Hi, So the news is that Google has started giving more importance to sites with HTTPS i.e. it is now a new ranking signal. It says that as of now it affects fewer than 1% of global queries, and carrying less weight than other signals such as high quality content but it may decide to strengthen it as they would like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web. In that case, what should we do? Switching from http:// to https:// means change in urls and low traffic. How to cope with it? Do we have to implement 'n' number of redirects? Regards,
On-Page Optimization | | IM_Learner3 -
How much content does Google Crawl on your site?
Hi, We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us? Thanks,
On-Page Optimization | | mdorville
Matt0 -
Author snippet image in results, question!
Hi, Quick question really, I can see how a Google+ profile picture showing up in the results can increase CTR and even trust but would this be a no no for a normal website? I mean if you had a website which offered personal training, so a brand rather than an individual would you not use this snippet? I've seen some website, even ecommerce sites using this tag to just increase their CTR.
On-Page Optimization | | Bondara0 -
What precaution should we take to change the default page of the site
For some reason we wanted to change the default page of my site from example.com to example.com/default.aspx. We will be using 301 redirection to get the back link benefits. Do we need to make any changes in webmaster tool and sitemap too??
On-Page Optimization | | CyrilWilson0 -
Articles on our own site?
Hi Guy's Is it better to have articles on our own website under a news section, with keyword phrases that point to relevant pages or is it better to post them on our blog (that is seperate, we use blogger) and point them to relevant pages on our site? or maybe a bit of both? Thanks Daniel
On-Page Optimization | | LushDuck0 -
Very basic hands-on type of question about SEO
Hello, I am a complete newbie to the world of SEO and I have read few thing available on the net about things to begin with (so google seo guide and then seomoz beginers tutorial) and I just wanted to ask if I understand the process correctly: So I have my website lest call it abcd.com and there I might have a subpage about certain type of robots or specific parts that these robots are built from. I do my keyword search using adwords keyword search tool. Where I get 10 phrases (phrase1, phrase2, phrase3 so on..) that could be used by users to search information about certain type of robots I wrote on my website. Let's asume all 10 of those phrases have low competition so they can all be used. And they are from the long tail ofcourse so let's say I can get 10,000 searches from them per month. Some of them would have 200 searches, some 1500, some 5000 per month in that google adwords report. After reading those basic tutorials I understand it this way: I put one or two best phrases of those 10 in my <title></p> <p>2) I describe the website as accurately as possible to those keywords in <meta description> i.e. Those specific robots are built from those specific parts - we know all about them.</p> <p>3) I put all of them in <meta keywords> phrase1,2,3, etc.</p> <p>4) I use one or two phrases in <h1> on the page with article</p> <p>5) then I use those phrases in the text that's on that particular page of my website - text is about certain type of robots and parts etc.</p> <p>6) I put photos on that page with alt descriptions that may contain some of these phrases</p> <p>7) to be honest I don't know how am I suppose to build links from that page about certain type of robots to anywhere else on my website - but I undestand that's the part of SEO as well</p> <p>And that's pretty much how I understand the basics of SEO. I read about it and I just don't know if that's what I am suppose to do. Silly me!</p> <p> </p></title>
On-Page Optimization | | lolskizz1