Disallow my store in robots.txt?
-
Should I disallow my store directory in robots.txt?
Here is the URL: https://www.stdtime.com/store/
Here are my reasons for suggesting this:
- SEOMOZ finds crawl "errors" in there that I don't care about
- I don't think I care if the search engines index those pages
- I only have one product, and it is not an impulse buy
- My product has a 60 day sales cycle, so price is less important than features
-
You seem to have evaluated things well, i will give you one reason why yopu should not, links pointing to the non indexed pages will be pouring their link juice away. , you are better off using a no-index,follow meta tag, at least then the link juice will flow back out of the links when followed.
Robots text is a nasty tool to use, you need a more surgical approch
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots blocked by pages webmasters tools
a mistake made in software. How can I solve the problem quickly? help me. XTRjH
Intermediate & Advanced SEO | | mihoreis0 -
How to target slight variations in ecommerce store
Hi friends, I'm trying to target a variety of keywords for an e-commerce/woocommerce store. These keywords have slight differences, and I have a writer to come up with a paragraph or two of unique content for each SEO landing page. I plan on making them with WordPress pages, and linking them with in the content to other related blog posts and product categories. Would it be best to make individual landing pages for terms that vary like: mens wallets under $100 mens wallets under $50 mens wallets under $25 mens leather wallets qaulity mens wallets affordable mens wallets buy mens wallets online shop mens wallets online Should these each be individual pages? How should I link these to the main navigation? or Should I interlink them elsewhere?
Intermediate & Advanced SEO | | JustinMurray0 -
Help with Robots.txt On a Shared Root
Hi, I posted a similar question last week asking about subdomains but a couple of complications have arisen. Two different websites I am looking after share the same root domain which means that they will have to share the same robots.txt. Does anybody have suggestions to separate the two on the same file without complications? It's a tricky one. Thank you in advance.
Intermediate & Advanced SEO | | Whittie0 -
Pages getting into Google Index, blocked by Robots.txt??
Hi all, So yesterday we set up to Remove URL's that got into the Google index that were not supposed to be there, due to faceted navigation... We searched for the URL's by using this in Google Search.
Intermediate & Advanced SEO | | bjs2010
site:www.sekretza.com inurl:price=
site:www.sekretza.com inurl:artists= So it brings up a list of "duplicate" pages, and they have the usual: "A description for this result is not available because of this site's robots.txt – learn more." So we removed them all, and google removed them all, every single one. This morning I do a check, and I find that more are creeping in - If i take one of the suspecting dupes to the Robots.txt tester, Google tells me it's Blocked. - and yet it's appearing in their index?? I'm confused as to why a path that is blocked is able to get into the index?? I'm thinking of lifting the Robots block so that Google can see that these pages also have a Meta NOINDEX,FOLLOW tag on - but surely that will waste my crawl budget on unnecessary pages? Any ideas? thanks.0 -
Can't find X-Robots tag!
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following: X-Robots-Tag:
Intermediate & Advanced SEO | | Blink-SEO
noindex, nofollow It's not in the page HTML though. Could it be being picked up from somewhere else?0 -
Retail Store Detail Page and Local SEO Best Practices
We are working with a large retailer that has specific pages for each store they run. We are interested in leveraging the best practices that are out their specifically for local search. Our current issue is around URL design for the stores pages themselves. Currently, we have store URL's such as: /store/12584 The number is a GUID like character that means nothing to search engines or, frankly, humans. Is there a better way we could model this URL for increased relevancy for local retail search? For example: adding store name:
Intermediate & Advanced SEO | | mongillo
www.domain.com/store/1st-and-denny-new-york-city/23421
(example http://www.apple.com/retail/universityvillage/) fully explicit URI www.domain.com/store/us/new-york/new-york-city/10027/bronx/23421
(example http://www.patagonia.com/us/patagonia-san-diego-2185-san-elijo-avenue-cardiff-by-the-sea-california-92007?assetid=5172) the idea with this second version is that we'd make the URL structure more rich and detailed which might help for local search. Would there be a best practice or recommendation as to how we should model this URL? We are also working to create an on-page optimization but we're specifically interested in local seo strategy and URL design.0 -
Could you use a robots.txt file to disalow a duplicate content page from being crawled?
A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO. I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?
Intermediate & Advanced SEO | | gregelwell0 -
SEO Penalties for Splitting Page for Two Store Locations
Hello fellow SEO'ers! I have a question regarding the overall SEO implications of using a single page to describe the services/products offered at two different locations. The locations are in two different states/cities. I have tried to explain to the client that I working with that this is essentially splitting the page in two from a search ranking perspective. I have a feeling that Google sees this page as partially dedicated to one city, and partly to another... meaning that it won't rank as well as it could for either city. Is my thinking correct? Seems logical. The client has done this site-wide for every service/product that they offer in their facilities. I'm offering some title/description recommendations for the entire site right now, and I'm going back and forth with myself whether to include the city names in the titles and descriptions at all. Let me know what you smart folks think. I appreciate it. Sam
Intermediate & Advanced SEO | | theBREWROOM0