Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Newbie SEO ?: Does my About page URL have to contain the word About?
New to WordPress and SEO. Built and launched my website last week. The URL was originally domain/about. However, I installed Yoast plugin and it told me "about" was a stop word. So, without too much thought (my first problem), I changed the url (before Google crawled me) to clearwingcommunications.com/storytelling. Since then, I've noticed that sites I know are optimized have their URL with the word "about." So, is this considered a bad practice? My site HAS been crawled at this point. If I change it back to About and do a 301 redirect, does that hurt reporting? Thanks for your help! Christy
On-Page Optimization | | christyr0 -
Is tracking code added to the end of a URL considered duplicate content
I have two URLs one with a tracking coded and one without. http://www.towermarketing.net/lets-talk-ux-baby and http://www.towermarketing.net/lets-talk-ux-baby/**#.U6ghgLEz64I ** My question is will this be considered as two separate URLs, will Google consider this as two pages with duplicate content. Any recommendations would be much appreciated.
On-Page Optimization | | TowerMarketing0 -
URL advice
Hi & thanks for looking, I'm not sure if I've adopted the best SEO URL structure for my site, www.vintageheirloom.com For instance, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/ Works great for the top level category 'All bags', as I'm trying to keyword authentic designer vintage bags. However the sub categories for instance 'Clutch bags' appears as, www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/vintage-clutch-bags/. As you can see at the moment this URL contains duplicate terms vintage & bags. I'm guessing that duplicate keywords in a url isn't too smart, but should amend with Option 1, 2, 3 or something completely different? Option 1 - keep the top level category url the same, change the subcategory: www.vintageheirloom.com/product-category/authentic-designer-vintage-bags/clutch/ Option 2 - amend the top level category: www.vintageheirloom.com/product-category/authentic-designer/vintage-clutch-bags/ Option 3 - amend the top level category as this: www.vintageheirloom.com/product-category/bags/authentic-designer-vintage-clutch/ By the way I'm using WordPress with Woocommerce. I've asked but it's not possible with some technical issues to remove the /product-category/ section. But each product is for example just: www.vintageheirloom.com/shop/vintage-coach-yellow-duffel-sac-bag/ .... sweet. Thanks again !!
On-Page Optimization | | well-its-1-louder0 -
Which URL Structure is best for New Website?
Hello All, I have one client and he want to develop his website but the he want the URL structure for his website page like below: http://www.example.com/category/xxproduct.html But I have suggest him below URL Structure http://www.example.com/category/xx-product.html So Can you people please suggest me that from above which URL structure is better from SEO side as well as from Visual side..? Is using - is better to separate the words in URL?
On-Page Optimization | | jemindesai0 -
Moving Top rank Page urls off my Home page and nesting them on one page? Good idea?
I am basically trying to cut down the amount of links on my home page to make it less eye boggling and move stuff around. So i have of my Urls on my home page that lead to pages that rank very well within google. My questions is can i remove those urls to a separate page to group them together and then showcase that one link to that page on my home page. Is that a good idea or i am going to loose my link juice and position in search? The physical urls on those pages wont change at all.
On-Page Optimization | | Dante130 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
How do you see a list of URLs with duplicate page titles?
When looking at the Duplicate Page Title report, the Other URLs column has various numbers that presumably indicate the number of pages that share the same title. When I click on one of these numbers, say a URL that shows 4 in that column, the next page reports "No sample duplicate URLs to report". Why isn't it showing me the other 3 URLs with the same page title?
On-Page Optimization | | jkenyon0 -
Switching URL from keyword heaven to actual brand name?
Our client has a site, we'll say it's delicious-lemonade.com (That's an example.) Their brand name, however, is PowerSky. (That's an example, but the point is that it's the name of a technology, and has NOTHING to do whatsoever with being a drink. Someone would never guess what the product was, by the brand name.) The client has this domain name for a long time. The domain name itself is a top keyword for their product. We are building a new site for the client and have planned to use the brand name (PowerSky) as the primary domain, so PowerSky.com, with the product page being PowerSky.com/Delicious-Lemonade. And we are planning to redirect Delicious-Lemonade.com to PowerSky.com. However, we are concerned about the SEO hit the site is going to take. What recommendations can you make in this situation?
On-Page Optimization | | grayloon0