Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing/shortening a url hurt SEO?
Hi all, I am in the process of making small optimization changes to my site. I noticed Moz identified quite a few URLs that could be shortened. I intend to shorten these URLs and create 301 redirects to ensure website users land on the right page. My question is, will this change in URL damage rankings and engagement(assuming the URL remains content relevant)? I have read in some places that when creating URL redirects for a change in domain, people saw a dip in rankings and engagement. I, however, am not intending to change the main domain of the site, but rather the URL slug. Any thoughts?
On-Page Optimization | | annegretwidmer0 -
Change of URL + SEO Impact
We recently changed the URLs from our site http://www.website.com (old)
On-Page Optimization | | ederdesign
https://website.com/us/en/ (new) Our ranking is plummeting every since and I wonder if the new URL had something to do with it. Do you know if that change, could have impacted the ranking?0 -
Will changing a URL negatively affect ranking?
Hello Mozzers, We would like to change the URL for a page on our website which ranks well for some our keyphrases/words. We are hoping the change of URL, through the addition of an additional keyword would help boost the rank of that URL further. At the moment out page gets 2 x A and 2 x B 1xF on the MOZ page rank tool using 5 keyphrase/word variations . One phrase ranks 4, one ranks 3 and the other 3 are 'not in the top 50' Our plan was to change the URL, using SHF404, and use 'Fetch' in the Google search console to re-submit the page to Google. Appreciate you can't give any guarantees how Google will behave, just wondered what your thoughts were on the wisdom of changing the URL in the first place? Thanks Ian
On-Page Optimization | | Substance-create0 -
What are the benefits of the URL meta tag?
We have too many meta tags and want to get rid of all the outdated ones. However, we don't want to eliminate valuable meta tags by mistake. So, before we say goodbye to the URL meta tag, we want to make sure we understand the pros and cons, if any. By the way, we are not referring to canonical URL tags, just URL as in:
On-Page Optimization | | GRIP-SEO0 -
Custom Landing Page URLs
I will begin creating custom landing pages optimized for long-tail keywords. Placing the keywords in the URL is obviously important -- Question: would it be detrimental to rankings to have extra characters extending past the keyword? I'm not able to use tracking code, but need to put an identifier in the URL (clp = custom landing page). For example, is "www.domain.com/silver-fish.html" going to perform meaningfully better than "www.domain.com/silver-fish-clp.html" for the kw phrase "silver fish"? There will obviously be a lot of on-page optimization in addition to just structuring the URLs. Thank you. SIMbiz
On-Page Optimization | | SIMbiz0 -
0 urls indexed in GWT, many found with site: command
Hi, This is happening with a brand new site. We have created sitemaps and submitted them to Google Webmaster Tools. GWT says sitemaps are ok, "x" number of urls submitted, but no urls indexed. When I check in Google with site:domain.com I see that many of the urls are already indexed. Why this discrepancy between GWT and reality? Thanks for your time!
On-Page Optimization | | gerardoH0 -
How to Define Best URL Structure for Product Pages?
I am working on my website to edit structure with help of Google's search engine optimization starter guide. There is really good instruction to define URL structure which help us to perform well over Google's organic search. I have resolved issues regarding category pages but, I have confusion to define best URL structure for product pages. My website's product page URL structure is as follow. http://www.vistastores.com/marketumbrellas-californiaumbrella-slpt758-f13-red.html http://www.vistastores.com/homefurniture-winsomewood-93630.html URL structure is constructed with following terms. 1. Root Category Name (Market Umbrellas or Home Furniture or ....) 2. Brand Name 3. Manufacturer Part Number I am not happy with this structure and also not performing well over Google's organic search. I am thinking to include product name or title tag in URL after root domain. But, it may create very long URL and create issues in organic search display. Does it really matter to perform well over Google's organic search? How can I define best URL structure for product pages?
On-Page Optimization | | CommercePundit0 -
Duplicate content issue with dynamically generated url
Hi, For those who have followed my previous question, I have a similar one regarding dynamically generated urls. From this page http://www.selectcaribbean.com/listing.html the user can make a selection according to various criteria. 6 results are presented and then the user can go to the next page. I know I should probably rewrite url's such as these: http://www.selectcaribbean.com/listing.html?pageNo=1&selType=&selCity=&selPrice=&selBeds=&selTrad=&selMod=&selOcean= but since all the results presented are basically generated on the fly for the convenience of the user, I am afraid google my consider this as an attempt to generate more pages as there are pages for each individual listing. What is my solution for this? Nofollow these pages? Block them thru robots txt?
On-Page Optimization | | multilang0