Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi i have a few pages with duplicate content but we've added canonical urls to them, but i need help understanding what going on
hi google is seeing many of our pages and dupliates but they have canonical url on there https://www.hijabgem.com/index.php/maxi-shirt-dress.html has tags https://www.hijabgem.com/maxi-shirt-dress.html
On-Page Optimization | | hijabgem
has tagshttps://www.hijabgem.com/index.php/quickview/index/view/id/4693
has tags
my question is which page takes authority?and are they setup correct, can you have more than one link rel="canonical" on one page?0 -
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Ok to ignore Overly-Dynamic URL from Moz crawl?
I am developing an ecommerce site, just ran it through the Moz crawl to see what's what and it has come back with a lot of issues. Most of these issues are around duplicate page titles (it is not happy with paginated titles, ie Shoes, Shoes Page 2, Shoes Page 3 etc) and it has also found a lot of Overly-Dynamic URL's. Again, these seem to be from some of the search functions and filters used Accessories&pto_sort=priceAsc&pto_page=6 other than spending a lot of time and effort trying to rewrite these urls there is little I can do about them. Should I just ignore this? I wouldn't imagine it having a massive impact on the rankings of the pages. Thanks, Carl
On-Page Optimization | | GrumpyCarl0 -
Rewrite URL for big website
My website is currently have over 1000 000 links indexed by Google , i'm going to rewrite all of these links, is this crazy ? Can someone give me a checklist to avoid going wrong way ? My website: http://www.webtretho.com/forum/ My current url format: webtretho.com/forum/f[forum_id]/[thread-title-thread_id/ Ex: Old URL: http://www.webtretho.com/forum/f90/chia-se-dau-hieu-mang-thai-1357342-new/ New URL: http://www.webtretho.com/forum/chuan-bi-mang-thai/chia-se-dau-hieu-mang-thai-1357342-new/
On-Page Optimization | | firstjames0 -
Moving Top rank Page urls off my Home page and nesting them on one page? Good idea?
I am basically trying to cut down the amount of links on my home page to make it less eye boggling and move stuff around. So i have of my Urls on my home page that lead to pages that rank very well within google. My questions is can i remove those urls to a separate page to group them together and then showcase that one link to that page on my home page. Is that a good idea or i am going to loose my link juice and position in search? The physical urls on those pages wont change at all.
On-Page Optimization | | Dante130 -
Brand Name URL Redirecting to Actual URL
So we have already built a site under a parent company's URL: parentcompany.com And now we have their branded product lines in directories: parentcompany.com/brand-name1, and parentcompany.com/brand-name2 We also own the actual URL Brand Name 1 (which is also the exact description of the product): brandname1.com We do not yet own the URL for Brand Name 2 (which is also the exact description of the product): brandname2.com. This is because a squatter is sitting on it and is asking $10,000+ for it. What we are trying to determine is how valuable these brand name URLs are since they will be redirecting and not the actual site's primary domain name. Anybody know how much of an effect owning those and redirecting has on ranking for those brand names that are also very descriptive of the products? Would we be smarter to spend $10,000 on adwords or 10,000 on the domain? Thanks!
On-Page Optimization | | grayloon1 -
Canonical URL's - Fixed but still negatively impacted
I recently noticed that our canonical url's were not set up correctly. The incorrect setup predates me but it could have been in place for close to a year, maybe a bit more. Each of the url's had a "sortby" parameter on all of them. I had our platform provider make the fix and now everything is as it should be. I do see issues caused by this in Google Webmaster, for instance in the HTML suggestions it's telling me that pages have duplicate title tags when in fact this is the same page but with a variety of url parameters at the end of the url. To me this just highlights that there is a problem and we are being negatively impacted by the previous implementation. My question is has anyone been in this situation? Is there any way to flush this out or push Google to relook at this? Or is this a sit and be patient situation. I'm also slightly curious if Google will at some point look and see that the canonical urls were changed and then throw up a red flag even though they are finally the way they should be. Any feedback is appreciated. Thanks,
On-Page Optimization | | dgmiles
Dave0 -
URL question
Hi guys, the pro campaign thing you got going is wicked, love it. I'm recieving good results with my keywords and have noticed that categories that go beyond sub/sub/sub don't do to well. So I wanna move those that do one step up which makes it go from: http://spytunes.com/practice-guitar/advanced-routine/scales/aeolian to here http://spytunes.com/practice-guitar/advanced-routine/aeolian The existing menu system that follow all these categories across the site will soon go so it won't be a user friendly problem, I will have other type of menus. But, and here is the question: Would I greatly benefit from taking the non existent menu away and just go for: http://spytunes.com/practice-guitar/aeolian while i'm at it? Or do I stick with my current structure? I guess my real question is; how much is there to flat URLs? Cheers -dan lundholm spytunes.com
On-Page Optimization | | spytunes0