How to resolve - Googlebot found an extremely high number of URLs
-
Hi,
We got this message from Google Webmaster “Googlebot found an extremely high number of URLs on your site”. The sample URLs provided by Google are all either noindex or have a canonical.
- http://www.myntra.com/nike-stylish-show-caps-sweaters
- http://www.myntra.com/backpacks/f-gear/f-gear-unisex-black-&-purple-calvin-backpack/162453/buy?src=tn&nav_id=541
- http://www.myntra.com/kurtas/alma/alma-women-blue-floral-printed-kurta/85178/buy?nav_id=625
Also we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters.
Your comments on how to resolve this issue will be appreciated.
Thank You
Kaushal Thakkar
-
Hi Kaushal,
Thanks for the question.
There are a few ways to deal with this problem which are recommended by Google here. In summary, you can:
- Use parameter handling as you have done
- Add the nofollow attribute to problematic URLs
- Block problematic URLs in robots.txt
There is also a thread in the Google webmaster forums which may be useful to you:
Overall, it comes down to having a good site architecture and cutting down / removing / blocking URLs that you don't care about from a search perspective.
I hope that helps a bit!
Paddy
-
Thank you David, Its been more than 10 months since these parameters have been specified in webmaster. This and other activities like noindex and canonicals helped us to reduce the indexed URL count from 32 million to 1.2 million. As the url index reduced this warning from google stopped for 4 months. However we started receiving this message again from february 2014.
Thanks
Kaushal
-
"we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters."
How long ago was this done? Since there are so many URL's, it may take a while for them to recrawl and index the representative URL's per your request.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to improve PA of Shortened URLs
Why some of shortened urls like bitly/owly/googl has PA>40? I tried everything to improve PA of my shortened urls like facebook shares, retweets and backlinks to them but still i have PA-1. Checkout this URL: https://moz.com/blog/state-of-links in MOZ OSE and you will many 301 links from shortners
White Hat / Black Hat SEO | | igains
I asked many seo experts about this but no one answered this question so today subscribed MOZ pro for the solution. Please give me the answer.0 -
Duplicate content warning: Same page but different urls???
Hi guys i have a friend of mine who has a site i noticed once tested with moz that there are 80 duplicate content warnings, for instance Page 1 is http://yourdigitalfile.com/signing-documents.html the warning page is http://www.yourdigitalfile.com/signing-documents.html another example Page 1 http://www.yourdigitalfile.com/ same second page http://yourdigitalfile.com i noticed that the whole website is like the nealry every page has another version in a different url?, any ideas why they dev would do this, also the pages that have received the warnings are not redirected to the newer pages you can go to either one??? thanks very much
White Hat / Black Hat SEO | | ydf0 -
Do you get penalized for Keyword Stuffing in different page URLs?
If i have a website that provides law services in varying towns and we have pages for each town with unique content on each page, can the page URLS look like the following: mysite.com/miami-family-law-attorney mysite.com/tampa-family-law-attorney mysite.com/orlando-family-law-attorney Does this get penalized when being indexed?
White Hat / Black Hat SEO | | Armen-SEO0 -
Unique page URLs and SEO titles
www.heartwavemedia.com / Wordpress / All in One SEO pack I understand Google values unique titles and content but I'm unclear as to the difference between changing the page url slug and the seo title. For example: I have an about page with the url "www.heartwavemedia.com/about" and the SEO title San Francisco Video Production | Heartwave Media | About I've noticed some of my competitors using url structures more like "www.competitor.com/san-francisco-video-production-about" Would it be wise to follow their lead? Will my landing page rank higher if each subsequent page uses similar keyword packed, long tail url? Or is that considered black hat? If advisable, would a url structure that includes "san-francisco-video-production-_____" be seen as being to similar even if it varies by one word at the end? Furthermore, will I be penalized for using similar SEO descriptions ie. "San Francisco Video Production | Heartwave Media | Portfolio" and San Francisco Video Production | Heartwave Media | Contact" or is the difference of one word "portfolio" and "contact" sufficient to read as unique? Finally...am I making any sense? Any and all thoughts appreciated...
White Hat / Black Hat SEO | | keeot0 -
Where is the rule here that force "www." in URLs?
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)/(1)?$ /search/$1/sort-$2 [R=301]
White Hat / Black Hat SEO | | esiow2013
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$5&sort=$2 [L]
RewriteRule ^search/([a-zA-Z0-9,-]+)/(1)?$ /search/$1 [R=301]
RewriteRule ^search/([a-zA-Z0-9,-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$4 [L]
RewriteRule ^([r|R])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=1 [L]
RewriteRule ^([s|S])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=2 [L]
RewriteRule ^reset-password$ index.php?cmd=retrievePassword [L]
RewriteRule ^reset-password/([0-9]+)/([a-zA-Z0-9]+)$ index.php?cmd=retrievePassword&act=retrievePasswordConfirm&code=$1&key=$2 [L]
RewriteRule ^map-page.php$ index.php?cmd=mapPage [L]
RewriteRule ^advertise.php$ index.php?cmd=advertiseWithUs [L]
RewriteRule ^my-account.php$ index.php?cmd=protectedMyAccount [L]
RewriteRule ^bookmarked-properties(/((R|S)-([0-9]+)))?$ index.php?cmd=protectedBookmarkedProperties&id=$2 [L]
RewriteRule ^bookmarked-properties/([0-9]+)$ index.php?cmd=protectedBookmarkedProperties&page=$1 [L]
RewriteRule ^bookmarked-properties/delete/([0-9,]+)$ index.php?cmd=protectedBookmarkedProperties&act=protectedBookmarkedProperties_delete&id=$1 [L]
RewriteRule ^register.php$ index.php?cmd=register [L]
RewriteRule ^about-us.php$ index.php?cmd=aboutUs [L]
RewriteRule ^news(/([0-9]+)(.*))?$ index.php?cmd=news&id=$2&title=$3 [L]
RewriteRule ^registration/confirm/key=([a-zA-Z0-9]{150})/code=([a-zA-Z0-9]{5})$ index.php?cmd=register&act=registerConfirm&key=$1&code=$2 [L]
RewriteRule ^terms-and-conditions.php$ index.php?cmd=termsAndConditions [L]
RewriteRule ^privacy-policy.php$ index.php?cmd=privacyPolicy [L]
RewriteRule ^contact-us.php$ index.php?cmd=contactUs [L]
RewriteRule ^sitemap.php$ index.php?cmd=siteMap [L]
RewriteRule ^members/login.php$ login.php [L]
RewriteRule ^rss$ index.php?cmd=rss [L]
RewriteRule ^resources.php$ index.php?cmd=links [L]
RewriteRule ^links.php$ /resources.php [R=301]
RewriteRule ^resources2.php$ index.php?cmd=links2 [L]
RewriteRule ^results.php$ index.php?cmd=results [L]
RewriteRule ^residential-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&page=$3 [L]
RewriteRule ^commercial-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=2&page=$3 [L]
RewriteRule ^short-term-rentals((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&str=1&page=$3 [L]
RewriteRule ^buy((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=2&page=$3 [L]
RewriteRule ^rent((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=1&page=$3 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&sort=$2&page=$5 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&sort=$4&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&sort=$5&page=$8 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&page=$6 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=2&page=$4 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=2&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&page=$5 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&page=$5 [L]
RewriteRule ^rent/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=2&page=$4 [L]0 -
Moving content to a clean URL
Greetings My site was seriously punished in the recent penguin update. I foolishly got some bad out sourced spammy links built and I am now paying for it 😞 I am now thinking it best to start fresh on a new url, but I am wondering if I can use the content from the flagged site on the new url. Would this be flagged as duplicate content, even if i took the old site down? your help is greatly appreciated Silas
White Hat / Black Hat SEO | | Silasrose0 -
Will two numbers on a local listing affect me?
The reason I'm asking this question is. I was on the phone with a Google Rep yesterday for one of my google places. It was in reference to map maker and the fact that I only wanted one number on the listing. About a month ago I had it, so I deleted the listing's local number and than had an 877 number. The problem is when I checked on Friday the Local number was then added back by a Google moderator in Map Maker. So, now there's two numbers on the listing. He told me that would not affect my NAP info, which I don't believe.
White Hat / Black Hat SEO | | PeterRota
He also went on to say that Google goes through listings and if they have an 800 number they may delete it and replace place it with a local number. Has anyone delt with this and can Verify what he says to be true? Additionally, will my NAP be affected if this is the case? Thanks.0 -
Why are these sites so high with poor relevant links...
Hello, Keyword: TV Stands. I have been researching competitors for a client and we seem to be unable to understand why certains pages are ranking on page 1 of Google UK for keyword TV Stands. eg: http://www.furnitureinfashion.net/plasma-TV-stand.html (Google UK 8 - TV Stands) http://direct.tesco.com/q/N.1999542/Nr.99.aspx (Google UK 9 - TV Stands) The furniture in fashion has links from sites like: http://www.ummah.com/forum/ and http://www.muslimco.com/ which is totaly irrelevant to the site. Any ideas on other things as the tesco.com site does not have direct links to it. Cheers
White Hat / Black Hat SEO | | JohnW-UK0