Googlebot encountered extremely large numbers of links on your site??? How Do I resolve this?
-
I am working on a site with over 30 million pages. Every time I get about One Million indexed I get a Message in the Google Webmasters Tools saying "Googlebot encountered extremely large numbers of links on your site"
The indexing then starts dropping like a Rock. I need to get the site indexed. Please Help!
-
Kenneth
I work with extremely large sites quite often. There's no single answer to this because it depends on what's going on as to why the Googlebot is breaking down in its crawl. For example - how many links exist on any single page? Is it 100, 300, 1000 or more? The more links on every page the more likely the bot will choke, though it's a lot better than it used to be.
Does the site validate for markup? Or could there be choke-points due to validation errors?
Is the content organized in an intelligent funnel structure, or is everything one level off the root domain?
Is there only one way for the bot to navigate deep into the site, or are there multiple methods to get down deep?
Is some of the content only linked from within in a way that many of those links are not discovered until the bot has to first go through six or eight other layers, some of which could be timing out just when the bot gets there?
How many quality inbound links point to pages deep within the site?
These are all questions that need to be asked and answered and that's just scratching the surface of the problem potential.
The most important thing is to try and think like the bot - if I go here, will I become overwhelmed? if I go here, will I hit a road-block?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Hacked: Is it Faster and Better to 301 or 404 Irrelevant URLs?
Hey Everyone, So our site was hacked which created a large amount of irrelevant URLs on our domain; resulting in thousands of 404 errors and pages coming up for searches unrelated to our brand. The question is now that the issues have been resolved (and site re-submitted) would it be quicker (and more ideal) to redirect important 404 errors that see traffic, have links…etc. although not relevant or just let everything 404 out? We’re not as concerned with offering a relevant user experience because these are not in our demographic but want to avoid these pages convoluting our analytics as well as issues that might arise from Google thinking these topics do apply. Any help or insight would be very appreciated. Please let us know if you have any questions, concerns or we could provide further details that might help. Looking forward to hearing from all of you! Thanks in advance. Best,
Reporting & Analytics | | Ben-R0 -
How do you handle a "too many inbound links" notice on your crawl diagnostics report?
How do you handle a "too many inbound links" notice on your crawl diagnostics report? The url's identified come from our blog....is it reading all the blog post URL's and giving me a notice? Is this ok to leave all the links? Or would anyone recommend no follow tags on all of our blog posts?
Reporting & Analytics | | cschwartzel0 -
Does anyone know of a way to do a profile level filter to exclude all traffic if it enters the site via certain landing pages?
Does anyone know of a way to do a profile level filter to exclude all traffic if it enters the site via certain landing pages? The problem I have is that we have several pages that are served to visitors of numerous other domains but are also served to visitors of our site. We end up with inflated Google Analytics numbers because people are viewing these pages from our partners' domains but never actually entering our site. I've made an advanced segment that serves the purpose but I'd really like to filter it at the profile level so the numbers across the board are more accurate without having to apply an advanced segment to every report. The advanced segment excludes visits that hit these pages as landing pages but includes visits where people have come from other pages on our domain. I know that you can do profile filters to exclude visits to pages or directories entirely but is there a way to filter them only if they are a landing pages? Any other creative thoughts? Thanks in advance!
Reporting & Analytics | | ATIseo0 -
Difference between site: search and Total Indexed in Google Webmaster Tools.
This morning I did a search on Google for my site using the site: operator. I noticed that the number of results returned was significantly different than the "Total indexed" in Google Webmaster Tools. What is the difference and is it normal to have two very different numbers here?
Reporting & Analytics | | Gordian0 -
When I look at my SEOMOZ campaigns I see there are a lot of warnings in regards to missing Meta Tags Descriptions but they exist on a clien'ts wordpress site
when I look at my SEOMOZ campaigns I see there are a lot of warnings in regards to missing Meta Tags Descriptions but they exist on a clien'ts wordpress site
Reporting & Analytics | | Doug_Hay1 -
Quantifying link building efforts
Does anyone have a way to quantify the value of link building efforts? I am trying to devote resources to improve our link building initiatives but is there any way to correlate it to page rank etc? I know you can't tie it down directly but any rough gauge or studies in the past that people may be aware of? Any thoughts on how you would tackle this question? I need to demonstrate the value it provides in a slightly more scientific manner than saying "it just helps". Thanks,
Reporting & Analytics | | NicB1
Nic0 -
Analytics tagging parameters effect on site SEO
One of the effective tools used in analytics tagging is the use of analytics parameters that starts with '?' or '#'. Example on site tagging: Main link: www.domainname.com./category/sub-category/ www.domainname.com./category/sub-category/?lid=topnav www.domainname.com./category/sub-category/?lid=sidenav All three links link to the same landing page, with an extra parameter. Using email or campaign tagging: www.domainname.com./category/sub-category/ www.domainname.com./category/sub-category/?utm_source=launch&utm_medium=email&utm_term=html&utm_content=getscoop&utm_campaign=hwdyrwm2012 With that we create many tagged links based on the campaign internal strategy. How do these effect indexing, and link juice? How do thy effect SEO in general?
Reporting & Analytics | | RAPPLA0 -
Custom GA tracking and link value
Hi I'm working on a real estate agents web site which has a lots of links coming from paid listings in real estate linstings sites (this in France so I'm not sure examples will mean anything but basically the agents have 900 house listings in each site and each listing has a backlink) In analytics these are classified as referals and that's fine for the moment But because sites provide different types of links, we are considering tagging all paid links with analytics utm codes. Mainly to learn which ads are actually providing qualified traffic (providing contacts). I'm guessing that currently these links, there are thousands, bring seo value to my client's web site and are not considered as paid links. Will adding the analytics codes to these links cause a loss of their value by clearly indicating to Google that we paid for them? What's the current thinking on this? I'm tempted not to be worried about being honest about the origin of these links but I'm worried that there is a real danger of loosing current ranking Any arguments FOR tagging paid links ? Thanks for you help Neil
Reporting & Analytics | | NeilInFrance0