Howcome Google is indexing one day 2500 pages and the other day only 150 then 2000 again ect?
-
This is about an big affiliate website of an customer of us, running with datafeeds...
Bad things about datafeeds:
- Duplicate Content (product descriptions)
- Verrryyyy Much (thin) product pages (sometimes better to noindex, i know, but this customer doesn't want to do that)
-
Hi Dana,
Thanks for your detailed explanation. Appreciate it Off course I understand that site speed is a factor for crawling (+ ranking) and that the Google bots only want to spend a certain period of time on a website. It's more like, when servers are performing almost equal every day so page loads are igual to, what could it be?
I agree with your two points of considering, but I'm the type of guy that always wants to know why something is happening
@Nakul: Thanks for your responds!
The pages that are in and out of the index are mostly product pages. So the thing about "frequently updates" can be something. The website is pretty young so authority is not yet build as it should be for a big site. This can also be a factor cause the more authority the more time Google will spend indexing a website rightAnyway, great thanks for both of your answers!
Gr. Wesley
-
I agree with everything Nakul has said. Just to piggyback on that with additional information, try to think about it this way. Remember when someone gave you $1.00 when you were little and said "Don't spend it all in one place?" Well, someone at Google must have grown up with the same grandparents I did.
Okay, now, the analogy-free explanation
Google has a "crawl budget" every day. Every day that budget is allocated to millions of different sites. Now, by "sites" I mean "pages." Some pages change really frequently (i.e. the Yahoo New homepage). Some pages change hardly ever (i.e. an archived blog post). Also, some pages have very high PR and others, not so much. Also, some pages load extremely fast (consuming less of Google's bandwidth when the page is crawled) which leaves more Google resources available to Google to crawl more pages. Google likes it, and so should we all because people with fast sites are making it possible for everyone to get crawled more often (in essence, making them very considerate, well-behaved members of the Internet community).
So, based on all these, Google is going to apportion a part of its crawl budget to your site on any given day. Some days, it may have more room in its budget for you than others. Part of this might be effected by how fast pages, on any given day, load from your site. A ton of parameters can come into play here, including whether or not the pages on that day are heavier, or whether or not your servers are performing really fast on one day versus another.
I'd say the two things to be really concerned with after considering all of these things are:
- Is Google indexing all of the pages you want indexed?
- Is Google's cache date of your important pages recent enough? (i.e. 3 weeks or less)
If the answer is "no" to either one of those, then it's time to do some investigation to find out if there are technical issues or penalties that have been put in place that are hurting Google's ability or desire (not the right word to use about a bot, but I'm using it anyway) to crawl your pages.
Does that help?
-
Domain Authority / Pagerank is what Google looks to see how deep and how frequently Google will crawl a particular website. They also typically look into how frequently the content is being updated.
Think about it from Google's perspective. Why should they index that website, 2500 pages every day. What's changing ? Does the site have enough domain authority to warrant that kind of indexing ?
In my opinion, this is not a concern. Just submit XML Sitemaps and see what percentage of your submitted pages are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Difficulty with Indexing Pages - Desperate for Help!
I have a website with product pages that use the same URL, but load different data based on what's passed to them with GET. I am using a Wordpress website, but all of the page information is retrieved from a database using PHP and displayed with PHP. Somehow these pages are not being indexed by Google. I have done the following: 1. Created a site map pointing to each page. 2. Defined URL parameters in Search Console for these type of pages. 3. Created a product schema using schema.org, and tested it without errors. I have requested re-indexing repeatedly and these pages and images on the pages are still not being indexed! Does anybody have any suggestions?
Intermediate & Advanced SEO | | jacleaves0 -
No Index thousands of thin content pages?
Hello all! I'm working on a site that features a service marketed to community leaders that allows the citizens of that community log 311 type issues such as potholes, broken streetlights, etc. The "marketing" front of the site is 10-12 pages of content to be optimized for the community leader searchers however, as you can imagine there are thousands and thousands of pages of one or two line complaints such as, "There is a pothole on Main St. and 3rd." These complaint pages are not about the service, and I'm thinking not helpful to my end goal of gaining awareness of the service through search for the community leaders. Community leaders are searching for "311 request service", not "potholes on main street". Should all of these "complaint" pages be NOINDEX'd? What if there are a number of quality links pointing to the complaint pages? Do I have to worry about losing Domain Authority if I do NOINDEX them? Thanks for any input. Ken
Intermediate & Advanced SEO | | KenSchaefer0 -
Why one of my top pages dropped?
Hello here. Our website, virtualsheetmusic.com, is pretty popular in the sheet music realm, and we used to rank on the first page for the keyword "violin sheet music" until a few weeks ago with our violin dedicated page: http://www.virtualsheetmusic.com/downloads/Indici/Violin.html But a couple of weeks ago we dropped to over the 5th page on Google (I can't even find us!) and I have no idea why. Most of our top ranking pages are still there though. This never happened before, after 17 years on the web. Do you have any idea why that could have happened?
Intermediate & Advanced SEO | | fablau0 -
Wordpress blog in a subdirectory not being indexed by Google
HI MozzersIn my websites sitemap.xml, pages are listed, such as /blog/ and /blog/textile-fact-or-fiction-egyptian-cotton-explained/These pages are visible when you visit them in a browser and when you use the Google Webmaster tool - Fetch as Google to view them (see attachment), however they aren't being indexed in Google, not even the root directory for the blog (/blog/) is being indexed, and when we query:site: www.hilden.co.uk/blog/ It returns 0 results in Google.Also note that:The Wordpress installation is located at /blog/ which is a subdirectory of the main root directory which is managed by Magento. I'm wondering if this causing the problem.Any help on this would be greatly appreciated!AnthonyToTOHuj.png?1
Intermediate & Advanced SEO | | Tone_Agency0 -
Getting 260,000 pages re-indexed?
Hey there guys, I was recently hired to do SEO for a big forum to move the site to a new domain and to get them back up to their ranks after this move. This all went quite well, except for the fact that we lost about 1/3rd of our traffic. Although I expected some traffic to drop, this is quite a lot and I'm wondering what it is. The big keywords are still pulling the same traffic but I feel that a lot of the small threads on the forums have been de-indexed. Now, with a site with 260,000 threads, do I just take my loss and focus on new keywords? Or is there something I can do to get all these threads re-indexed? Thanks!
Intermediate & Advanced SEO | | StefanJDorresteijn0 -
301 Redirect for 2500 pages
Hi, We have an existing site done in DNN and we recreated it on a new platform (EPiServer) and now we're going live. However, there are 2500+ page URLs from the old site which is not exisitng on the new site. What do you reckon is the best way we can address this? Do we create a 301 redirect individually for each of these pages? These 2500+ pages have a domain authority 34-35 and I think it's best that we retain those. We'll be using the same domain name. Suggestions for ways to approach this issue would be greatly appreciated. I have access to the server and IIS. *Also, how do I create a virtual page in IIS? and redirect it to another URL within the site? Thanks.
Intermediate & Advanced SEO | | Peter.Huxley590 -
Why does this page not show in google at all?
www.lavenderblue-flowers.co.uk Sorry for formatting, below is the source. There are alot of blocks from robots.txt but is there anything easily rectified to get this site SOME visibility? Duplicate content maybe PANDA had it? No backlink profile too which isnt helping but even still, surprising to see a domain auth of 1. Thanks in advance for any responses. DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta http-equiv="expires" content="Fri, 17 Jun 2011 12:06:27 GMT"><title>Bridport Interflora Florist, Lavender Blue, Dorset, DT16 3XDtitle><meta name="description" content="Lavender Blue in Bridport, Dorset, DT16 3XD delivers to Interflora florist based in Bridport is a well established family run business with a dedicated team of florists. We specialise in beautiful wedding flowers and take great pride in our funeral tributes, floral arrangements designed for any occasion for local, national and worldwide delivery."><meta name="keywords" content="Bridport,Interflora Florist,Lavender Blue,Dorset,DT16 3XD"><meta name="abstract" content="Interflora florist based in Bridport is a well established family run business with a dedicated team of florists. We specialise in beautiful wedding flowers and take great pride in our funeral tributes, floral arrangements designed for any occasion for local, national and worldwide delivery."><meta name="robots" content="index,nofollow"><link rel="stylesheet" type="text/css" href="/kernel/styles/print.css?new=new" media="print"><link rel="stylesheet" href="/kernel/styles/d4.css?designtype=d4;theme=blue;" type="text/css"><style type="text/css">style><script language="JavaScript1.2" src="/kernel/utils.js?new" type="text/javascript">script><script language="JavaScript1.2" type="text/javascript" src="/kernel/interflora.js?head=1;si=1000343;">script><script language="JavaScript1.2" type="text/javascript">script><script language="javascript"> var b_site_url = getcookie('b_site_url');if (b_site_url != "" && !getcookie('referral_id') && location.protocol == 'http:' && b_site_url != location.host && location.pathname.indexOf('catalog2') == -1) location.href = location.protocol + "//" + b_site_url + location.pathname + location.search;script>head><body><img border="0" src="/kernel/images/speck.gif" width="1" height="1" alt class="nospace"><div id="page-body"><table class="page-topbanner" border="0" cellpadding="0" cellspacing="0"><tr><td background="/kernel/images/d4/border-blue_03.gif" align="left" valign="top"><img src="/kernel/images/d4/border-blue_01.gif" alt>td><td colspan="2" style="background-image: url(/kernel/images/d4/border-blue_03.gif); background-position: top; background-repeat: repeat-x;"><img src="/kernel/images/speck.gif" width="300" height="50" alt>td><td align="right" valign="top" background="/kernel/images/d4/border-blue_03.gif"><img src="/kernel/images/d4/border-blue_04.gif" alt>td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-repeat: repeat-y;" align="left" valign="top"><img src="/kernel/images/d4/border-blue_01b.gif" alt>td><td valign="top" class="sd-image_only" id="sd-logo_store" colspan="1" rowspan="1"><img src="/kernel/imageload?ttl2=15;table=content_images;key1=fd_img_2606422_1" alt="" title="">td><td class="logo-if" align="right"><img src="/kernel/images/logo-if.png" alt="interflora.co.uk the flower experts™">td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;" colspan="3" align="center"><table id="website" cellspacing="0" border="0" align="center"><tr><td colspan="3" id="fol_address">1 Lilliput Lane, Bridport, Dorset, DT16 3XDtd>tr><tr><td id="email" colspan="3"><b>Email:b> lavenderblueflowers@hotmail.co.uktd>tr><tr><td style="padding-right:10px;"><b>Phone:b> 01308 459145td><td style="padding-right:10px;"><b>Fax:b> 01308 458417td>tr>table>td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;" colspan="3" align="center"><div class="page-topmenu"><table class="page-topmenu" cellspacing="0"><tr><td id="account"><a href="/myaccount/"><img src="/kernel/images/d4/icon-account.gif" style="margin: 3px 3px 4px 3px; vertical-align: middle;" width="15" height="13" alt="My Account">My Accounta>td><td id="menu"><a href="/">Homea><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=about">About Usa><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=delivery">Delivery Infoa><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=contactus">Contact Usa>td><td id="cart"><a href="/shopcart/"><img src="/kernel/images/d4/icon-shopcart.gif" style="margin: 3px; vertical-align: middle;" width="14" hieght="14" alt="Shopping Basket">Shopping Basketa>td>tr>table>div>td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr>table><p id="browser-warning" style="display: block; padding: 2px; border: 2px solid #FC9F85; margin: 0px; background-color: #FDFFC4;"><b>For your information:b> This message has appeared because we've noticed your browser doesn't fully support all functions of this site. For further information please <a href="/page.xml?page_name=faq">click herea>.p><script language="JavaScript1.2" type="text/javascript">var theBrowser = navigator.userAgent.toLowerCase();if(is_nav7up || (parseInt(is_moz_ver) >= 1) || is_ie5_5up || theBrowser.indexOf("safari") != -1) {hideElement('browser-warning',0);}script><table class="body" border="0" cellspacing="0" cellpadding="0"><tr><td align="left" valign="bottom" style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;"><img src="/kernel/images/d4/border-blue_05.gif" alt>td><td class="menu" valign="top"><img src="/kernel/images/speck.gif" width="150" height="1" border="0" alt><br><form method="get" action="/search/index.xml" id="leftnav_search"><table border="0" cellspacing="0" class="global-search"><tr><th colspan="2">SEARCHth>tr><tr><td width="50%"><input class="text" type="text" name="keywords1" id="search" value maxlength="50" size="15">td><td align="left"><input type="submit" class="button" name="search" id="search" value="GO">td>tr><tr><td colspan="2" align="left"><a href="/search/advanced_search.xml">Advanced Searcha>td>tr>table>form><div class="menusection"><a class="menuParent_off" id="parentcat_2003443" href="/catalog/category.xml?category_id=2003443"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Anniversaryspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003443">div><a class="menuParent_off" id="parentcat_2003453" href="/catalog/category.xml?category_id=2003453"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Congratulationsspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003453">div><a class="menuParent_off" id="parentcat_4" href="/category/flower-arrangements/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">All Flower Bouquetsspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_4">div><a class="menuParent_off" id="parentcat_2003493" href="/catalog/category.xml?category_id=2003493"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Sympathy & Funeralspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003493">div><a class="menuParent_off" id="parentcat_2003463" href="/catalog/category.xml?category_id=2003463"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Thank Youspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003463">div><a class="menuParent_off" id="parentcat_2001478" href="/category/same-day-flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Same Day Flower Deliveryspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2001478">div><a class="menuParent_off" id="parentcat_2124203" href="/category/summer_flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Summer Flowersspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2124203">div><a class="menuParent_off" id="parentcat_2003403" href="/category/luxury-flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Luxury Flowersspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003403">div><a class="menuParent_off" id="parentcat_1000343" href="/catalo
Intermediate & Advanced SEO | | ewanstevenson0 -
Static index page or not?
Are there any advantages of dis-advantages to running a static homepage as opposed to a blog style homepage. I have be running a static page on my site with the latest posts displayed as links after the homepage content. I would like to remove the static page and move to a more visually appealing homepage that includes graphics for each post and the posts droppping down the page like normal blogs do. How will this effect my site if I move from a static page to a more dynamic blog style page layout? Could I still hold the spot I currently rank for with the optimized index content if I turn to a more traditional blog format? cheers,
Intermediate & Advanced SEO | | NoCoGuru0