Submitting sitemaps every 7 days
-
Question, if you had a site with more than 10 million pages (that you wanted indexed) and you considered each page to be equal in value how would you submit sitemaps to Google?
Would you submit them all at once: 200 sitemaps 50K each in a sitemap index?
Or
Would you submit them slowly? For example, would it be a good idea to submit 300,000 at a time (in 6 sitemaps 50k each). Leave those those 6 sitemaps available for Google to crawl for 7 days then delete them and add 6 more with 300,000 new links? Then repeat this process until Google has crawled all the links? If you implemented this process you would never at one time have more than 300,000 links available for Google to crawl in sitemaps.
I read somewhere that eBay does something like this, it could be bogus info though.
Thanks
David
-
Thanks Maurizio.
What I am really most concerned about is submitting hundreds of sitemaps to Google and giving them concern that we might be spamming them.
This is why I am considering the second approach where we would submit 6 sitemaps at a time which would total no more than 300,000 links rather than giving them 200 plus sitemaps with 10 million links.
I should have been clearer in my reason for this question. The main goal here is to not have Google freakout because we just gave them 10,000,000 links at one time.
-
hI
it's better divide the sitemap in many files, max 50k and create
how you can read in this page
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35738
"To fix this issue, break your Sitemap into several smaller Sitemaps, and list these in a Sitemap index file. (More information about Sitemap index files.) Upload your Sitemaps and Sitemap index files to your site, then submit these files individually."
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can anyone help me diagnose an indexing/sitemap issue on a large e-commerce site?
Hey guys. Wondering if someone can help diagnose a problem for me. Here's our site: https://www.flagandbanner.com/ We have a fairly large e-commerce site--roughly 23,000 urls according to crawls using both Moz and Screaming Frog. I have created an XML sitemap (using SF) and uploading to Webmaster Tools. WMT is only showing about 2,500 urls indexed. Further, WMT is showing that Google is indexing only about 1/2 (approx. 11,000) of the urls. Finally (to add even more confusion), when doing a site search on Google (site:) it's only showing about 5,400 urls found. The numbers are all over the place! Here's the robots.txt file: User-agent: *
Intermediate & Advanced SEO | | webrocket
Allow: /
Disallow: /aspnet_client/
Disallow: /httperrors/
Disallow: /HTTPErrors/
Disallow: /temp/
Disallow: /test/ Disallow: /i_i_email_friend_request
Disallow: /i_i_narrow_your_search
Disallow: /shopping_cart
Disallow: /add_product_to_favorites
Disallow: /email_friend_request
Disallow: /searchformaction
Disallow: /search_keyword
Disallow: /page=
Disallow: /hid=
Disallow: /fab/* Sitemap: https://www.flagandbanner.com/images/sitemap.xml Anyone have any thoughts as to what our problems are?? Mike0 -
Sitemap indexing
Hi everyone, Here's a duplicate content challenge I'm facing: Let's assume that we sell brown, blue, white and black 'Nike Shoes model 2017'. Because of technical reasons, we really need four urls to properly show these variations on our website. We find substantial search volume on 'Nike Shoes model 2017', but none on any of the color variants. Would it be theoretically possible to show page A, B, C and D on the website and: Give each page a canonical to page X, which is the 'default' page that we want to rank in Google (a product page that has a color selector) but is not directly linked from the site Mention page X in the sitemap.xml. (And not A, B, C or D). So the 'clean' urls get indexed and the color variations do not? In other words: Is it possible to rank a page that is only discovered via sitemap and canonicals?
Intermediate & Advanced SEO | | Adriaan.Multiply1 -
Sitemaps during a migration - which is the best way of dealing with them?
Many SEOs I know simply upload the new sitemap once the new site is launched - some keep the old site's URLs on the new sitemap (for a while) to facilitate the migration - others upload both the old and the new website together, to support the migration. Which is the best way to proceed? Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Best Way to Incorporate FAQs into Every Page - Duplicate Content?
Hi Mozzers, We want to incorporate a 'Dictionary' of terms onto quite a few pages on our site, similar to an FAQ system. The 'Dictionary' has 285 terms in it, with about 1 sentence of content for each one (approximately 5,000 words total). The content is unique to our site and not keyword stuffed, but I am unsure what Google will think about us having all this shared content on these pages. I have a few ideas about how we can build this, but my higher-ups really want the entire dictionary on every page. Thoughts? Image of what we're thinking here - http://screencast.com/t/GkhOktwC4I Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
Changing the XML Sitemap address
For technical reason we are having to change our XML sitemap URL's from domain.com/sitemap.xml to domain.com/sitemaps/sitemap.xml - What checklist do I need to do to make sure this transition goes smoothly and is there any problems that I might come across?
Intermediate & Advanced SEO | | JohnW-UK0 -
Do you have to wait after disavowing before submitting a reconsideration request
Hi all We have a link penalty at the moment it seems. I went through 40k links in various phases and have disavowed over a thousand domains that date back to old SEO work. I was barely able to have any links removed as the majority are on directories etc that no one looks after any more etc and / or which are spammy and scraped anyway. According to link research tools link detox tool, we now have a very low risk profile (I loaded the disavowed links into the tool for it to take into consideration when assessing our profile). I then submitted a reconsideration request on the same day as loading the new disavowed file (on the 26th of April). However today (7th May) we got a message in webmaster central that says our link profile is still unnatural. Aaargh. My question: is the disavow file taken into consideration when the reconsideration request is reviewed (ie is that information immediately available to the reviewer)? Or do we have to wait for the disavow file to flow through in the crawl stats? If so, how long do we have to wait? I've checked a link that I disavowed last time and it's still showing up in the links that I pull down from Webmaster Central, and indeed links that I disavowed at the start of April are still showing up in the list of links that can be downloaded. Any help gratefully received. I'm pulling my hair out here, trying to undo the dodgy work of a few random people many months ago! Cheers, Will
Intermediate & Advanced SEO | | ArenaFlowers.com0 -
Page Crawling Check after Modification Done without staying 7 days
Page Crawling Check after Modification Done without staying 7 days. I have dome modification to my site and uploaded .so i wanna check remaining errors but Moz Crawl web site once per 7 days ,is there any way to check before that . Thank you
Intermediate & Advanced SEO | | innofidelity0 -
How Do I Create Multiple Pages In HTML Sitemap?
I'm working on an html sitemap for our ecommerce site and want to limit the links on each page to less than 100. I've created an article for the initial page, but what is the proper way to go to the next page? Do I create another article page (and so on and so on) until I have the sitemap completed? If so, how do I link from one page to the next? Would my on page text read: "sitemap continued" with anchor text on the link "sitemap page 2.."? It seems like all the sitemaps I've seen just fill one page with links and very little regard for "link saturation" and continuous pages. Thanks!
Intermediate & Advanced SEO | | AWCthreads0