Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt advice
Hey Guys, Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you? user-agent: AhrefsBot User-agent: trovitBot
Intermediate & Advanced SEO | | eLab_London
User-agent: Nutch
User-agent: Baiduspider
Disallow: / User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ? I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers?0 -
Invest in a Image Sitemap - Yes or No?
Hey Mozers, 2 part question I'm reaching out to see if you all think Image Sitemaps are totally worth it for a big company. I can totally understand its value for a smaller mom & pop company. With a larger company they would have way more products so is it worth it having an image site map? I cant find examples of image sitemaps online. Would you be able to provide a website that is doing it? I can only find video sitemaps.
Intermediate & Advanced SEO | | rpaiva0 -
Need advice and smart solution for H1
Hi! I've been quite a long time trying to find a solution to the html structure of a webpage that I'am auditing right now. But I need your valuable help! The problem is the following: Actually, they have different sections inside. The structure is something like this Masters - Degrees - Programs - About us - News And if you go to the Masters Section you will find something like this Masters Master number 1 in Tourism (brand.com/master/master-number-1-tourism) Presentation
Intermediate & Advanced SEO | | teconsite
(brand.com/master/master-number-1-tourism) Objectives
(brand.com/master/master-number-1-tourism/objectives) Professional opportunites
(brand.com/master/master-number-1-tourism/professional-opportunities) Faculty
(brand.com/master/master-number-1-tourism/faculty) Qualification
(brand.com/master/master-number-1-tourism/qualification) Financial
(brand.com/master/master-number-1-tourism/financial) Master number 2 with a long name Presentation Objetives Profesisional opportunities Faculty ... Master number 3 in Sports and so on The Degrees section, has inside exactly the same structure with the same names. My doubt is related with the use of h1 tag What would be the best h1 strategy for each content page? Each master has 6 pages (presentation, objectives, faculty,...) For page Objetives,
brand.com/master/master-number-1-tourism/objectives If I choose to use as H1 just the word Objetives, what will happen is that I will have a lot of pages (one per master, degree or program), with the same H1, because each master will have its own page Objectives. If we have 10 masters + 10 degrees + 2 programs y will have 22 pages with the same H1 Objecives If I choose to use as H1 the following: "Objectives of the Master Number 1 in Tourisim and so on with long name" it will be difficult for the users to visually see the difference between the different pages. for instance Objectives of the Master Number 1 in Tourisim and so on with long name and Faculty of the Master Number 1 in Tourisim and so on with long name because they only differ in one word. What do you think of this solution? Objectives Master Number 1 in Tourisim and so on with long name is it correct to do this inside the h1? or would you use combinations of h1 and h2 like these
h1: Objectives
h2: Master Number 1 in Tourisim and so on with long name would be this appropiate?0 -
100% links within the website - could it rank?
Hi, If I create blog posts inside my website and link it back to my website, Would it still rank? I understand that its better to get links from the established domain. But I just wonder what kind of impact would my site have if the blog posts within my site link back to itself for ranking. Please let me know. thanks
Intermediate & Advanced SEO | | zsyed0 -
Same website, seperate subfolders or separete websites? 12 stores in two cities
I have a situation where there are 12 stores in separate suburbs across two cities. Currently the chain store has one eCommerce website. So I could keep the one website with all the attendant link building benefits of one domain. I would keep a separate webpage for each store with address details to assist with some Local SEO. But (1) each store has slightly different inventory and (2) I would like to garner the (Local) SEO benefits of being in a searchers suburb. So I'm wondering if I should go down the subfolder route with each store having its own eCommerce store and blog eg example.com/suburb? This is sort of what Apple does (albeit with countries) and is used as a best practice for international SEO (according to a moz seminar I watched awhile back). Or I could go down the separate eCommerce website domain track? However I feel that is too much effort for not much extra return. Any thoughts? Thanks, Bruce.
Intermediate & Advanced SEO | | BruceMcG0 -
Purpose of a Blog in a website
How internal blog or external blog is helpful in SEO?why it is good to have a site with blog?
Intermediate & Advanced SEO | | Alick3000 -
Domain advice needed, please
Could i get a little domain advice please. Launching a new website project and want to put it on a domain we already own (both domains are in the same niche as the larger project). The new project will be aimed at the UK market. The choice is: .co.uk we own with a good name, however it's Domain Authority rank is 7 and it's only about 8 months old. .com domain which is 6 years old, has a Domain Authority rank of 33 but is not as good a domain name. The Competitive Link Analysis tells me that the rivals for the keywords we would be targeting are between 24 and 42. Which domain would people go with? All things equal it would be a fair guess that the older, higher Domain Authority ranked .com will require less work to rank in the engines, however it's not as good brand wise. Thanks Carl
Intermediate & Advanced SEO | | Grumpy_Carl0 -
Website rebranding, what should I worry about?
Hey guys, A client of mine will be doing a rebranding exercise, this include changing their brand name and their domain name. They are considered a well known brand within their industry (Their brand name shows up in Google's "Search Related to..." section) My question is: Apart from making sure all 301 are put in place,changing all the links to point to the new domain and doing PR exercise, is there anything else I should keep in mind / be aware of to ensure a smooth transition? Also can anyone come up with possible issues we might encounter during the move? Apart from having a significant drop in traffic and rankings? Thanks, Clement
Intermediate & Advanced SEO | | NextDigital510