Sitemap Query
-
I've decided to write my own sitemap because frankly, the automated ones pull all kinds of out of I don't know where. So to get around that, manual it is. But I have some products appear in various categories, should I still list every product in each category in the sitemap, regardless of some being duplicates, or should I choose the most relevant category and list them there?
I do have a canonical URL extension which should resolve any duplicate content I have.
-
Hi,
It's ideal if your XML sitemap is an accurate representation of URLs you want indexed, i.e. the canonical versions. If you're using Screaming Frog to manually build your sitemaps, you make sure the 'Include Canonicals' button is unchecked. Doing so will trigger Screaming Frog to automatically leave out any URLs that canonicalize towards another URL, thus solving your problem.
-
If you have canonical you can put all the links, because in this way Google-Bot will crawl all the pages and see easier which have canonical and which is unique.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
PLEASE HELP - Old query string URL causing problems
For a long time, we were ranking 1st/2nd for the term "Manual handling training". That was until about 5 days ago when I realised that Google had started to index not only a query stringed URL, but also an old version of the URL. What was even weirder was that when you clicked on the result it 301 redirected to the page that it was meant to display... The wrong URL that Google had started to index was: www.ihasco.co.uk/courses/detail/manual-handling?channel=retail The correct URL that it should have been indexing is: https://www.ihasco.co.uk/courses/detail/manual-handling-training I can't get my head around why it has done this as a 301 was in place already and we use rel canonical tags which point to the main parent pages. Anyway, we slapped a noindex tag in our robots.txt file to stop that page from being indexed, which worked but now I can't get the correct page to be indexed, even after a Google fetch. After inspecting the correct URL in the new search console I discovered that Google has ignored the rel canonical on the page (Which points to itself) and has selected the wrong, query stringed URL as the canonical. Why? and how do I rectify this?
Intermediate & Advanced SEO | | iHasco1 -
2015/2016 Sitemaps Exclusions
Hello fellow mozrs!
Intermediate & Advanced SEO | | artdivision
Been working on a few Property (Real Estate for our American friends) websites recently and and two questions that constantly come up as we spec the site are: 1. What schema (schema.org) should the website use (throughout all pages as well as individual pages). Did anyone found that schema actually helped with their ranking/CTR?
2. Whilst setting up the sitemaps (usually Yaost is our preferred plugin for the job), what page would you EXCLUDE from the site map? Looking forward to some interesting comments.
Dan.0 -
Should we be showing last modified date for each page in our sitemap?
We're not currently displaying the last modified date in our sitemap, e.g.: <url><loc>http://www.soschildrensvillages.org.uk/about-our-charity</loc></url> Are their any advantages to including this data? One benefit that occurred to us is that it will enable Google to determine which pages have fresh content and which are therefore worth crawling, helping Google index beneficial changes quicker. Thanks!
Intermediate & Advanced SEO | | SOS_Children1 -
Do you lose link juice when stripping query strings with canonicals?
It is well known that when page A canonicals to page B, some link juice is lost (similar to a 301). So imagine I have the following pages: Page A: www.mysite.com/main-page which has the tag: <link rel="canonical" href="http: www.mysite.com="" main-page"=""></link rel="canonical" href="http:> Page B: www.mysite.com/main-page/sub-page which is a variation of Page A, so it has a tag I know that links to page B will lose some of their SEO value, as if I was 301ing from page B to page A. Question: What about this link: www.mysite.com/main-page?utm_medium=moz&utm_source=qa&utm_campaign=forum Will it also lose link juice since the query string is being stripped by the canonical tag? In terms of SEO, is this like a redirect?
Intermediate & Advanced SEO | | YairSpolter0 -
XML sitemaps questions
Hi All, My developer has asked me some questions that I do not know the answer to. We have both searched for an answer but can't find one.... So, I was hoping that the clever folk on Moz can help!!! Here is couple questions that would be nice to clarify on. What is the actual address/name of file for news xml. Can xml site maps be generated on request? Consider following scenario: spider requests http://mypage.com/sitemap.xml which permanently redirects to extensionless MVC 4 page http://mypage.com/sitemapxml/ . This page generates xml. Thank you, Amelia
Intermediate & Advanced SEO | | CommT0 -
Hreflang Sitemap
Hi all, Have you ever created an hreflang sitemap using in-house resources or a third-party company for a group of over 70 sites, each with hundreds of pages and all with localised URLs? If so, would you mind sharing how you did it, or the contact details of the company you used? In this specific case there is nothing in the URL or code that I can use to group the alternatives automatically. Thanks, Carlos
Intermediate & Advanced SEO | | Carlos-R0 -
Is there a maximum amount of pages that should be added on a sitemap daily?
I started a new music site that has a database of 8,000,000 songs and 500,000+ artists that we are cross referencing with free & legal content sources. Each song essentially has its own page. We are about to start adding links to a sitemap and wanted to find the best practices. Should we add all 8,000,000+ links at once? Should we add a maximum amount a day? Maybe max 5,000? What are the pros and cons of slowly adding the pages or adding them all at once. Any risks? At the rate google is crawling our page it will take 8 years to have all of our songs indexed (It would be very hard to crawl all of our songs as our system is more of an app). I wan't to play it safe and not do anything that will come off as spammy. I have been trying to find some actual evidence on what the best course of action is. Thanks in Advance!
Intermediate & Advanced SEO | | mikecrib10 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110