XML sitemap generator only crawling 20% of my site
-
Hi guys,
I am trying to submit the most recent XML sitemap but the sitemap generator tools are only crawling about 20% of my site. The site carries around 150 pages and only 37 show up on tools like xml-sitemaps.com. My goal is to get all the important URLs we care about into the XML sitemap.
How should I go about this?
Thanks
-
I believe it's not a significant issue if the sitemap encompasses the core framework of your website. As long as the sitemap is well-organized, omitting a few internal pages is acceptable since Googlebot will crawl all pages based on the sitemap. Take a look at the <a href="https://convowear.in">example page</a> that also excludes some pages, yet it doesn't impact the site crawler's functionality.
-
Yes Yoast on WordPress works fine for sitemap generation. I would also recommend that. Using on all of my blog sites.
-
If you are using WordPress then I would recommend to use Yoast plugin. It generates sitemap automatically regularly. I am also using it on my blog.
-
I'm using Yoast SEO plugin for my website. It generates the Sitemap automatically.
-
My new waterproof tent reviews blog facing the crawling problem. How can I fix that?
-
use Yoast or rankmath ot fix it
آموزش سئو در اصفهان https://faneseo.com/seo-training-in-isfahan/
-
Patrick wrote a list of reasons why Screaming Frog might not be crawling certain pages here: https://moz.com/community/q/screamingfrog-won-t-crawl-my-site#reply_300029.
Hopefully that list can help you figure out your site's specific issue.
-
This doesn't really answer my question of why I am not able to get all links into the XML sitemap when using xml sitemap generators.
-
I think it's not a big deal if the sitemap covers the main structure of your site. If your sitemap is constructed in a really decent structure, then missing some internal pages are acceptable because Googlebot will crawl all of your pages based on your site map. You can see the following page which also doesn't cover all of its pages, but there's no influence in terms of site crawler.
-
Thanks Boyd but unfortunately I am still missing a good chunk of URLs here and I am wondering why? Do those check on internal links in order to find these pages?
-
Use Screaming Frog to crawl your site. It is free to download the software and you can use the free version to crawl up to 500 URLs.
After it crawls your site you can click on the Sitemaps tab and generate an XML sitemap file to use.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemap: unique sitemap or different sitemaps by Country
Hi guys, i have a question about sitemaps. We are doing an international site, e.x. www.offers.com for landing page and www.offers.com/br for brazil, www.offers.com/it for italy, etc... i don't if we should do an unique sitemap for all countries or separate sitemaps by country, e.x.: unique sitemap: www.offers.com/sitemap.xml - including all sitemaps www.offers.com/br/sitemap.xml - sitemap for brazil market only. Thank you
Intermediate & Advanced SEO | | thekiller990 -
Google can't access/crawl my site!
Hi I'm dealing with this problem for a few days. In fact i didn't realize it was this serious until today when i saw most of my site "de-indexed" and losing most of the rankings. [URL Errors: 1st photo] 8/21/14 there were only 42 errors but in 8/22/14 this number went to 272 and it just keeps going up. The site i'm talking about is gazetaexpress.com (media news, custom cms) with lot's of pages. After i did some research i came to the conclusion that the problem is to the firewall, who might have blocked google bots from accessing the site. But the server administrator is saying that this isn't true and no google bots have been blocked. Also when i go to WMT, and try to Fetch as Google the site, this is what i get: [Fetch as Google: 2nd photo] From more than 60 tries, 2-3 times it showed Complete (and this only to homepage, never to articles). What can be the problem? Can i get Google to crawl properly my site and is there a chance that i will lose my previous rankings? Thanks a lot
Intermediate & Advanced SEO | | granitgash
Granit FvhvDVR.png dKx3m1O.png0 -
Consolidate Local sites to one larger site
I am a partner in a real estate company that operates in 10 different markets across the country. Each of these markets has it's own individual domain. My question is should we consolidate each of these markets into one domain that services all markets? What would we possibly gain or lose from an organic traffic standpoint? In some of our more established markets (Indianapolis, Las Vegas, Tampa, Orlando and Charlotte) our organic traffic accounts for 50-60% of our total traffic. In some of our newer markets (Denver, Phoenix, San Diego) it accounts for less than 15%. We do operate under two different brand names. EasyStreet Realty and Highgarden Real Estate. EasyStreet has been around since 2000 with most of our Highgarden sites only up for 6-24 months. Another question is we are considering converting all EasyStreet divisions to Highgarden. I am a little reluctant to do so, since most of our organic traffic is coming from our EasyStreet sites. Thoughts? You can find links to all our sites at www.easystreetrealty.com or www.highgarden.com Thank you in advance for your insight.
Intermediate & Advanced SEO | | EasyStreet0 -
Why does this site rank above us?
We own www.discountbannerprinting.co.uk and over the last 8 months have built some decent guest post, charity and customer links but still we seem to be beaten on good words such as banners, banner, vinyl banner, pvc banner etc by this website www-signfirm.com we just cannot figure out how this is happening and would be very grateful if someone with great wisdom could give us an in-site into why this is happening and we would be very grateful..
Intermediate & Advanced SEO | | BobAnderson0 -
XML question - not finding all of the pages
When I run http://www.xml-sitemaps.com/ on my site, it doesn't find all of my pages. The pages do not have any no follows in them (I thought that was the original problem). Has this happened to anyone else? What is the solution?
Intermediate & Advanced SEO | | digitalops0 -
Niche sites: how to optimize them?
Dear SEOmozzers, I am focusing on those keywords I find using the "Keyword Analysis" tool that are not too competitive (among 20%-30% competitiveness). I then buy domains that include those keywords. The sites are in Italian and targeting the Italian search engines, so the competition is lower than it would be in the US. Basically, I'd like to build niche sites and I'd like to ask a few questions that I hope somebody with a good experience in this field can answer: How do I optimize a niche site. Specifically, how do I go about link building? How many backlinks should I get to see some results? How long does it take for a typical niche site to start appearing in the search engines for a certain keyword, after launching an effective link building campaign? Please kindly provide any recommendations you believe to be important when building niche sites. For example, is there a company/professional you know that specializes in this field and who is trustworthy/reliable? Thank you very much for your help. All best, Sal
Intermediate & Advanced SEO | | salvyy0 -
Which is best structure for Multiple XML Sitemap?
I have read such a great blog posts on Multiple XML Sitemaps on following websites before a week. SEOmoz Distilled Google Webmaster Central Blog Search Engine Land SEO Inc I have created multiple XML sitemaps for my eCommerce website with following structure and submitted to Google webmaster tools. http://www.vistastores.com/main_sitemap.xml http://www.vistastores.com/products_sitemap.xml But, I am not satisfy with my second XML sitemap because it contain more than 7K+ product page URLs and looks like very slow crawling by Google! I want to separate my XML sitemap with following structure. With Root Level Category http://www.vistastores.com/outdoor_sitemap.xml http://www.vistastores.com/furniture_sitemap.xml http://www.vistastores.com/kitchen_dining_sitemap.xml http://www.vistastores.com/home_decor_sitemap.xml OR::: End Level Category http://www.vistastores.com/table_lamps_sitemap.xml http://www.vistastores.com/floor_lamps_sitemap.xml . . . . . . . etc.... So, Which is best structure for Multiple XML Sitemap?
Intermediate & Advanced SEO | | CommercePundit0