Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question about my website ranking
Hi everyone I have sports website. My website DA, DR is good. I have created good quality backlinks. But still my website ranking is not improving. Can you guide me how to improve my website ranking. Thanks in advance!
On-Page Optimization | | fktos.tam.906n0 -
Should you do on-page optimization for a page with rel=canonical tag?
If you ad a rel=canonical tag to a page, should you still optimize that page? I'm talking meta description, page title, etc.
On-Page Optimization | | marynau0 -
Page title contents
In my page title, I have my product name. Is it beneficial to also include another keyword like: Buy wedding dress online Australia: e..g. (page title) amelie wedding dress | buy wedding dress online Australia. Or is it better just using: Amelie wedding dress
On-Page Optimization | | CostumeD0 -
Why is my contact us page ranking higher than my home page?
Hello, It doesn't matter what keyword I put into Google (when I'm not signed in and have cleaned down my browsing history) the contact us page ranks higher than the home page. I'm not sure why this is, the home page has a higher page authority, more links and more social media shares, the website is an established one. When I have checked Google Analytics my home page gets more people landing on it than the contact us page. It looks like people are ignoring the contact us page and scrolling down until they find the home page. I'd appreciate any help or advice you might have. Thank you.
On-Page Optimization | | mblsolutions2 -
Page rank question
Hi all, pretty standard SEO question that I'm sure you are all getting asked day in, day out... the following page: http://www.3wisemonkeys.co.uk/dvd/portable-dvd-player-car/ It's targetting keywords "portable dvd", "portable DVD player", "portable DVD players" & "portable dvd player for car". The page gets a grade A via Moz analytics On-page grader for the keywords. We have incoming links and I believe the page is structured correctly with all meta tags, H1 etc and tried not too overload the tags with keywords. We have the keywords in the URL etc. We are still stuck on page 2 for all phrases. Any thoughts or suggestions as to what we are doing wrong please? Any help would be great.
On-Page Optimization | | jasef0 -
Too Many On-Page Links
Hello. So, my SEO team has worked very hard to finally resolve RogerBot/GoogleBot specific Crawl Errors either manually or programmatically can be fixed for our Budget Blinds USA Pro Campaign. We've done a good job even if a lot of it came from Robots.txt file entries as this was the most efficient way our client chose to do it. Good news is most of it is CMS configuration and not bad site architecture. That being said our next big volume of Crawl Errors is "Too Many On-Page Links". Our Moz DomainRank is 61. Our client, on this new version of the website, added a large nav-based footer which has duplicate links from the Header Main Navigation. I believe our solution is to put in No-Follow Metatags at the Footer Link Level, so we don't zap Page Authority by over-dividing as you recommend. Is this the best way to resolve this? Is there any risk in this? Or is a 61 DomainRank high enough for RogerBot and GoogleBot to crawl these anyway? Please advise,
On-Page Optimization | | Aviatech0 -
Autogenerated pages
My main product is database conversion software. As it supports tons of databases, it's fairly easy to generate thousands of landing pages simply by variating source/target database names, connection information etc. In fact, I autogenerated almost 25k pages that way. As I didn't want to jeopardize my main site, I placed all that content to a new microsite (www.fullconvert.com) which had no history and no inbound links. Results were nice - site is live two months and in second month already had 1300 visitors. Now, my question is - should I create the same thing on my (old and rather authoritative) main site www.spectralcore.com? I could use a different template to avoid duplicate content. Of course, my main concern is being penalized by Google. In my opinion, this autogenerated content is fine because it provides (tons of) laser-focused landing pages, so visitors will instantly recognize they found what they're looking for. But Google might disagree! What do you think? Is there a danger in trying to leverage authority of my main site in adding 20k+ autogenerated pages with inbound no links to them?
On-Page Optimization | | metadata0 -
On-page keyword usage
SEOMOZ gave me all zeros for keyword usage. Why? The site is www.grass2greens.com and the keywords are "Asheville Landscaping Edible." The site includes these words in the title page and throughout the body text. I am not really sure, but maybe one cause for these low keyword usage ratings might be redirects or some meta tag issues, but I am really not sure. Any ideas?
On-Page Optimization | | dcaudio0