Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Cache missing on SERP
Hi, My website cache is missing on SERP month ago and its not come back anymore also same on description too. I checked on my coding META Description still there.I no idea about this how to retrieve its back to show on SERP.Any suggestion about this?Thank you.,
On-Page Optimization | | Joycee_psp0 -
How to rank well on 2 keywords - 2 separate pages or 1 combined page
Hi, I have a website about allergy. We ar developing new content, and through keyword research I have discovered that "dog allergy" and "cat allergy" are both very common searches. However, the cause, and symtoms are very alike for these 2 types of allergy so it would make sense to combine the two allergies on one page. So my question is: What do I choose to increase my chances to ranke the best I can for both "cat allergy", and "dog allergy"? Should I develop 2 separate pages for cat & dog allergy or should I do a combined page? (We would of course review the texts so no duplicate content/text would be used if we chose to have 2 pages) I would be so greatful for your advice!! Kind regards, Jeanette
On-Page Optimization | | Mylan-GDM0 -
Too Many On-Page Links
If a page has more than 100 links, rather than splitting up the page into multiple pages, is it ok to use name="robots" content="noindex, follow" />? The page in question lists links to articles so the page itself isn't that important to appear in serps, but the articles are the helpful content pages: www.ides.com/articles/processing/injection-molding/
On-Page Optimization | | Prospector-Plastics0 -
View all Page for Product Overview Pages
Hi everybody! We have an ecommerce site with product overview pages, where sometimes there are hundreds of products listed. Usually, we just display 30 and have a button where users can click to see 30 more - or all products listed at once. This is the overview page (as indexed in google): http://www.geschenkidee.ch/aussergewoehnliches.html
On-Page Optimization | | zeepartner
And this is the view-all page: http://www.geschenkidee.ch/aussergewoehnliches.html#all What should I do here? The product overview page will hardly generate more traffic by listing all products (because the overview page will rank for generic keywords, while the product keyword searches will be referred to the specific product pages themselves). I was originally thinking of using rel=canonical pointing to the view-all page. But this would just lead to longer load time. Should we just leave those overview pages or is there a best practice for how to deal with such pages? Thanks for your thoughts on this!0 -
Duplicate page titles
We have search results pages on our site. They share the same page title, there is no real differentiator between the result pages, other than page 1, page 2 etc. How do we de-dup the titles? just add page 1/2/3 etc to the end of them?
On-Page Optimization | | lilibooz0 -
Seomoz tabbed search
Hi, this question might not be SEO specific but i was wondering how seomoz implemented a direct search from the chrome omnibox (you write www.seomoz.org and the press TAB)? BR Carl
On-Page Optimization | | careeron0 -
Why is this page ranking highest?
I've just used Open Site Explorer to compare some sites whose (unpaid) Google ranking I aspire to. They all have higher authority than my site, but the top ranking site out of the 3 I've looked at has the lowest Page Authority, hardly and links (when the others have hundreds), lowest page rank and lowest page trust. In fact, when you look at the top ranking page (ranks #1), it does not even have the search term in it as a complete phrase. One thing I do notice is that it does have 100,000s of linking root domains from one linking root domain. So how can it rank number one on Google?
On-Page Optimization | | Beemer0 -
Will duplicate content supplied from a hotel provider damage my website, or simply just the pages that it appears on?
Hi, I currently have a lot of hotel listings pages with little or no content, as I'm scared that if I place duplicate hotel descriptions on the pages then Google will stop ranking the page. I've found that having descriptions of some kind do help conversion significantly, so I'm considering generating unique hotel descriptions on each main page (page 1 in each set of listings) - these are the pages that Google indexes. On subsequent pages (page 2, page 3 etc.) I'm thinking about resorting to displaying the duplicate affiliate content hotel descriptions - these pages can be crawled but are set to noindex. My question is, do you think this is likely to have an effect on my website in the rankings, and as a result push my primary pages (that contain 100% unique content) down in SERPs. Thanks Mike
On-Page Optimization | | mjk260