Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Title Length
Hi Gurus, I understand that it is a good practice is to use 50-60 characters for the a page title length. Google appends my brand name to the end of each title (15 characters including spaces) it index. Do I need to count what google adds as part of the maximum recommended length? i.e.
On-Page Optimization | | SunnyMay
is the maximum 50-60 characters + the 15 characters brand name Google adds to the end of the title or 50-60 including the addition? Many thanks!
Lev0 -
Why are my pages de-indexed?
<form id="form-t3_37nfib9dz" class="usertext" action="http://www.reddit.com/r/SEO/comments/37nfib/why_were_my_pages_deindexed/#"> Hello all, I am very new to SEO. For some reason many of the pages on my site were de-indexed. Specifically the ones linked from this page: However other pages, like the ones linked from this page and this page were not de-indexed. http://www.lawyerconnection.ca/practice-areas/car-accident-injury-lawyers/[1] However the pages linked from this page were not de-indexed: http://www.lawyerconnection.ca/practice-areas/slip-and-fall-lawyers/[2] http://www.lawyerconnection.ca/podcastresources/[3] That first page itself was not de-indexed, just the site that it links to. It just happened today, so maybe I am jumping the gun but I doubt it. When I enter the page into google webmaster tools again and press fetch, one of the child pages, it re-indexes. What could be the problem here? I had someone re-write the content for every city but I have a feeling that there is less differences in the car accidents pages? Is this considered duplicated content do you think? Am I making some other mistake I can't think of? Is it just a one day blip (I doubt it) Let me know, thanks. </form>
On-Page Optimization | | RafeTLouis0 -
Are internal and external footers on all pages of a large website detrimental to seo?
Hi, I am doing an seo clean up of our site - we have about 12 footers on the bottom of each page plus an internal site map including core keywords. Most footers go to internal sites and some go to external sites which are replicated on those external sites linking back to ours. (these sites are part of the same large company but are not related in any way) Are these bad from an seo ranking perspective? Do the look like spam? We get considerable traffic from one of the external footers that is replicated on their site so need to weigh up what the best option is whether to keep the traffic or to loose it as part of the seo clean up. thanks
On-Page Optimization | | Pday0 -
Why Is this Website Not Ranking?
HI There - I have been working on this site: http://limohireauckland.co.nz - primarily for our keyword phrase 'limo hire Auckland' - I am running a campaign for this site from the Pro tools and using the pon-page report card this url achieves an A grade for that particular phrase. The client is working on links in the local community etc but we are not appearing in the SERPS at all for most of the phrases we are optimising for. There has recently been a huge redesign on the site (approx 6 weeks ago) and the old content was not great. Am I missing something really glaringly obvious? Or am I being too impatient?
On-Page Optimization | | AllieMc0 -
Jquery in top of page vs text on bottom page
Is it the best way to use jquery sliders on the top of your page to still get all your text above the fold and score in search engines? for example: http://www.wolf-howl.com/wp-conte... is much better to score high ranks in search engines than http://www.wolf-howl.com/wp-conte... ?? Thanks!
On-Page Optimization | | HMK-NL0 -
Crawl Diagnostics not working?
i've been in the crawl diagnostics of my website. I only have 1 page crawled and no errors. Last Crawl Completed: Jul. 12th, 2012 Next Crawl Starts: Jul. 19th, 2012 Do you have an idea of how to fiw it? Thanks a lot
On-Page Optimization | | Ericc220 -
To Many Links On Page
I'm having a problem on a crawl warning for our main site. The warning is that every one of my pages has to many links, a little over 1,000 on almost all of them. I think this is because our category list on our left hand sidebar has so many categories, and that sidebar appears on every last one of our pages even all the way into our products. Can anyone take a look and tell me if this is the reason why and what I could possibly do about this? Thanks in advance! www.Ocelco.com
On-Page Optimization | | Mike.Bean0 -
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
Why would my homepage be ranked lower (Page Rank 2) than my other pages on the site (PR3) ?
On-Page Optimization | | dmurtagh0