Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Missing canonical tag error - office pages
Moz is throwing an error for our office pages (10 office pages in the format /office/location-1; /office/location-2 etc) but the content is different. How should we handle the canonical tag? Thanks
On-Page Optimization | | AztekMedia0 -
2 Similar Pages
Hello, I have two very similar pages. The first page is an apartment rental page with a map, rental listings and some neighborhood data below. The neighborhood data includes useful info about the area with photos, text about the area, crime rates, avg. rental rates, etc. The second page is a neighbourhood guide that includes virtually the same data as the rental page, but in longer form ie. more photos, more text, etc. I want the rental page to rank, while ranking the neighborhood page is not important as it would be used more for link bait. But since the information on the two pages is the same, I don't want them to compete with each other. I'm thinking of putting a cannonical tag on the neighborhood page pointing back to the rental page. Is that the correct thing to do in this instance? Thanks for your help. J
On-Page Optimization | | torontojon0 -
Critical Page Problems.
As I design my new pages I place them into the page optimizer. When I place the keyword in that I am trying to rank for my page is coming up as an A. With that said, I am still receiving error messages with critical fix results. Here is a screenshot of this: http://screencast.com/t/DuUrP8xM Curious on how I should fix this? The pages are directly on my blog which makes it accessible to search engines ( So I think). Any input would be greatly appreciated.
On-Page Optimization | | Jasonalanmagic0 -
SEOmoz's On-page Checker upto date?
Helllo Mozzers, Just wondering if SEOmoz's on-page optimisation checker is upto date with google recent updates? If not... what do you suggest?
On-Page Optimization | | Prestige-SEO0 -
Why is SEOMoz telling me that meta descriptions are missing?
We have descriptions using an Ektron CMS on a .NET platform. IT is wondering if using "meta ID" vs "meta name" is the issue? Here is an example of the HTML code: <meta http-equiv="<a class="attribute-value">Content-Language</a>" name="<a class="attribute-value">Language</a>" content="<a class="attribute-value">en</a>" /> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaKeywords</a>" name="<a class="attribute-value">Keywords</a>" content="<a class="attribute-value">hunting, fishing, outdoor, sportshooting, rodeo, off-road, hunt, fish, outdoors, community, shotgun, rifle, bass, trout, sportfishing, tarpon, ATV, television</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaDesc</a>" name="<a class="attribute-value">Description</a>" content="<a class="attribute-value">Deep-cranking technique lands Starks his second Elite Series victory.</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaSubject</a>" content="" name="<a class="attribute-value">subject</a>">meta> <meta id="<a class="attribute-value">ctl00_metaTagControl1_metaRating</a>" content="" name="<a class="attribute-value">rating</a>">meta>
On-Page Optimization | | cmoise0 -
Tilde + On-page Optimization
I'm analyzing my Hispanic site www.escolares.net, and I noticed that on-page optimization rank with an F pages using a tilde in target keywords. Does anybody know if SeoMoz software can completely understand in their analysis special characters as tildes? Cristian Majluf
On-Page Optimization | | motorpod1 -
Landing page too long
In my first seomoz test I have many tilte pages for products that are over 70 charachers long. The part #'s are long like 10-782-10-10-10-PPxxxx etc. All these part #'s are not my key products and I could delete or truncate. My question is if the part numbers are not that important, is it OK to leave them as is or is ranking being damaged because they exist?
On-Page Optimization | | Wales0 -
Creating New Pages Versus Improving Existing Pages
What are some things to consider or things to evaluate when deciding whether you should focus resources on creating new pages (to cover more related topics) versus improving existing pages (adding more useful information, etc.)?
On-Page Optimization | | SparkplugDigital0