Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
More Singular KW Targeted Landing Pages vs. Less Multiple KW Targeted Landing Pages
So my question is... I have a adopted a site which currently ranks quite well for some industry competitive keywords with a number of poor quality landing pages which specifically target a singular keyword. I am wondering if its worth merging some of these pages together into one authoritative, better quality landing page targeting multiple keywords (as the intent for some of these keywords are largely the same). What i don't want to do is jeopardise the existing rankings in doing so. The alternative option would just be to improve the content on the existing landing pages without merging. What are peoples thoughts on this? Are there any positive case studies out there where merging has had a positive effect? Any help would be great. Regards,
On-Page Optimization | | NickG-1231 -
New Page Not ranking?
One of this client's top keyword is "oak beams". They already rank well in the UK for other related terms like "reclaimed oak beams" at /reclaimed-oak-beams/ and "air dried oak beams" at /air-dried-oak-beams/ We have created a page at /oak-beams/ but this page ranks nowhere? Instead the reclaimed oak beams or air dried oak beams page ranks for the term "oak beams". Any ideas why Google is swapping between those pages and not choosing the /oak-beams/ page? A few notes are that the /oak-beams/ page is newest page on the site and yes I know there are no links pointing to it but there are no links pointing to the other pages either?
On-Page Optimization | | Marketing_Today0 -
Should a website have the same standard headings for product pages? And how does that affect SEO?
Hello! I hope my question is self-explanatory but let me give you an example. Suppose I have website X selling A, B and C and I have a dedicated page for each of those products. For SEO practises, Is it better to have the same webpage structure for all product pages (i.e. the same headings and sections) or is it better to vary the structure of the page? Or is it really not important? Again going back to the example, is it better for product page A, B and C to have the same sections in the same order (i.e. Pricing, Product Details, Product Reviews) or is it better to keep it as a flowing content page (product page A has same information in B and C but all those pages are different with different headings and questions asked depending on the product)? Many thanks!
On-Page Optimization | | MarketingDude0 -
Duplication in landing page
This is driving me mad, I have a site that for some reason google and moz pick up the landing page as a duplicate. They see "mysite/" and "mysite/index.html" as two different pages and giving me warnings for duplication. I have no 301 included at this time and I am using foundation as the base. This is occurring both on a localhost test bed and live....... anyone got an idea how to correct.
On-Page Optimization | | AndyBirtles0 -
How to update a crawl
Hi I crawled my site, it found lots of errors and warning which i have fixed, I told seomoz to recrawl, and it says nothing has changed... It defiantly has though! How do I make it see these changes?
On-Page Optimization | | chrisleonard0 -
What if Paginated Pages all have PageRank?
Paginated Pages, page 2,3,4 etc.... they aren't supposed to have a PageRank, right? If they are only linked to from themselves, only the original page, Page 1, is supposed to be showing PageRank? I'm trying to double check that I am handling this right. I'm not using canonical, or noindex or any of that... just using rel next and prev, which I thought would be fine. Thoughts?
On-Page Optimization | | MadeLoud0 -
Too Many On-Page Links
Hi All, New to SEOMoz, so thanks in advance for any answers! Looking at our Crawl Diagnostics and "Too Many On-Page Links" is first on the list. The site was build with the intention of users being able to quickly get to where they want to go with drop down menus (sub nav), so we built the navigation using bullet points/css. Yes, agreed there are too many links on each page from our navigation, main nav cats are 4 with sub nav about 40, but what is the best way to resolve the problem other then removing most of the links (from the sub nav drop down)? Could we just use the attribute rel=nofollow for the sub nav links? TIA
On-Page Optimization | | bmmedia0 -
A good title for each page on my website.
Dear SEOs, I know these facts: a) not more than 70 chars b) relevant to the page subject - probably best keyword at the beginning My problem - should I rebuild the current structure of my titles which is: Main Page keywords (in my case 4) | domain name - Main Page Sub-page - one click from main page keywords (again same 4) | domain name - Sub Page Name (or as you like Category Name) Specific page with an article / content keywords (again same 4 as on any other) | domain name - category name - title of article same as used in tag To be more precise, should this title be changed for following pages: Sub Page - where really it's page that let's you chose particular subjects (there will be few article for each subjects) so should the keywords be changed to - "chose your category" Specific page with an article - should the category name - title of article be completely removed and first part of title which at the moment contains same 4 keywords used on main page be replaced with keywords specific for the content ? To be honest I find those titles at the moment bit confusing and sort of illogical. But since I can't really change anything in the code I would like to know what's the right way before I keep pressing on the programmer 🙂 Thanks.
On-Page Optimization | | lolskizz0