Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If I put 'keyword/url' combination to 'stop run weekly', will it dissapear from the summary page in the on-page grader?
The summary page of the on-page grader chooses the keyword and url combination itself. Now if I choose another combination, I would like the former to dissapear from the summary page. The only option is 'stop running weekly'. But will it disappear from the list also?
On-Page Optimization | | jongeneelbv0 -
Which page to rank for a Keyword? Home Page or Deep Page?
So, we have a situation where there is one particular keyword we want to rank for. We have been up and down over the years, at our best probably position 4-5, and now at 20ish. Thats for our home page of course, which the majority of our linking is probably pointing at. We also have a sub page which is optimised for that particular service. The term is "web design brisbane".
On-Page Optimization | | MauriceKintek
So as you can imagine, Web Design is in itself a service and we offer others. Should we optimise our home page for it and remove the sub page?
Keep the sub page because its one our services and optimise both?
Do some kind of canonical thing?
Change our interlinking? All our competitors home pages seem to be the ones that rank, and it feels and looks better in results if its the home page, but if switching up to our sub page is better im all ears. Also if our sub page is somehow hurting or leaking SEO from the home page, id like to know as well. Would prefer to not have to provide a link, due to competition but if someone wants to know we can always PM.0 -
Optimizing web page
Hi I have implemented most of the suggestions, which are offered threw SEOMOZ on our web site www.zaposlitev.net. But rankings are not improving. Could anyone help please? Thanks Tomaz
On-Page Optimization | | tomaz770 -
Why are some of page indexed and others not
I have created a site structure like this: domain/for-sale/brand domain/for-sale/brand-model domain/for-sale/brand-model/pg1 domain/for-sale/brand-model/pg2 domain/for-sale/brand-model/pg3 etc.... I cannot understand why the domain/for-sale/brand-model does not seem to be indexed, yet the domain/for-sale/brand-model/pg6 is? This is a new site, but I cannot understand why this URL would be indexed without the others... Any ideas? My home pages has links to the domain/for-sale/brand, this page has links to domain/for-sale/brand-model1, domain/for-sale/brand-model2 etc, each of these pages have links to domain/for-sale/brand-model/pg1, domain/for-sale/brand-model/pg2 etc...
On-Page Optimization | | MirandaP0 -
Limiting On Page Links
Right now, we have about 160 or so links on the home page. It's been recommended that we keep it to under 100, though that's not as big of a deal as it once was. Is it helpful to make a bunch of those links "nofollow" in order to preserve link juice? Is it going to make a difference, or be at all helpful? I assume it won't be harmful, especially as a bunch of them are to the same page but on different sections of the page. Would live your advice and thoughts! Thanks!
On-Page Optimization | | DeliaAssociates0 -
How do you fix on page SEO ?
I have been trying to push my foundation website in organic search results for competitive keywords , i have been not been so consistent in raking our website in top search results of Google. can some one recommend the guidelines and activities which can really push my websites to Google first page. More Info: about our foundation We are the worlds largest school meal run ngo in the world feeding over 1.3 million school children in India Wesite url www.akshayapatra.org
On-Page Optimization | | AkshayaPatra0 -
SEO for spanish website
Hi, A client has given us the site http://www.comtranslations.com/Home.html for optimization. He wants to optimize only the spanish part ( the link is on the top right ). By clicking on the link Espanol, the url opened is - http://www.comtranslations.com/Principal.html. He wants seo for this website for spanish keywords. The keywords are - Traducción
On-Page Optimization | | seoug_2005
Traductor
traducir español inglés
traducción My question is how do we go about this ? Shall we purchase a software that translates spanish to english ? Thanks0 -
Avoiding "Duplicate Page Title" and "Duplicate Page Content" - Best Practices?
We have a website with a searchable database of recipes. You can search the database using an online form with dropdown options for: Course (starter, main, salad, etc)
On-Page Optimization | | smaavie
Cooking Method (fry, bake, boil, steam, etc)
Preparation Time (Under 30 min, 30min to 1 hour, Over 1 hour) Here are some examples of how URLs may look when searching for a recipe: find-a-recipe.php?course=starter
find-a-recipe.php?course=main&preperation-time=30min+to+1+hour
find-a-recipe.php?cooking-method=fry&preperation-time=over+1+hour There is also pagination of search results, so the URL could also have the variable "start", e.g. find-a-recipe.php?course=salad&start=30 There can be any combination of these variables, meaning there are hundreds of possible search results URL variations. This all works well on the site, however it gives multiple "Duplicate Page Title" and "Duplicate Page Content" errors when crawled by SEOmoz. I've seached online and found several possible solutions for this, such as: Setting canonical tag Adding these URL variables to Google Webmasters to tell Google to ignore them Change the Title tag in the head dynamically based on what URL variables are present However I am not sure which of these would be best. As far as I can tell the canonical tag should be used when you have the same page available at two seperate URLs, but this isn't the case here as the search results are always different. Adding these URL variables to Google webmasters won't fix the problem in other search engines, and will presumably continue to get these errors in our SEOmoz crawl reports. Changing the title tag each time can lead to very long title tags, and it doesn't address the problem of duplicate page content. I had hoped there would be a standard solution for problems like this, as I imagine others will have come across this before, but I cannot find the ideal solution. Any help would be much appreciated. Kind Regards5