Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Product Page Links
I have a product category page at https://www.hurtlegear.com.au/s1000rr/ which currently has 38 products on it. Problem is, all the product titles start with the name of the text: "bmw s1000rr" (because that's what they are) - so that means there are 38 anchored internal links on that page, all starting with the same keyword. You can see how that might look to the Google crawler. Recently that page dropped from around 15 to outside the top 100, and Moz tells me that the page is keyword stuffed with "bmw s1000rr" (no suprise) so I'm guessing that may be the reason the page has disappeared out of the SERPs. I don't really want to change all the product titles (then they wouldn't make sense) so I'm just wondering if there is any way around this? Is there some way of telling Google that this is a product category page and therefore to ignore the anchor text in all of those product links? Can/should the links have some kind of markup on them? Or is the page beyond help? Basically I'm looking at a way of keeping the product titles as they are, but avoiding a page penalty from Google somehow. I'm a bit of a newbie, any suggestions would be most appreciated. Cheers, Graeme
On-Page Optimization | | graeme720 -
What to do about pages I have deleted?
I have been working through the dead links on my page and recreating the page with new content for those pages that it still makes sense to have on the site. But I have a few that were just changes of the title, spelling mistakes or other ways of saying the same thing In other words I created a page called "areas of the UK we cover" but decided to change it to "areas covered" However, I must have created links to this page and now it is a dead link with a page authority of 19 I think it would be spammy to have two pages, one called "areas covered" and the other called "areas of the UK we cover. It's not a disallow in Robots.txt because the page does not exist Please note I do not have access to the header to add code for a 301 redirect. I'm still using webs.com but not for new sites. I also have a page called singing telegrams london, that I changed from singagrams london. These are two words for the same thing but they are two very different keywords would it be ok to recreate this page and create content for singagrams london. Help is much appreciated
On-Page Optimization | | singingtelegramsuk0 -
Duplicate Page Titles in Crawl Errors (although Google is rewriting in serps ??)
Hi Im working on a client/project and crawl report is showing thousands of dupe page titles In the case of the blog/news section its aprox 50 since aprox 50 posts and they all have the same meta-title: "Brand News | Brand" as opposed to: "Title Unique to Page/Topic/KW Relating to Content | Brand" Since these are the main content pages we want to rank (in addition to the main site category pages) then i have instructed dev must prioritise populating these pages meta-titles with the actual post/article titles, as per the latter version of the above example. (I should mention that i have requested they fix all dupe titles but main content pages are the priority). Whilst this will reduce the number of dupe titles in crawl error/warning report which is a good thing, is it actually likely to increase the ranking of these news/content pages given that Google does seem to be rewriting the titles correctly in the serps based on the page content ? Many Thanks in advance for your input
On-Page Optimization | | Dan-Lawrence0 -
Home page or landing page?
Hello, I want to ask a question related to that - Should we put keywords in the home page title if we wish to position another landing page better for particular keywords? I have read in one website about SEO that it's good the main keywords of your website to be positioned in homepage title also. f.e. Let's say we have website about web-design and our company is named Company Ltd. The title of the home page is "Company Ltd. - Web design, SEO, etc" We have also another inner page named "Web design | Company Ltd.". So should we leave the first page name only "Company Ltd." and the landing page's name "Web design | Company Ltd." . I don't know if they both have the same keyword in their title they won't compete with each other.
On-Page Optimization | | HrishikeshKarov0 -
800 number on website
Hi, My client just sent me an 800 number that he would like to use to replace his number on the website. I know that it is best to keep a local phone number on the website and across all citations for NAP/Local SEO reasons. Is there anywhere that I could still incorporate the 800 number and not have it affect SEO? Thanks, Erin
On-Page Optimization | | HiddenPeak0 -
How to improve On-Page Grades for Top Ranking Pages
please help me - i dont know or understand how to improve on-Page Grades for Top Ranking Pages
On-Page Optimization | | pwwukpw0 -
Why does my on-page report card say my page title is 403 forbidden when its not?
I'm trying to get on top of my on page stuff and I'm going through the SEO Moz on-page report cards and it says I'm scoring a fail on certain elements within the 'critical' and 'high importance' factors as my page title is '403 forbidden' but when I go on to my site, my sites CMS it's not '403 forbidden' it's the text I entered?
On-Page Optimization | | jamesj35mm0 -
Duplicate Page Title
Hi Guys, First off, it's an honour to be a part of this awesome community. I'm using WordPress and getting top 3 rankings for great keywords and I'm very excited, however my page titles are in this format "keyword optimised title here - site name here" eg: "This is my keyword - this is the name of my blog", "This is another keyword - this is the name of my blog", "This is a longtail keyword - this is the name of my blog" SEOMoz is reporting errors because of duplicate page title tags due to the "this is the name of my blog" being in every page title. Will this hurt my rankings? Thanks in advance and keep up the great work! Cheers, Troy.
On-Page Optimization | | TroyDean710