Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving from Bigcommerce to Woocommerce on WP. Should we redirect size pages into one page?
We are moving from Bigcommerce to Woocommerce on WP. On Bigcommerce, due to some bizarre reasoning the previous developer had 3 separate URLS for the same product in different sizes - S, M and L. Now we plan to have one product page where the sizes can be selected and 301 redirect the 3 urls to the new one. Is this advisable? Or should we just have 3 separate pages. OR should we have one of the sizes pages as the new page and then redirect the other 2 to this one? I ask this because the site has a LOT of ranking power and we do not want to jeopardise that.
On-Page Optimization | | MashBonigala0 -
Page Layout Updates and Mobile Pages with Ads
I have been trying to do some research on the Page Layout Algorithm and Top Heavy Ads and much of what I read does not mention about mobile pages as apposed to desktop. I am curious if the Page Layout updates can be effected by mobile pages as well and if there is any good articles on this subject. Also is this Algorithm been incorporated into its regular algorithm or do we still have to wait for refreshes to see the impact? Cesar
On-Page Optimization | | cbielich0 -
Webs Pages not correct
First of all I am a beginner but that will show with this question. I have moved my website from Wix to Web.com. I had the dns I think that is correct re-direct my site to web.com. Since then some of the pages from wix still show up even though they are no long apart of my site. Secondly when I go to google and type in cestoday.com some of the pages gives me a 401 or 404 error code, and others do not link to the proper page on my website. They go to my web-site but not the correct page. What do I do. Thank you Dave
On-Page Optimization | | redsman944
cesyes@hotmail.com In reference to my last question about moz not being able to see my pages and giving me an "F" I thought you might find it interesting to know that that same page once moved got an A amazing.0 -
Spammy page titles
Over the last couple of weeks, I have noticed that Google aren't showing the page titles for my online shop anymore. They're set up with a third party plug-in piece of software, and while it's an old version of the software, the developer said it wouldn't be causing issues. They have suggested that I re-write my page titles to be less spammy. The thing is, Google haven't attacked just spammy looking titles, they're just taking a swoop through my whole site and not showing any of my page titles in their search results. I'm getting "Category Name - Shop Name" showing. Here's some of the page titles no longer appearing and I honestly have no idea how to rewrite these to not be spammy. Are there any good articles on what's spammy and what isn't? "Coconut oil - best tasting in Australia. Buy online from <my business="" name="">"</my> "Discount Vitamix Blender. Best deal in Australia. Buy online from <my business="" name="">."</my> "Natural & Organic skin care for the face | buy online in Australia from <my business="" name="">."</my> There are others that are showing the real page titles, but I think it's only a matter of re-indexing before they're all not showing. Any clue?
On-Page Optimization | | sparrowdog0 -
Too Many On-Page Links
Hi, I did a SEOmoz campaign and got results today, One of the results is Too "Many On-Page Links" when i am drilling down, i see that that's include inside links. for example, i sale food, i have my main department window - inside i have 30 products - each product is linked to a detailed page about the product. so automatically i have 30 links - not including all the others in this page, and i easily get over 100 and even sometimes 200 is this a big issue? does it damages my SEO? If yes, is there a way to write the HTML in a way that internal links like that wont be counted? Thank you SEOWiseUs
On-Page Optimization | | iivgi0 -
On-page SEO reviews
Hi everyone 🙂 I was hoping someone could point me in a direction on where I could get my website's on-page SEO a review with recommendations. Title tags, meta description, code, H1s, content and so on... I know you can find most of what I'm asking online, but I would like a professional with a new set of eyes to help out. Thanks in advance for your time in helping!
On-Page Optimization | | AutoGlassRescue0 -
Can you 301 redirect to a page that has other pages 301 to it?
Two years ago updated url page to include better keywords and used a 301 redirect from the old page to the new. so www.example.com/keyword-1st-generation.html now points to ... www.example.com/keyword-2nd-generation.html That moved the pages up in ranking, but now have better kw for the url, so is it okay to redirect the /keyword-2nd-geration-html to www.example.com/keyword-3rd-generation.html And what is a good length of time before removing the 1st-generation url? It's been 3 years and there is no chance of using it again. Plus, no sign of it in analytics.
On-Page Optimization | | AllIsWell0 -
Duplicate Page Title
Not sure how to fix this. I am getting a duplicate page title for my main url, and the index page. I have attached an image. Thanks. 0RnG6.jpg
On-Page Optimization | | pixel830