Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Understanding why our new page doesn't rank. Internal link structure to blame? + understand canonical pages more.
Hi guys. Sorry it's an essay...BUT, i think a lot of you will find this an interesting question. This question is in 2 (related) parts, and I imagine it would be an 'advanced' SEO question. Hoping you guys can help bring some real insight 🙂 Always amazed at the quality for this forum/ community. **Context... ** We had a duplicate content issue caused by this page and it's product permutations, so we placed canonical tags on all the product permutations to solve it. Worked a treat. However, we now have more **product ranges. **We now sell Diaries, Notebooks & Music books, which are clearly different from one another. So...we've placed canonical tags on all the product permutations leading back to the 'parent' theme. In other words, all the diary permutations 'lead back' to the diary page. All the notebooks permutations 'lead back' to the main notebook page. So on and so forth. Make sense so far? Context end..... Issue. Amazingly our Diary page outranks our notebook pagefor the search term 'Design your own Notebook'. The notebook page is well optimised for this search term, and the diary page avoids the word 'notebook' altogether (so no keyword cannibalisation going on). Possible reason? Our Diary page has a vast amount of internal links to it throughout our site. The notebook page has only a few. Could this be the issue? If so, what reading/ blogs/ content/ tools would you recommend to help understand and solve this problem? i.e) Better understanding internal link structure for SEO. 2nd part of the question (in the context of internal linking for SEO). When there are internal links to a page with a conical tag does that 'count' towards the 'parent page', or simply towards that specific page? I really hope that makes sense. If it's clear as mud just shout. Isaac. EDIT: All pages in question have been indexed since we added these changes to the site.
On-Page Optimization | | isaac6630 -
Over-Optimized Website
I'm looking for advice for what you would start with if you were working on a website that was extremely over-optimized for 1 keyword. So, for example, I'm going to pretend this client is a dog trainer in Toronto (I can't publicly post the URL). I've read places that having exact-match anchor text links to inner pages in the footer of the site can cause problems and removing it has resulted in big ranking jumps. I'm looking to see if there are other big items that you would tackle first if this was your client. Some examples of things the site has: There is a page for dog training under their Services menu. However, internal links on their site link "dog training" to both the homepage and to this service page. Is that going to cause issues? The anchor text for internal linking is almost always the exact same word - "Dog Training". There is a banner that goes across the top of the site that appears on every page that says "Dog Training Toronto". I'm guessing I should remove that. Would the same keyword being overly used on every page cause confusion? Almost every image on his site is saved in the format "Dog Training Toronto". I'm looking to see if anyone has general tips on where to start with a site that has been over-optimized for 1 keyword. He actually has a ton of good content on his blog that gets a ton of traffic (because it's actually useful) so it's not that his content sucks - it's just been overly structured and SEO'd to death. I found a few articles on this but other than the footer advice I didn't find too many case studies of others that have run into this issue and done a few steps that actually worked.
On-Page Optimization | | ImprezzioMarketing0 -
Why will a Page not rank or improve on a website?
My website is performing well, but I have a couple of popular Brand pages that no matter what I do will not rank organically, they are being indexed, but will just not perform or improve. The rest of the pages throughout the website all seem to be fine, can anyone advise if they have experienced a similar problem or advise why this would happen or even suggest what I should do to fix the problem, we have even gone as far to create a new page and delete the old one, and it still hasn't worked. I have used various tools to test the page and variations of search terms and although it is appearing in Google it is not ranked, although it is ranking and visible in Yahoo and Bing. Any help and advise would be appreciated. Jacqui
On-Page Optimization | | EurekaSolutions0 -
Changing a page url
I have a page that ranks well (#4) for a good keyword. However, the url has the keyword in it but is misspelled. I would like to change the url to have the correct spelling but do not want to lose the ranking that I have. What is the best and safest way to proceed?
On-Page Optimization | | bhsiao0 -
On-page SEO optimization
hi there! Is it possible not to be in the first 20 or 30 positions in the SERPs after executing onpage SEO actions (keyword optimization, metatags, ....) even for keywords for which there's not "too much" competition? Is there a way of visualize the pages indexed by the google bot? (the pages especifically, not the number) in order to discard indexing problems? Thank you!
On-Page Optimization | | juanmiguelcr1 -
Moving Top rank Page urls off my Home page and nesting them on one page? Good idea?
I am basically trying to cut down the amount of links on my home page to make it less eye boggling and move stuff around. So i have of my Urls on my home page that lead to pages that rank very well within google. My questions is can i remove those urls to a separate page to group them together and then showcase that one link to that page on my home page. Is that a good idea or i am going to loose my link juice and position in search? The physical urls on those pages wont change at all.
On-Page Optimization | | Dante130 -
Home Page
We are re-design our home page, one are of the current home page has a drop down window called "popular products" . We wrote short articles for our keywords and have them linked to product page. In the past, it has helped us rank. However, with new Google rules, our feeling is that such practice is no good. So, we lean towards to remove it. Still, we'd like to hear some opinions and ask some questions too: www.butterflycraze.com is it clear to you that this is not good in Google's eyes? how do I determine if these links serving any SEO purpose now after Panda? depend on the answer to 2), what should we do about these pages? shall be re-direct or shall we remove them from Google index?
On-Page Optimization | | ypl0 -
Page Authority
I have recently optimised a set of images for a client of ours: I'm looking through all the PA of these newly optimised images, and have varying PA {from SEOmoz toolbar} I understand that internal linking will pass link juice, and obviously external links will add to the overall PA. I have several pages with a PA of 36: { Fairly deep pages} Yet they have no external or internal links going to them. My question is "How can a page gain any authority when it has no visible links pointing at it?" Obviously there must be a link pointing at it {internally} as Google wouldn't have crawled the page right? Also lets say all the keywords are of equal competitiveness would the keywords with highest PA rank higher than those on O PA pages. Many Thanks
On-Page Optimization | | Yozzer0