Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I utilize URL re-writes to include keywords and other optimised page elements on my website?
Hi there, I am working on a medical recruitment website: https://wave.com.au/ I have noticed that our Job Search and Job Listings section of the website is lacking a little bit in terms of SEO optimisation. For example, at the moment this is our search page (with locum work type selected): https://wave.com.au/search-locum/results If you add a location, for example, NSW and then click search again, it updates the URL to: https://wave.com.au/search-locum/results?LocumSearchForm_Location[]=NSW&action_doSearch=Search+jobs I did a check what some competitors and leaders in the recruitment industry were doing and I came across: https://www.ochrerecruitment.com.au/jobs/anaesthetics/new-south-wales/locum/ If you click the different filters/options, it updates the URL to include more clearly defined categories. Some websites would even have a H1 heading tag that would update based on the filters/options you selected. Should I set up a set of URL re-writes and re-structure my website a little bit so that dynamic URLs change to static etc.? Does anyone have any best practice knowledge in regards to this? I have been referencing the following article: https://moz.com/blog/dynamic-urls-vs-static-urls-the-best-practice-for-seo-is-still-clear
On-Page Optimization | | Wavelength_International0 -
What's better for SEO a page per review or a page with all reviews?
Was wondering what's better for SEO. We have a platform where consumers can read and write reviews. But the question is: is it better to give one page per company with all the reviews on it? Or should we have different pages for the specific company? Example: Itunes has a company page with all reviews on the page, but not the whole review. You have to click further to view the whole review (new page), at the moment this the current situation. What if we place the whole reviews on the company page, so you don't have specific pages for the reviews? Hopefully can someone help us out. Contact me if it's not clear or you want more extended information. Kind regards
On-Page Optimization | | MozzieJr0 -
What am i missing?
I compare my competitors and i am always higher than others (Domain authority, Page authority, Page mozrank, page moztrust, domain mozrank, domain moztrust). But some queries i am under these sites. What is the problem or how can i beat them? P.S. : I am the newest website among them. (E-commerce and my domain register date: january 2013)
On-Page Optimization | | hepsimedikal0 -
Why is this page not ranking?
Can you please tell me why this page is not ranking. http://goo.gl/BqoRT The page doesn't rank at all for keywords but even if I copy a line or 2 of text it still doesn't rank for that text. Any help will be much appreciated.
On-Page Optimization | | JillB20130 -
800 number on website
Hi, My client just sent me an 800 number that he would like to use to replace his number on the website. I know that it is best to keep a local phone number on the website and across all citations for NAP/Local SEO reasons. Is there anywhere that I could still incorporate the 800 number and not have it affect SEO? Thanks, Erin
On-Page Optimization | | HiddenPeak0 -
Duplicated Page Content
I have encountered this weird problem about duplicate page content. My site got 3 duplicate content similar on the link structure below. If I'm going to use rel canonical does it help to resolve the duplication problem? Thanks http://www.sample.com http://www.sample.com/ http://www.sample.com/index.php
On-Page Optimization | | mattvectorbpo0 -
Duplicate Page Title
Not sure how to fix this. I am getting a duplicate page title for my main url, and the index page. I have attached an image. Thanks. 0RnG6.jpg
On-Page Optimization | | pixel830 -
Duplicate page titles
We have search results pages on our site. They share the same page title, there is no real differentiator between the result pages, other than page 1, page 2 etc. How do we de-dup the titles? just add page 1/2/3 etc to the end of them?
On-Page Optimization | | lilibooz0