Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Finacial pages markup
I was wondering if there is away to use Schema for stock market pages?
On-Page Optimization | | ciznerguy0 -
Does the title tag on the home page affect sub-pages?
Hello. I am thinking of changing our home page title tag to include our two most valuable keywords from two of our sub-pages. Would this help the rankings of those two sub-pages? Thank you!
On-Page Optimization | | nyc-seo0 -
Duplicate Page Title
I have a dating site, it's got a lot of duplicate page titles, most of them are the language buttons for the users to view the site in there language. but I think it's obvious that the buttons don't have anything to do with it. I'm thinking that page tittle is basically a description of what the site is. like for an example "online-dating" is this it? please tell me in terms for a dummy, how to fix it.
On-Page Optimization | | clickit2getwithit0 -
How to update a crawl
Hi I crawled my site, it found lots of errors and warning which i have fixed, I told seomoz to recrawl, and it says nothing has changed... It defiantly has though! How do I make it see these changes?
On-Page Optimization | | chrisleonard0 -
Why doe SEOmoz include my categories and subcategories when counting on page links?
Hi everyone, I am new to SEOMOZ, but it looks like really good program. I ran a report last week, and SEOMOZ says that out of 10001 pages, I hve a total of 9995 warnings. When I take a look at the breakdown, it says I have too many on page links. I removed the links from the bottom of each page, but my on page links are still over 100. Does this mean that I will have to get rid of categories and sub-categories in the navigation bar? Thanks, Don
On-Page Optimization | | ge01734000 -
Website tables
Hi guys , my website www.starpluservices.com on google first page for the past 2 years and we were page rank 4, 2 months ago we changed all page titles and Keewords after that the page rank droped to 1 and we are not anymore in page 1 in google, we have done all this changes to target another keyword Office Cleaning London, now after 2 months I had 3 quotes for SEO , and 2 SEO companies told me that I need a new website because my one was done with tables and the other company told me that if I still on first page with some keywords andon the 2 page with cleaning companies to dont make a new website just update my one!! Could anyone let me know what should I do? Regards Sergio
On-Page Optimization | | starplus0 -
Page Rank Drop
Just trying to get more feedback - so we recently edited title and meta descriptions for existing website SEO and we've noticed in the past several weeks, our client's website has dropped out of the top 50 in a variety of terms we were targeting that they used to show up for (note: when updating SEO, we DID NOT remove any relevant terms we were targeting). When the website does come up in searches, it is the old meta description and title. So far, the feedback we've gotten is that first, it takes Google a few weeks to recrawl and index - however, we are now on week 3 after the changes and still no rebound in rankings. We were also told to check with the SEO Moz page grader to be sure the keywords were being optimized correctly - got As and Bs for the test terms I tried. We also submitted an XML site map to speed up the crawl process as another user suggested. We've tested the site with various tools to make sure there are redirect errors, etc. and everything looks fine. Again, it's now been 3+ weeks and no ranking rebound. Any other suggestions on what could be happening?
On-Page Optimization | | laidlawseo0 -
Why is this page ranking highest?
I've just used Open Site Explorer to compare some sites whose (unpaid) Google ranking I aspire to. They all have higher authority than my site, but the top ranking site out of the 3 I've looked at has the lowest Page Authority, hardly and links (when the others have hundreds), lowest page rank and lowest page trust. In fact, when you look at the top ranking page (ranks #1), it does not even have the search term in it as a complete phrase. One thing I do notice is that it does have 100,000s of linking root domains from one linking root domain. So how can it rank number one on Google?
On-Page Optimization | | Beemer0