Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question about my website ranking
Hi everyone I have sports website. My website DA, DR is good. I have created good quality backlinks. But still my website ranking is not improving. Can you guide me how to improve my website ranking. Thanks in advance!
On-Page Optimization | | fktos.tam.906n0 -
Should I change our main category pages to product listing pages?
With the thought of improving user experience, as well as rankings in Google, I'm considering changing our main category pages to product listing pages (with sub-categories remaining, still). These main category pages are very standard and don't link to any informational content, such as buyers guides, etc. What's driven this is the latest Google core update. I've noticed our main competitor (who we were out-ranking before... but not now) now uses this approach. I can see the benefit from a user perspective, i.e. less clicks to reach products. What's the pros/cons from an SEO point of view, please? Could the potential duplication of content be an issue? For context, we have about 2,000 products and website is on Magento 2.
On-Page Optimization | | alifeofjoy1 -
Combine poorly ranking pages into a single page?
I'm doing on-page optimizations for an apartment management company, and they have about seven apartments listed on their site. Rather than include everything on the same page - /apartments/apartment-name/ - they have the following setup: /apartments/apartment-name/contact /apartments/apartment-name/features /apartments/apartment-name/availability /apartments/apartment-name/gallery /apartments/apartment-name/neighborhood With very few exceptions, none of these pages appear to rank for anything, and those that do either rank very poorly for seemingly random keywords or for keywords like the apartment complex name (alongside the main landing page for the complex). I'm of the mind to recommend combining the pages into a single one that contains all the info, eliminates the chances for duplicate content (all of the neighborhood pages contain the same content verbatim), and prevents keyword cannibalization. Thoughts? Thanks.
On-Page Optimization | | Alces1 -
Duplication in landing page
This is driving me mad, I have a site that for some reason google and moz pick up the landing page as a duplicate. They see "mysite/" and "mysite/index.html" as two different pages and giving me warnings for duplication. I have no 301 included at this time and I am using foundation as the base. This is occurring both on a localhost test bed and live....... anyone got an idea how to correct.
On-Page Optimization | | AndyBirtles0 -
Duplicate Home Page
Hi, I have a question around best practise on duplicate home pages. The /index.aspx page is showing up as a top referrer in my analytics. I have the rel=canonical tag implemented for the www.mysite.com on both pages. Do I need to 301 the /index.aspx to the mysite.com? I have a lot of links pointing to the /index.aspx (half of those are coming from the mysite.com). www.mysite.com/index.aspx www.mysite.com Many thanks Jon
On-Page Optimization | | JonRaubenheimer0 -
Home page or landing page?
Hello, I want to ask a question related to that - Should we put keywords in the home page title if we wish to position another landing page better for particular keywords? I have read in one website about SEO that it's good the main keywords of your website to be positioned in homepage title also. f.e. Let's say we have website about web-design and our company is named Company Ltd. The title of the home page is "Company Ltd. - Web design, SEO, etc" We have also another inner page named "Web design | Company Ltd.". So should we leave the first page name only "Company Ltd." and the landing page's name "Web design | Company Ltd." . I don't know if they both have the same keyword in their title they won't compete with each other.
On-Page Optimization | | HrishikeshKarov0 -
On-Page Analysis Question
Hi, I have a question about the On-Page Analysis report. I am tracking two different keywords for our campaign: "Private Dining" and "Private Dining Sacramento". We are ranked 8th for Private Dining Sacramento but we have an On-Page analysis rating of F. While on the other hand we are not ranked in the top 50 for Private Dining but have an A on-page report. When looking at the on-page report it makes sense that we have an F for Private Dining Sacramento as we don't use that keyword anywhere on the page. We only use Private Dining. However, we are still ranked for Private Dining Sacramento and not for Private Dining. Should we update our keywords/text to use the Private Dining Sacramento keyword instead of the Private Dining? If we add Sacramento will we also get credit for Private Dining because it will still be part of all H,P and A tags we use? Sampe Report | Keyword | Grade | Google US |
On-Page Optimization | | Three29
| URL | Current | Change | Rank | Change |
| | Private Dining /private-dining | A | No-change-icon | no data |
| | Private Dining Sacramento /private-dining | F | No-change-icon | 8 | No-change-icon |0 -
Authority of a page
What factors contribute towards the authority of a page ? No. of links to a page ?
On-Page Optimization | | seoug_20050