Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Renamed a page and created a 301, page lost its rankings.
We changed a page name to fall under the root of our site from domain.com/page1/page301d/ to domain.com/page301d/ and after 2 weeks it still is not back to its #3 position. Now it is on the bottom of page 3. I cant figure out what im doing wrong here. The original .com/page1/ that this page fell under was removed totally and redirected to antoher page that was more relevant. I went ahead and re-enabled this page and its contnent, because the page was linking out to the page we 301d. This page we re-enabled had about 150 links poitning to it and therefore i was thinking that maybe the link juice from this page (or relevancy) via an internal link was helping it rank. This was updated about 6 days ago and the internal link is back Any other ideas why this might not be working. Ive checked all the 301s, content has not changed on the page. We have updated the strcuture for many pages. Instead of having the pages in question fall under anotehr page, they all fall under the root and its sub content is now only 2 levels deep , instead of being 3. hope that makese sense.
On-Page Optimization | | waqid0 -
Category Page Content
Hey Mozzers, I've recently been doing a content audit on the category and sub-category pages on our site. The old pages had the following "profile" Above The Fold
On-Page Optimization | | ATP
Page Heading
Image Links to Categories / Products
Below the Fold
The rest of the Image Links to Categories / Products
600 words+ of content duplicated from articles, sub categories and products My criticisms of the page were
1. No content (text) above the fold
2. Page content was mostly duplicated content
3. No keyword structure, many pages competed for the same keywords and often unwanted pages outranked the desired page for the keyword. I cleaned this up to the following structure Above The Fold
H1 Page Heading 80-200 Word of Content (Including a link to supporting article)
H2 Page Heading (Expansion or variance of the H1 making sure relevant) 80-200 150 Words of Content
Image Links to Categories / Products
Below the Fold
The rest of the Image Links to Categories / Products The new pages are now all unique content, targeted towards 1-2 themed keywords. I have a few worries I was hoping you could address. 1. The new pages are only 180-300 words of text, simply because that is all that is needed to describe that category and provide some supporting information. the pages previously contained 600 words. Should I be looking to get more content on these pages?
2. If i do need more content, It wont fit "above the fold" without pushing the products and sub categories below the fold, which isn't ideal. Should I be putting it there anyway or should I insert additional text below the products and below the fold or would this just be a waste.
3. Keyword Structure. I have designed each page to target a selction of keywords, for example.
a) The main widget pages targets all general "widget" terms and provides supporting infromation
b) The sub-category blue widget page targets anything related and terms such as "Navy Widgets" because navy widgets are a type of blue widget etc"
Is this keyword structure over-optimised or exactly what I should be doing. I dont want to spread content to thin by being over selective in my categories Any other critisms or comment welcome0 -
Is it better to have an hreflang go to the home page in a different language if there's no corresponding page
If I have some pages in English, but not in Spanish on my website: Should my hreflang go to the home page on the Spanish site? Or should I not have an "es-MX" hreflang for that page? Ideally I would have all the pages translated, but this has not all been done yet.
On-Page Optimization | | RoxBrock0 -
Spammy page titles
Over the last couple of weeks, I have noticed that Google aren't showing the page titles for my online shop anymore. They're set up with a third party plug-in piece of software, and while it's an old version of the software, the developer said it wouldn't be causing issues. They have suggested that I re-write my page titles to be less spammy. The thing is, Google haven't attacked just spammy looking titles, they're just taking a swoop through my whole site and not showing any of my page titles in their search results. I'm getting "Category Name - Shop Name" showing. Here's some of the page titles no longer appearing and I honestly have no idea how to rewrite these to not be spammy. Are there any good articles on what's spammy and what isn't? "Coconut oil - best tasting in Australia. Buy online from <my business="" name="">"</my> "Discount Vitamix Blender. Best deal in Australia. Buy online from <my business="" name="">."</my> "Natural & Organic skin care for the face | buy online in Australia from <my business="" name="">."</my> There are others that are showing the real page titles, but I think it's only a matter of re-indexing before they're all not showing. Any clue?
On-Page Optimization | | sparrowdog0 -
Home page or landing page?
Hello, I want to ask a question related to that - Should we put keywords in the home page title if we wish to position another landing page better for particular keywords? I have read in one website about SEO that it's good the main keywords of your website to be positioned in homepage title also. f.e. Let's say we have website about web-design and our company is named Company Ltd. The title of the home page is "Company Ltd. - Web design, SEO, etc" We have also another inner page named "Web design | Company Ltd.". So should we leave the first page name only "Company Ltd." and the landing page's name "Web design | Company Ltd." . I don't know if they both have the same keyword in their title they won't compete with each other.
On-Page Optimization | | HrishikeshKarov0 -
Autogenerated pages
My main product is database conversion software. As it supports tons of databases, it's fairly easy to generate thousands of landing pages simply by variating source/target database names, connection information etc. In fact, I autogenerated almost 25k pages that way. As I didn't want to jeopardize my main site, I placed all that content to a new microsite (www.fullconvert.com) which had no history and no inbound links. Results were nice - site is live two months and in second month already had 1300 visitors. Now, my question is - should I create the same thing on my (old and rather authoritative) main site www.spectralcore.com? I could use a different template to avoid duplicate content. Of course, my main concern is being penalized by Google. In my opinion, this autogenerated content is fine because it provides (tons of) laser-focused landing pages, so visitors will instantly recognize they found what they're looking for. But Google might disagree! What do you think? Is there a danger in trying to leverage authority of my main site in adding 20k+ autogenerated pages with inbound no links to them?
On-Page Optimization | | metadata0 -
On-Page Report Card
How long will it take to see the results of changes made on my pages based on the On-Page Report Card recommendations? On-Page Report Card
On-Page Optimization | | sansonj0 -
Authority of a page
What factors contribute towards the authority of a page ? No. of links to a page ?
On-Page Optimization | | seoug_20050