Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Renamed a page and created a 301, page lost its rankings.
We changed a page name to fall under the root of our site from domain.com/page1/page301d/ to domain.com/page301d/ and after 2 weeks it still is not back to its #3 position. Now it is on the bottom of page 3. I cant figure out what im doing wrong here. The original .com/page1/ that this page fell under was removed totally and redirected to antoher page that was more relevant. I went ahead and re-enabled this page and its contnent, because the page was linking out to the page we 301d. This page we re-enabled had about 150 links poitning to it and therefore i was thinking that maybe the link juice from this page (or relevancy) via an internal link was helping it rank. This was updated about 6 days ago and the internal link is back Any other ideas why this might not be working. Ive checked all the 301s, content has not changed on the page. We have updated the strcuture for many pages. Instead of having the pages in question fall under anotehr page, they all fall under the root and its sub content is now only 2 levels deep , instead of being 3. hope that makese sense.
On-Page Optimization | | waqid0 -
Is it better to try and boost an old page that ranks on page #5 or create a better new page
Hello Everyone, We have been looking into our placements recently and see that one of our blog posts shows on page #5 for a popular keyword phrase with a lot of search volume. Lets say the keyword is "couples fitness ideas" We show on page 5 for a post /couples-fitness-ideas-19-tips-and-expert-advice/ We want to try and get on the first page for that phrase and wanted to know if it is better if we did one of the following: 1. Create a new page with over 100 ideas with a few more thousands of words. with a new url (thinking /couples-fitness-ideas) 2. Create a new page with a new url (thinking /couples-fitness-ideas) with the same content as the currently ranking post. We would want to do this for more freedom with layout and design of the page rather than our current blog post template. Add more content, let's say 100 more ideas. Then forward the old URL to the new one with a 301 redirect. 3. Add more content to the existing post without changing the layout and change the URL. Look forward to your thoughts
On-Page Optimization | | MobileCause0 -
Category Page Content
Hey Mozzers, I've recently been doing a content audit on the category and sub-category pages on our site. The old pages had the following "profile" Above The Fold
On-Page Optimization | | ATP
Page Heading
Image Links to Categories / Products
Below the Fold
The rest of the Image Links to Categories / Products
600 words+ of content duplicated from articles, sub categories and products My criticisms of the page were
1. No content (text) above the fold
2. Page content was mostly duplicated content
3. No keyword structure, many pages competed for the same keywords and often unwanted pages outranked the desired page for the keyword. I cleaned this up to the following structure Above The Fold
H1 Page Heading 80-200 Word of Content (Including a link to supporting article)
H2 Page Heading (Expansion or variance of the H1 making sure relevant) 80-200 150 Words of Content
Image Links to Categories / Products
Below the Fold
The rest of the Image Links to Categories / Products The new pages are now all unique content, targeted towards 1-2 themed keywords. I have a few worries I was hoping you could address. 1. The new pages are only 180-300 words of text, simply because that is all that is needed to describe that category and provide some supporting information. the pages previously contained 600 words. Should I be looking to get more content on these pages?
2. If i do need more content, It wont fit "above the fold" without pushing the products and sub categories below the fold, which isn't ideal. Should I be putting it there anyway or should I insert additional text below the products and below the fold or would this just be a waste.
3. Keyword Structure. I have designed each page to target a selction of keywords, for example.
a) The main widget pages targets all general "widget" terms and provides supporting infromation
b) The sub-category blue widget page targets anything related and terms such as "Navy Widgets" because navy widgets are a type of blue widget etc"
Is this keyword structure over-optimised or exactly what I should be doing. I dont want to spread content to thin by being over selective in my categories Any other critisms or comment welcome0 -
On page links
Hi I am really intrigued by Bloomberg strategy. if you look at their article pages they are full with internal links done with what I assume to be an automated process (too many pages to be done manually). it seems to work for them. I would love to hear your opinions.
On-Page Optimization | | ciznerguy
http://www.bloomberg.com/news/2014-11-26/uber-said-close-to-raising-funding-at-up-to-40b-value.html0 -
Too many on page links
Hello, I have a page that isn't ranking very well. http://nicontrols.com/uk/drives-motors/variable-speed-drives According to the MOZ research tools I have too many on page links. I believe most of these are a result of the advanced filtration options on the left hand side of the page. I don't want to remove the filters as they are extremely useful for customers but I am also worried about the number of links Anyone get any ideas about the number of links? Should I care?
On-Page Optimization | | DavidLenehan0 -
International Website(s)
I have an e-commerce site currently selling in Canada and am now looking to branch it into the USA. We have both the .ca and .us domain names, but I am not sure what the best approach to this would be. If we put up a website on the .us domain name it would be virtually the same.. So I would assume we would have a duplicate content issue. What would be the best way to approach this type of situation?
On-Page Optimization | | wishmedia0 -
What is on page links?
Hi - i would like to know exactly what an on page link is? i understand the linking system however cant work what exactly what an on page link is? Thanks
On-Page Optimization | | OasisLandDevelopment0 -
Third party pages
Suppose you are using a third party tool such as an affiliate program. Typically, all the files are organized under one subdirectory. In addition, you may have little or no ability to modify any of the files in terms of SEO. Would you recommend hiding the entire subdirectory with a noindex? Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0