Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We have new website abexpress.ae. The website is not ranking in the first page when I search with the Domain name itself. Any tips to improve the SEO for a new website?
We have launched a website abexpress.ae. The website is not ranking well even if we search with the Domain name. The Domain authority of the website it 10. How to increase this?
On-Page Optimization | | AlliedTransport0 -
Should I optimize my home-page or a sub-page for my most important keyword
Quick question: When choosing the most important keyword set that I would like to rank for, would I be better off optimizing my homepage, or a sub page for this keyword. My thinking goes as follows: The homepage (IE www.mysite.com) naturally has more backlinks and thus a better Google Page Rank. However, there are certain things I could do to a subpage (IE www.mysite.com/green-widgets-los-angeles ) that I wouldn't want to do to the homepage, which might be more "optimal" overall. Option C, I suppose, would be to optimize both the homepage, and a single sub-page, which is seeming like a pretty good solution, but I have been told that having multiple pages optimized for the same keywords might "confuse" search engines. Would love any insight on this!
On-Page Optimization | | Jacob_A2 -
Unique Pages with Thin Content vs. One Page with Lots of Content
Is there anyone who can give me a definitive answer on which of the following situations is preferable from an SEO standpoint for the services section of a website? 1. Many unique and targeted service pages with the primary keyword in the URL, Title tag and H1 - but with the tradeoff of having thin content on the page (i.e. 100 words of content or less). 2. One large service page listing all services in the content. Primary keyword for URL, title tag and H1 would be something like "(company name) services" and each service would be in the H2 title. In this case, there is lots of content on the page. Yes, the ideal situation would be to beef up content for each unique pages, but we have found that this isn't always an option based on the amount of time a client has dedicated to a project.
On-Page Optimization | | RCDesign741 -
Moz crawl
Hi, 1. Is there any way to get the moz crawl to analyse any site like the website auditor tool in seo powersuite on command. It would be good to see what keyword my competitors are using. would also like to analyse my site when i have corrected errors on command. 2. Is google adwords keyword tool still the best keyword research tool? I Don't see anything in the newbie guides to the moz pro tool regarding keyword research. Thanks! David.
On-Page Optimization | | WallerD0 -
Page Speed
Google recommends a page load speed of 1.4 seconds, is it recommended to have that page speed for every page on the site, or just the landing pages. Is there a tool that will check the load speed of every page on a site and report the slow pages? The free online tools only check one page at a time.
On-Page Optimization | | Bryan_Loconto0 -
One Page Website vs. Multipage Site, if you want to target one specific Keyword only.
Hello! suppose I want to start a website about, let's say spray adhesives. My aim is to rank on the first page for the keyword "spray adhesive". I don't care about my ranking on more specific keywords like "Tesa spray adhesive" or "3M spray adhesive". My ranking for more general keywords like "glue" is unimportant, too. So I thought about creating a single-page website, that writes about spray adhesives, the pros & cons of every manufacturer, and shows the best discounts for spray adhesives. Each section can be accessed through a top-navigation, that links via anchors to the individual sections. The page will be updated every day On the other hand, i could create a blog and write an article for every specific spray adhesive. So I would have a home page that lists the latest articles for every product, with titles like "3M spray adhesive CreativeMount", "3M spray adhesive SprayMount", "Tesa Spray adhesive" ... I will write one article every day What do you think would be the better strategy? Is there a risk to create competing articles for the keyword "spray adhesive" and thus rank lower if I go with the blog strategy? On the other hand, does google rate singe-page websites lower, because google thinks those websites are less valuable than websites with many pages for the same topic? Thank you ver much for you help in advance!
On-Page Optimization | | MGMT0 -
How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect... Obviously have it in robots.txt but is their any other form of action that should be done?
On-Page Optimization | | Romancing0 -
Https and secure pages
Hello, I have a facebook badge in my footer. Is it okay if I make the code call on https. It makes the page secure for IE. I have also have done this for images. These secure urls are also being called on non secure pages. But I don't think that matters does it? Code below. Thanks Tyler
On-Page Optimization | | tylerfraser0