Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How my website is dropping?
Hello there guys! My website www.ntcbrasil.com.br is dropping comparing to my competitors. My main keyword is: geotextil
On-Page Optimization | | ffbrunoff
And my main competitor is: www.mantasbrasil.com.br We have better and original content. We have better structure website with better SEO practice (so a i thought). I'm not able to find out what's making the difference between my website dropping and my competitors raking first. This is in google Brazil www.google.com.br If anyone could give me any direction i would appreciate. Thanks a lot!! Bruno Nunes0 -
Is it OK to put a Blog Post and a Page within the same folder on a Wordpress hosted website?
Our education company website (hosted on wordpress) has evolved into having content on key topics distributed across both blog posts and pages. For example, "top-pharmaceutical-companies" lends itself to being published as a page. However other content "top-pharmaceutical-companies-usa-2016" lends itself to being published as a blog post as it's more temporal in nature. Now we'd like to establish topical domain relevance for the root keyword "pharmaceutical companies" and build a folder www./ourcompany/pharmaceutical-companies/ But when we look through our blog content, we notice we have "Blog Posts" that would be an excellent fit for certain folders within our "Page" url structure. So would it be OK to amend these blogs post urls addresses to place them within the folder structure of the pages.
On-Page Optimization | | GetReskilled0 -
Pages for similar keywords?
I have a site that wants to target the keywords listed below. They are a small company with just a few ski chalets in Val d'Isere, a ski resort in France. They don't have ski chalets in any other ski resort. Val d'Isere chalets
On-Page Optimization | | Marketing_Today
luxury ski chalets
luxury ski chalets Val d'Isere
catered chalet Val d'Isere
catered chalet val d isere
catered ski chalet val d'isere
catered ski chalets
chalets in Val d'isere
chalets in val d isere
luxury catered ski chalets
luxury ski chalets Their domain name includes "valdisere" but I can't get this site onto Page 1, it keeps lingering on Page 2 and 3. I wondered how you would approach this site with pages? Would you reply on the homepage to rank for all these terms or create seperate pages for the terms, and if so how would you group the terms per page?0 -
Duplicate Content on our own website
Our website sells tickets for events. We also have an news articles section with information about events / artists / venues. From time to time we release a product page and a related news article on a separate page. Some of the content in the news article would be perfect for our product page. Essentially its our product page we want too rank. Would it harm our SEO if we had some of the same content on both of these pages?
On-Page Optimization | | Alexogilvie0 -
Dynamic pages on a static html pages websiite
Good evening everybody. I am new on SeoMoz that I find very helpful in my work. I am not a web developer so please excuse me if my technical language is poor.I have an issue that maybe you can help me to solve. I work for a company who has a website, which is old and very well positioned on google for the most important keywords for the field of the company I work for (dentistry). It is a website made some years ago and it's made of static html pages.I would like to add a section in my website where I can post articles, like a blog (I think of a wordpress-style website) with daily posts (so with a dynamic page). Is there a way to do this without modifying the structure of the website and without losing pages, urls and ranking? We're on the first google page for many keywords of interests in our city and it would be a great damage for us to lose those positions.Thank you very much!
On-Page Optimization | | adec0 -
Web Page Refresh
Hi there, we redesign our Website, changing it for a jquery based version. This new design is much more usable and nice for our users, however the average page views for user decreased a lot. Basically this is due to the fact that once the user is logged in, it spends most of the time in the same Web form which is updated through jquery without refreshing it. We were thinking about adding a meta refresh tag, or ad some javascript for getting this task done in order to get the relation page views/visitor increased. Do you think refreshing the page every 4 minutes could be penalized by Google (or other Search engines) ? Which should be the interval between refresh ? Would it be better to make it very explicit (i.e. adding a meta refresh tag) or using a kind of hide javascript ? We want to increase the pageviews but of course, we don't want to get penalized
On-Page Optimization | | martincad0 -
SEOmoz crawl error
Hi, I'm getting a crawl error and it complains about there being missing meta description... But, the errors are all for non existent index files in directories that only contain pdf files and some thumbs of the front page... Just started trying to learn this stuff...! Cheers Rod
On-Page Optimization | | DrWho0 -
Page title
So if we have a main category page on our site (mines an ecommerce site), do we go for more than that main keyword phrase for that category of products, or is it better to just keep it by itself, and not utilize the 65-70 characters available?
On-Page Optimization | | azguy0