Can Google see all the pages that an seomoz crawl picks up?
-
Hi there
My client's site is showing around 90 pages indexed in Google. The seomoz crawl is returning 1934 pages.
Many of the pages in the crawl are duplicates, but there are also pages which are behind the user login.
Is it theoretically correct to say that if a seomoz crawl finds all the pages, then Google has the potential to as well, even if they choose not to index?
Or would Google not see the pages behind the login? And how come seomoz can see the pages?
Many thanks in anticipation!
Wendy
-
Well, that could be your easy solution. Make sure they're all set not to be indexed, then you'll be able to (mostly) ensure Google won't crawl them, and they'll probably disappear from your moz crawl report as well. As far has how moz is finding them to begin with behind your login wall, sorry, I have no idea.
-
The pages behind the login? No not yet - they are a new client, so I am just auditing at the moment to identify what we need to do
Many thanks for your replies!
-
This may be an obvious question, but to you have those pages set to noindex?
-
Hi Marisa
seomoz are crawling unecessary pages, (they return pages ignored by screaming frog for example)
BUT my concern is that if Google can also see them, even if they choose to ignore them my client maybe getting slammed for duplicate issues or the pages behind the login may suddenly appear in the index.
We'll get no index / no follow added, and fix the dupes, but am really interested as to how seomoz sees behind the login
-
Here's the real question: Do you WANT Google to see all these pages, or is SEOmoz crawling unnecessary pages?
-
Great, many thanks Nakul - they are a new client so am waiting on getting access to WMT - will go through with a fine tooth comb! Just seems really weird with regards to the pages behind the login ...
-
Wendy, if SEOMOZ can see it, I am sure Google can see it as well. I would login to your webmaster console and check the index status. Do you have an XML sitemap submitted for your website ? Once you do, you'll have a more accurate read on the number of pages you submitted and how many of them are indexed. The new index status Google introduced last month also lets you see pages Google ignored for multiple reasons.
I hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does google see the keyword "e cig" and "e-cig" as the same word? on MOZ it shows that they have a totally different amount of search quarries.
does google see the keyword "e cig" and "e-cig" as the same word? on MOZ it shows that they have a totally different amount of search quarries.
Moz Pro | | smokingllc0 -
Why can't I see last week's stats?
Can see stats week ending May 1st but nothing after that- and it's May 9th!?
Moz Pro | | locumhunter0 -
Functionality of SEOmoz crawl page reports
I am trying to find a way to ask SEOmoz staff to answer this question because I think it is a functionality question so I checked SEOmoz pro resources. I also have had no responses in the Forum too it either. So here it is again. Thanks much for your consideration! Is it possible to configure the SEOMoz Rogerbot error-finding bot (that make the crawl diagnostic reports) to obey the instructions in the individual page headers and http://client.com/robots.txt file? For example, there is a page at http://truthbook.com/quotes/index.cfm month=5&day=14&year=2007 that has – in the header -
Moz Pro | | jimmyzig
<meta name="robots" content="noindex"> </meta name="robots" content="noindex"> This page is themed Quote of the Day page and is duplicated twice intentionally at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2004 and also at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2010 but they all have <meta name="robots" content="noindex"> in them. So Google should not see them as duplicates right. Google does not in Webmaster Tools.</meta name="robots" content="noindex"> So it should not be counted 3 times? But it seems to be? How do we gen a report of the actual pages shown in the report as dups so we can check? We do not believe Google sees it as a duplicate page but Roger appears too. Similarly, one can use http://truthbook.com/contemplative_prayer/ , here also the http://truthbook.com/robots.txt tells Google to stay clear. Yet we are showing thousands of dup. page content errors when Google Webmaster tools as shown only a few hundred configured as described. Anyone? Jim0 -
20000 site errors and 10000 pages crawled.
I have recently built an e-commerce website for the company I work at. Its built on opencart. Say for example we have a chair for sale. The url will be: www.domain.com/best-offers/cool-chair Thats fine, seomoz is crawling them all fine and reporting any errors under them url great. On each product listing we have several options and zoom options (allows the user to zoom in to the image to get a more detailed look). When a different zoom type is selected it adds on to the url, so for example: www.domain.com/best-offers/cool-chair?zoom=1 and there are 3 different zoom types. So effectively its taking for urls as different when in fact they are all one url. and Seomoz has interpreted it this way, and crawled 10000 pages(it thinks exist because of this) and thrown up 20000 errors. Does anyone have any idea how to solve this?
Moz Pro | | CompleteOffice0 -
"Duplicate Page Title" and "Duplicate Page Content" issue
Hi I am having an issue with my site showing duplicate page title and content issues for www.domain.com and www.domain.com/ Is the trailing slash really an issue? Can someone help me with a mod_rewrite rule to sort this please? Thanks,
Moz Pro | | JoeBrewer
Joe0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0 -
How does moztrust compare with google page rank?
I have a trial Seomoz membership. I've been comparing my site to some of the competitor websites and Im confused about the moztrust vs google page rank. For instance some of my competitors have higher moztrust but my site has a higher google page rank. Isn't moztrust meant to duplicate google pagerank?? Why would these vary so much?
Moz Pro | | blac67890 -
Is there a way to see all SEOMoz questions, comments, posts, responses that I've given a "thumbs up" to?
Lots of times I'll Thumbs Up a question, blog post or response not only if I think it's high quality but if I think I'd like to revisit it in the future. Essentially, I want to use a the SEO Moz Thumbs Up as a bookmark. Is there any way to see all the SEO Moz content that I have given a "Thumbs Up" to? I know I can review all the questions I've asked or answered in the "My Q&A" page. Perhaps, I'm just missing a "Thumbs Up" page somewhere on my profile?
Moz Pro | | TaitLarson2