Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
-
Situation:
I have a dual domain site:
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages.I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option.
My Question:
Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2?
I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages? -
Hello!
I can answer this from a Google / SEO perspective (a non-moz tool perspective).
First you want to be sure the secure subdomain content is not indexed.
-
If the secure subdomain is NOT indexed, leave the robotos.txt crawl blocking in place. You don't want and don't need Google crawling secure pages and payment pages. Just be sure they truly all are private pages. If they are NOT indxed, the crawl block is best - this will prevent google from crawling, and if they can't crawl they can't index.
-
If the secure pages ARE indexed
-
remove the robots.txt crawl block.
-
Add meta noindex on all the pages
-
Wait for them to be noindexed (removed from google)
-
Then, block them from being crawled with robots.txt - which will prevent them from being crawled, and thus prevent them from being indexed as well.
-
-
Hey, Dave here from the Help Team!
Jumping in to answer the technical question, you can definitely use the meta robots tag instead of a disallow directive in your robots.txt file. I would like to point out that Meta Noindex is something we report in Site Crawl so you would see an influx in that issue category but you can mark them as "ignored" as you see fit.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz site crawl doesn't work
The Moz site crawl isn't working for my campaign, but works for the site's on demand crawl. The search should not be disallowed by robots.txt or the headers. I'd like to be able to track the website for the campaign so I can see SEO gains / losses and increases / decreases in indexing.
Getting Started | | DrainKing0 -
Crawler Accessibilit
In Insights section of MOZ campaign, I'm seeing this: https://imgur.com/Gu2K9dz Here are the contents of robots.txt: User-agent: *
Getting Started | | Avatardesk1
Disallow: /wp-admin/ Sitemap: http://website.com.com/sitemap_index.xml Can you please let me know what is wrong here? Gu2K9dz1 -
Site Crawl - Crawls only homepage?
Hi Moz Comunity! Joined Moz just 2 weeks ago and slowly trying to get used to tools available in here! Great tools and info available on this site! My concern is that Site Crawl of Moz in my Campaign seems to have crawled only my homepage and no other sub-domains, is there any reason for this? FOr some reason it seems that Moz interacts only with my homepage? Even when I tried the Keyword Exlporer set on Keyword to see if any of my pages rank for any keywords, it seems only my homepage was ranking for a few keywords. It's possible my other sub-domains don't rank for any keywords yet but still, seems suspicious... I have added a link to Site Crawl that says it has crawled only 3 pages on my site, and all are my homepage... Thanks for any help! Jacob s!AlxV7sobbcgmhJB_fXcF4EPzbPSovA
Getting Started | | Shotlife_Studio0 -
What is 'domain authority'?
IN seomoz, it mentions domain authority but it doesnt define it. what does domain authority mean?
Getting Started | | torbett0 -
How do I identify NEW linking root domains?
Under moz analytics there are metrics for Linking Root Domains. It shows an increase of n%. Is there any easy way of identify which domains are new domains for any given period?
Getting Started | | Lucien_Taylor0 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
How long does it usually take Moz to populate information for a new Web site?
We recently launched (9/13/2013) an e-commerce Website and added the campaign to SEO MOZ. Week after week the Domain Rank is 1 and none of our keyword stats or link stats are populated. We have another Moz campaign that posts weekly updates and is doing extremely well. I'm just wondering how long it usually takes Moz to start populating all the analysis stats? I'm also wondering if there might be a campaign setting buried somewhere that I need to enable or maybe it just takes more than 5 weeks? Any insights would be much appreciated. Here's the new URL we need to track with MOZ: http://www.imsportshq.com
Getting Started | | Tripper0