Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
-
Situation:
I have a dual domain site:
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages.I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option.
My Question:
Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2?
I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages? -
Hello!
I can answer this from a Google / SEO perspective (a non-moz tool perspective).
First you want to be sure the secure subdomain content is not indexed.
-
If the secure subdomain is NOT indexed, leave the robotos.txt crawl blocking in place. You don't want and don't need Google crawling secure pages and payment pages. Just be sure they truly all are private pages. If they are NOT indxed, the crawl block is best - this will prevent google from crawling, and if they can't crawl they can't index.
-
If the secure pages ARE indexed
-
remove the robots.txt crawl block.
-
Add meta noindex on all the pages
-
Wait for them to be noindexed (removed from google)
-
Then, block them from being crawled with robots.txt - which will prevent them from being crawled, and thus prevent them from being indexed as well.
-
-
Hey, Dave here from the Help Team!
Jumping in to answer the technical question, you can definitely use the meta robots tag instead of a disallow directive in your robots.txt file. I would like to point out that Meta Noindex is something we report in Site Crawl so you would see an influx in that issue category but you can mark them as "ignored" as you see fit.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Domain Authority hasn't recovered since August
I really need some major advice on this one. Back in September, I asked a question on here as follows: "A client wanted to change their domain name, which we have now done. The site content itself is exactly the same. We put 301 redirect links in so that Google searchers would redirect from the old site to the new one. However Moz then said that it couldn't crawl the old domain because of the redirects and advised creating a brand new campaign for the new domain. We have done this but now Moz says that the domain authority of the new site is 2 (it was 14 on the old domain)." My original question and the answers I got are here: https://moz.com/community/q/new-domain-wipes-out-domain-authority). Generally the responses I got were that we should give Moz time to crawl the new domain and process all the "new" pages. It is now February, ie 6 months after the domain rename, and on Moz the site still has a DA of 2. It seems like 6 months is enough time to wait. We checked all the recommended guides and believe we have done it all correctly. I really don't know what to do now. Can anyone help or have a quick look and work out why this is so bad? Specifics are:
Getting Started | | mfrgolfgti
old domain: https://ryemeadcleaning.co.uk
new domain: https://ryemeadgroup.co.uk0 -
What is PA and DA in SEO and how to improve it?
I want to understand what is PageAuthority and Domain Authority and how we scale for SEO? Suggestions highly appreciated
Getting Started | | SathishFirecompass0 -
Moz Site Crawl can't index WIX sites
We've been attempting to work on some SEO for a new potential client however they are using a WIX site. We've noticed that Moz SEO tools will not index any WIX sites. e.g. https://www.sharonradisch.com/ (which is one of their case studies). Anyone seen this that can offer any advice? Thanks,
Getting Started | | monkeex
Mark2 -
Can someone help me to gain Moz trust and domain authority?
Hi someone help me how to gain moz trust and domain authority for my web site. here is my web addresss: www.bassinotary.com/ please tell me how i can improve ranking for my site. please help. thanks.
Getting Started | | grbassi0 -
High total links, but very few root domains?
Hi Moz community!I've just joined and am getting to grips with SEO basics. Right now, I'm looking at the Competitive Link Metrics in Moz Pro, and I'm curious about the following- Of the three competitors that we're following, I'm trying to figure out some differences between two of them - we'll call them A and B. 'A' has 3.6k external followed and total links, with 5 total linking root domains. 'B' (a more prestigious and established company with a much higher DA) has 2.2k total external links, with 180 root domains. So my question is, how can A have nearly 1,000 more links, but only from 5 domains? Any feedback much appreciated! Thanks!
Getting Started | | thegildedteapot0 -
Duplicate Content after Moz Site Audit
Hello folks, So I signed up for the trial version of the Moz tool and ran an initial site audit. One of the site audit results is confusing me.
Getting Started | | jjimen03
It reports that there are two pages with duplicate content ( Each page has a duplicate page with duplicate content in it).
When I take a look at what those pages are, here is what I see: mysite.com/Contact-Us.html
mysite.com/contact-us.html
( The difference in the above is the Contact and Us, the first letters are capitalized on one of the URLS) mysite.com/index.html
mysite.com Now I am confused because for one thing, I don't have 2 Contact Us html files uploaded on my hosting server.
Why is Moz seeing 2 Contact Us pages? How to remove one? Regarding my home page, why is it flagging the same page as two different pages? How to remove of them?0 -
SEO-Off Page
Hi, I am Harika. Past few months we are working on Off-Page but there were no changes in the keyword.So anyone please help to overcome this problem.Following all the strategies off On-Page but we are not getting proper results.
Getting Started | | Harika0