Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Referring domain issues
Our website (blahblah).org has 32 other domains pointing to it all from the same I.P address. These domains including the one in question, were all purchased by the website owner, who has inadvertently created duplicate content and on most of these domains. Some of these referring domains have 301's, some don't - but it appears they have all been de-indexed by Google. I'm somewhat out of my depth here (most of what I've said above has come from an agency who said we should address this before being slapped by Google). However I need to explain to my line manage the actual issues in more detail and the repercussions - any anyone please offer advice please? I'm happy to use the agency, or another - but would like some second opinions if possible?
Intermediate & Advanced SEO | | LJHopkins0 -
How can I prevent duplicate pages being indexed because of load balancer (hosting)?
The site that I am optimising has a problem with duplicate pages being indexed as a result of the load balancer (which is required and set up by the hosting company). The load balancer passes the site through to 2 different URLs: www.domain.com www2.domain.com Some how, Google have indexed 2 of the same URLs (which I was obviously hoping they wouldn't) - the first on www and the second on www2. The hosting is a mirror image of each other (www and www2), meaning I can't upload a robots.txt to the root of www2.domain.com disallowing all. Also, I can't add a canonical script into the website header of www2.domain.com pointing the individual URLs through to www.domain.com etc. Any suggestions as to how I can resolve this issue would be greatly appreciated!
Intermediate & Advanced SEO | | iam-sold0 -
Changing domain for a magento store
Hi all, wondering if i could gather some views on the best approach for this please... We currently have a magento site up with about 150,000 pages (although only 9k indexed in Google as product pages are set to no index by default until the default manufacturer description has been rewritten). The indexed pages are mainly category pages, filtering options and a few search results. While none of the internal pages have massive DA - seem to average about 18-24 which isn't too bad for internal pages, I guess - I would like to transfer as much of this over to the new domain. My question is, is it really feasible to have an htaccess with about 10,000 301 redirects on the current domain? The server is pretty powerful so could probably serve the file without issue but would Google be happy with that? Would it be better to use the change url option in WMT instead. Ive never used that so not sure how that would work in this cause. Would it redirect users too? As a footnote, the site is changing because of branding reasons and not because of a penalty of the site. Thanks, Carl
Intermediate & Advanced SEO | | daedriccarl0 -
Unique domains vs. single domain for UGC sites?
Working on a client project - a UGC community that has a DTC model as well as a white label model. Is it categorically better to have them all under the same domain? Trying to figure which is better: XXX,XXX pages on one site vs. A smaller XXX,XXX pages on one site and XX,XXX pages on 10-20 other sites all pointing to the primary site. The thinking on the second was that those domains would likely achieve high DA as well as the primary, and would passing their value to the primary. Thoughts? Any other considerations we should be thinking about?
Intermediate & Advanced SEO | | intentionally0 -
Community inside the domain or in a separate domain
Hi there, I work for an ecommerce company as an online marketing consultant. They make kitchenware, microware and so on. The are reviewing their overall strategy and as such they want to build up a community. Ideally, they would want to have the community in a separate domain. This domain wouldn't have the logo of the brand. This community wouldn't promote the brand itself. The brand would post content occassionally and link the store domain. The reasoning of this approach is to not interfere in the way of the community users and also the fact that the branded traffic acquired doesn't end up buying at the store I like this approach but I am concerned because the brand is not that big to have two domains separated and lose all the authority associated with one strong domain. I would definitely have everything under the same domain, store and community, otherwise we would have to acquire traffic for two domains. 1. What do you think of both scenarios, one domain versus two? Which one is better? 2. Do you know any examples of ecommerce companies with successful communities within the store domain? Thanks and regards
Intermediate & Advanced SEO | | footd0 -
XML Sitemap on another domain
Hi, We've rebuilt our website and created a better sitemap index structure. There's a good chance that we not be able to append the XML files to existing site for technical reasons (don't get me started). I'm reaching out because I'm wondering if can we place the XML files on another website or subdomain? I know this is not best practice and probably very grey but I'm looking for alternatives. If there answer is DON'T DO IT let me know too. Thx
Intermediate & Advanced SEO | | WMCA0 -
Primary Domain or Redirect?
We are starting a new travel guide for a resort town. I have bought an expired domain with decent related links and PR (which seems to have survived the transfer (4 months ago). Beofre we launch the new site I am trying to decide if we should use this expired domain as the primary URL for the new site or just do a permanent redirect and buy a new domain that better matches the theme of the site. I am obviously concerned with starting from scatch with a new domain. I am confident we can build some good rellevant links in a short time but this space is very competetive. Any input would be greatly appreciated.
Intermediate & Advanced SEO | | Locals0 -
Hosting images on multiple domains
I'm taking the following from http://developer.yahoo.com/performance/rules.html "Splitting components allows you to maximize parallel downloads. Make sure you're using not more than 2-4 domains because of the DNS lookup penalty. For example, you can host your HTML and dynamic content on www.example.org and split static components between static1.example.org and static2.example.org" What I want to do is load page images (it's an eCommerce site) from multiple sub domains to reduce load times. I'm assuming that this is perfectly OK to do - I cannot think of any reason that this wouldn't be a good tactic to go with. Does anyone know of (or can think of) a reason why taking this approach could be in any way detrimental. Cheers mozzers.
Intermediate & Advanced SEO | | eventurerob0