How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you index a Google doc?
We have updated and added completely new content to our state pages. Our old state content is sitting in a our Google drive. Can I make these public to get them indexed and provide a link back to our state pages? In theory it sounds like a great link building strategy... TIA!
Intermediate & Advanced SEO | | LindsayE1 -
How to Get Permalinks Indexed?
Hey Everyone, I'm so happy to be apart of this community and assert knowledge where and when I can. I joined the community for one specific reason and I hope to employ the help of everyone here in conjunction with solving my SEO problem. I have a few years experience in SEO/SEM and have been continuously learning, while learning to adapt to continuous changes (I think we can all relate lol). At any rate, here is what I am experiencing frustration with. I'm the SEO Analyst for a company that is trying to compete for the keyword phrase "Lyft Promo Code". We have been trying to place page one on google for over a year now to no avail. I have gotten my direct domain url to appear on pages 1 & 2, but can't seem to get permalinks or "Sub-URL's" indexed. If you google this phrase you will see what I mean. The top result is:http://rideshareapps.com/lyft-promo-code-credit/
Intermediate & Advanced SEO | | Number_One_Deisgns
This url has an aggregated rating and appears page one for the phrase aforementioned above. What we have managed to do, as I mentioned is get www.couponcodeshero.com on page two. However, we have noticed that the page one trend is all permalinks. However when we have tried to emulate the pages structure and index priority, we are unable too. Our page:
http://couponcodeshero.com/lyft-promo-code-rideshare-guide/ I have ran multiple on-page graders from many resources and have not been able to get this page indexed as a permalink on any page that directly correlates with the Keyword Phrase. In essence, I'm looking for some direction from individuals who may have experienced this before. I have spent a good amount of time Googling and searching forum databases but can not find any direct content that explains how to index a permalink. I hope to get some great ideas from the individuals here! If you do know of any articles or even previously answered questions here please direct me there. it is only my intention to add value to the community! Schieler Mew
Number One Designs0 -
[wtf] Mysterious Homepage De-Indexing
Our homepage, as well as several similar landing pages, have vanished from the index. Could you guys review the below pages to make sure I'm not missing something really obvious?! URLs: http://www.grammarly.com http://www.grammarly.com/plagiarism-checker It's been four days, so it's not just a temporary fluctuation The pages don't have a "noindex" tag on them and aren't being excluded in our robots.txt There's no notification about a penalty in WMT Clues: WMT is returning an "HTTP 200 OK" for Fetch, is showing a redirect to grammarly.com/1 (alternate version of homepage, contains rel=canonical back to homepage) for Fetch+Render. Could this be causing a circular redirect? Some pages on our domain are ranking fine, e.g. https://www.google.com/search?q=grammarly+answers A month ago, we redesigned the pages in question. The new versions are pretty script-heavy, as you can see. We don't have a sitemap set up yet. Any ideas? Thanks in advance, friends!
Intermediate & Advanced SEO | | ipancake0 -
When does Google index a fetched page?
I have seen where it will index on of my pages within 5 minutes of fetching, but have also read that it can take a day. I'm on day #2 and it appears that it has still not re-indexed 15 pages that I fetched. I changed the meta-description in all of them, and added content to nearly all of them, but none of those changes are showing when I do a site:www.site/page I'm trying to test changes in this manner, so it is important for me to know WHEN a fetched page has been indexed, or at least IF it has. How can I tell what is going on?
Intermediate & Advanced SEO | | friendoffood0 -
Is my other domain making me not rank?
Hi there, We have a .co.uk website which was ranking well for a number of highly competitive keywords, however in February 2012 those rankings for those keywords suddenly dropped off Google all together and have never came back. A few possibilties to why this has happened: We launched a .ie website which has exactly the same content, could this be the reason for the drop? I have put in all the necessary steps in making sure Google ranks these geographically correct by using hreflang and making sure everything is setup properly in webmaster tools. Why I think it could be this: If I copy and paste the first few paragraphs of text from the pages in the .co.uk website that were ranked highly in Google.co.uk it's the .ie version that appears not the .co.uk version. Here is the webpages in question: http://www.avogel.co.uk/health/menopause/ http://www.avogel.ie/health/menopause/ Forgot to mention, the reason we have these two websites is due to different currency and legalities. Hope someone can help me out with this.
Intermediate & Advanced SEO | | Paul780 -
What would be the best domain choice?
Hello I got a website www.keywordCA.com and I'm ranking #1 spot on "keyword" but what I notice if you have the exact match you get more site links and etc. Like this keyword that match with my domain name "keyword CA" The ideal name will be www.keyword.com but is taken and the owner don't want to sell the domain (at least he is not using it, is just parked) and I also got the domain www.keyword.net Do you think www.keyword.net will be much better than KeywordCA.com in order to get more exposure and google will generate more site links?
Intermediate & Advanced SEO | | jpgprinting0 -
Keyword weight in Domain Name
Hi All, I'm looking for some advice on URL structure. Our domain name includes 1/2 of many keyword strategies that we're considering. For example our domain is like, dive.com Keyword strategies that we're looking at targeting would include things like, "dive shop", "dive equipment" etc etc Are we competitive to have a structure like this? dive.com/shop/ dive.com/equipment/ Or will this structure yield stronger results? dive.com/dive-shop/ dive.com/dive-equipment/ Your thoughts are appreciated. Thanks! Malcolm
Intermediate & Advanced SEO | | MAGNUMCreative0 -
Removing large section of content with traffic, what is best de-indexing option?
If we are removing 100 old urls (archives of authors that no longer write for us), what is the best option? we could 301 traffic to the main directory de-index using no-index, follow 404 the pages Thanks!
Intermediate & Advanced SEO | | nicole.healthline0