Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two blogs on a single domain?
Hi guys, Does anyone have any experience of having (trying to rank) two separate blogs existing on one domain, for instance: www.companysite.com/service1/blogwww.companysite.com/service2/blogThese 2 pages (service 1 and service 2) offer completely different services (rank for different keywords).(for example, a company that provides 2 separate services: SEO service and IT service)Do you think it is a good/bad/confusing search engine practice trying to have separate blogs for each service or do you think there should be only one blog that contains content for both services?Bearing in mind that there is an already existing subdomain for a non-profit part of business that ranks for different keywords: non-profit.companysite.comand it will potentially have another blog so the URL would look like: non-profit.companysite.com/blogAny ideas would be appreciated!Thanks
Intermediate & Advanced SEO | | kellys.marketing0 -
301s being indexed
A client website was moved about six months ago to a new domain. At the time of the move, 301 redirects were setup from the pages on the old domain to point to the same page on the new domain. New pages were setup on the old domain for a different purpose. Now almost six months later when I do a query in google on the old domain like site:example.com 80% of the pages returned are 301 redirects to the new domain. I would have expected this to go away by now. I tried removing these URLs in webmaster tools but the removal requests expire and the URLs come back. Is this something we should be concerned with?
Intermediate & Advanced SEO | | IrvCo_Interactive0 -
Removing index.php
I have question for the community and whether or not this is a good or bad idea. I currently have a Joomla site that displays www.domain.com/index.php in all the URLs with the exception of the home page. I have read that it's better to not have index.php showing in the URL at all. Does it really matter if I have index.php in my URL? I've read that it is a bad practice. I am thinking about installing the sh404SEF component on my site and removing the index.php. However, I rank pretty high for the keywords I want in Google, Bing and Yahoo. All of the URLs that show up in the searches have index.php as part of the URL. Has anyone ever used sh404SEF to remove the index.php and how did you overcome not loosing your search engine links? I don't want an existing search showing www.domain.com/index.php/sales and it not linking to the correct page which would now be www.domain.com/sales. I guess I could insert the proper redirects in the htaccess file. But I was hoping to avoid having every page of my site in the htaccess file for redirecting. Any help or advice appreciated.
Intermediate & Advanced SEO | | MedGroupMedia0 -
Unique domains vs. single domain for UGC sites?
Working on a client project - a UGC community that has a DTC model as well as a white label model. Is it categorically better to have them all under the same domain? Trying to figure which is better: XXX,XXX pages on one site vs. A smaller XXX,XXX pages on one site and XX,XXX pages on 10-20 other sites all pointing to the primary site. The thinking on the second was that those domains would likely achieve high DA as well as the primary, and would passing their value to the primary. Thoughts? Any other considerations we should be thinking about?
Intermediate & Advanced SEO | | intentionally0 -
Domain Forwarding - SEO Impacts?
I have a site that has been active for years - thinkbiglearnsmart.com. Awhile ago I had purchased about 50 domain names that were relevant to my company. I still have those urls and would like to use them to point to different pages on my site - just because they have good key words in the URLs. For example - one is dreamweavertrainingclassesonlinelive.com. Currently they are all redirecting to my homepage. A. is that hurting me? B. I would like to redirect to the more relevant page. ie the page dedicated to Dreamweaver training (http://thinkbiglearnsmart.com/dreamweaver-creative-cloud-training-course/ ) Will this hurt my Dreamweaver keyword for example because there is already a 301 redirect on that page from a very old Dreamweaver link which was something like thinkbiglearnsmart.com/dreamweaver C. On my hosting account where I can select where the URL forwards to - it has an option for "Location forwarding" and "Frame forwarding" - currently they are set to Frame forwarding - which one is best? Any help is much appreciated!!! Thank you!
Intermediate & Advanced SEO | | webbmason0 -
How to find all indexed pages in Google?
Hi, We have an ecommerce site with around 4000 real pages. But our index count is at 47,000 pages in Google Webmaster Tools. How can I get a list of all pages indexed of our domain? trying to locate the duplicate content. Doing a "site:www.mydomain.com" only returns up to 676 results... Any ideas? Thanks, Ben
Intermediate & Advanced SEO | | bjs20100 -
How do I list the subdomains of a domain?
Hi Mozers, I am trying to find what subdomains are currently active on a particular domain. Is there a way to get a list of this information? The only way I could think of doing it is to run a google search on; site:example.com -site:www.example.com The only issues with this approach is that a majority of the indexed pages exist on the non-www domain and I still have thousands of pages in the results (mainly from the non-www). Is there another way to do it in Google? OR is there a server admin online tool that will tell me this information? Cheers, Dan
Intermediate & Advanced SEO | | djlaidler0 -
Yoast SEO Plugin: To Index or Not to index Categories?
Taking a poll out there......In most cases would you want to index or NOT index your category pages using the Yoast SEO plugin?
Intermediate & Advanced SEO | | webestate0
