Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google: What factors contribute to rank a landing page in a specific country?
Hi community, I would like to know what is the criteria to rank a landing page in specific country Google search. For example, if we want to rank our landing page for "GDPR Australia", what are the factors which will impact beside writing the cotent related to the above mentioned keyword? Thanks
Algorithm Updates | | vtmoz0 -
Ranking gone for the original page and a shortened url ranks instead.
Hi Experts!!! Wishing you all a Merry Christmas and Happy New Year In Advance. I am been facing a issue with a few of my SERP results for "Singapore Visa" and related keyword. Until last to last Saturday i.e 16th December, I ranked for Singapore visa keyword with this url https://in.musafir.com/Visa/singapore-visa.aspx !!! But since 18th December I am ranking for "Singapore Visa" keyword with this url and message below it in place of description. Singapore visa - Musafir.com
Algorithm Updates | | sainath
go.musafir.com/Singapore-visa
No information is available for this page.
Learn why The go.musafir.com/Singapore-visa redirects to https://in.musafir.com/Visa/singapore-visa.aspx with some UTM parameters. The URL go.musafir.com/Singapore-visa is a shortened URL which was used for SMS marketing and all of a sudden Google has picked it in SERP instead of Singapore VIsa Landing Page. The Singapore visa Main page is not blocked by Robots.txt file. Please help me to resolve this.1 -
Duplicate Product Pages On Niche Site
I have a main site, and a niche site that has products for a particular category. For example, Clothing.com is the main site, formalclothing.com is the niche site. The niche site has about 70K product pages that have the same content (except for navigation links which are similar, but not dupliated). I have been considering shutting down the niche site, and doing a 301 to the category of the main site. Here are some more details: The niche sites ranks fairly well on Yahoo and Bing. Much better than the main site for keywords relevant to that category. The niche site was hit with Penguin, but doesn't seem to have been effected much by Panda. When I analyze a product page on the main site using copyscape, 1-2 pages of the niche site do show, but NOT that exact product page on the niche site. Questions: Given the information above, how can I gauge the impact the duplicate content is having if any? Is it a bad idea to do a canonical tag on the product pages of the niche site, citing the main site as the original source? Any other considerations aside from duplicate content or Penguin issue when deciding to 301? Would you 301 if this was your site? Thanks in advance.
Algorithm Updates | | inhouseseo0 -
Member's Badge as Link Building to Homepage or Internal Pages?
Providing members and embeddable badge is a well known link building tactic. Is it better to have the badges from hundreds or even thousands of members link back to the homepage of a website, or a lot of different inner pages? The inner pages would the their individual's profile which sits under a category (such as a service and organisation by location). Member's websites would be related to the content of the website generally. What are the advantages of each? 1. Links to homepage make it easier to rank for competitive keywords on the homepage? If the types of websites were to vary a lot, say a carpet cleaning website and a web designer website, if they all linked to the homepage, would it cause some confusion for the link profile?
Algorithm Updates | | designquotes0 -
How to get Yahoo visitors to my site
I get great traffic from Google but Yahoo is at about a 20 to 1 ratio on visitors. Is there anything I should do to increase Yahoo traffic? I bought a Yahoo Directory listing about 3 months ago but it did no good. Thanks, Boo
Algorithm Updates | | Boodreaux0 -
Too Many On-Page Links
After running a site analysis on here it has come up and said that I have a lot o pages with too many on page links and that this might be why the site is being penalized. Thing is I am not sure how to remedy this as one page that says it has 116 links is this one : http://www.whosjack.org/10-films-with-some-crazy-bitches/ Although there is only one link in the body Then again our home page has 165 http://www.whosjack.org which again it says is too many. The thing is is that surely it doesn't count on links all over the page as other wise every news homepage would be penalised? For example what would happen here on this home page? : http://www.dazeddigital.com/ Can anyone help me see what I am missing? Are there possible hidden links anywhere I should be looking for etc? Thanks
Algorithm Updates | | luwhosjack0 -
Google showing different pages for same search term in uk and usa
Hi Guys, I have an interesting question and think Google is being a bit strange.. Can anyone tell me why when I input the term design agency in Google.co.uk it shows one page, but when i tyupe in the same search term in Google.com (worldwide search) it shows another page.. Any ideas guys? Is this not bit strange?? Any help here be much appreciated.. Thanks Gareth
Algorithm Updates | | GAZ090 -
Website moving up and down SERPs alongside others in 'blocks'.
I have noticed that since the so-called 'Panda' updates my website has been moving up and down the Google SERPs in a kind of 'block' alongside other unrelated websites for certain keyword phrases. Whenever there is upwards or downwards movement it happens in tandem with the other websites in those blocks and it is very frustrating. Why is this and has anybody experienced anything similar? The website - http://bit.ly/jIFHpm The search engine - Google, US The keyword phrase - First phrase of website meta title Sorry for being so cryptic I just don't like openly giving out certain information - think it's a bad hangover from the Google Webmaster help forums where everything you post is indexed for Joe nobody to read at his leisure. If anybody would like to mention anything else related to any on or off-page factors on the website then your time would be much appreciated. One thing I am a bit concerned about, for example, is the repition of 'monitors' on the 'monitor shop' dropdown on the top navigation menu and also the sidebar. This would put a localised high density of the keyword in these navigation areas and I am slightly concerned about that. I have no malicious intent and it is appropriate for the user but perhaps the manufacturer names alone would suffice? Thanks in advance.
Algorithm Updates | | teebus0