Robots.txt issue with indexation
-
Hello
i have a problem with one of the rules for robots.txt
i have a multilingual mutation of entire page on www.example.com/en/
I want to make indexable /allow/ the main page under /en/
but not indexable /disallow/ everything else under /en/*
Please help me how to write the rule.
-
Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
-
There is no option like
/en/index.html
The only adress where you can reach the english main page version is www.example.com/en/
-
Name the page you want indexing something and you can use the following:
Disallow: /en/
Allow: /en/index.html
Always test robots.txt in google webmaster tools.
Hope that helps,
Keith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to get multilingual posts indexed on Google
Last year on June I decided to make my site multi-lingual. The domain is: https://www.dailyblogprofits.com/ The main language English and I added Portuguese and a few posts on Spanish. What happened since then? I started losing traffic from Google and posts on Portuguese are not being indexed. I use WPML plugin to make it multi-lingual and I had Yoast installed. This week I uninstalled Yoast and when I type on google "site:site:dailyblogprofits.com/pt-br" I started seeing Google indexing images, but still not the missing posts. I have around 145 posts on Portuguese, but on Search Console it show only 57 hreflang tags. Any idea what is the problem? I'm willing to pay for an SEO Expert to resolve this problem to me.
International SEO | | Cleber0090 -
Why are some regions/countries not indexing correctly?
Hi All, Recently I've added different regions (website.com/se/ etc) to Google search console and pointed them to their relevant countries, but only half are working when I search from a regions IP with a VPN and use the correct Google search ( Google.se etc ). Will this correct over time? or is something else causing them not to be indexed up correctly? Thanks in advance <colgroup><col width="81"><col width="104"></colgroup>
International SEO | | WattbikeSEO
| Country | Appear in SERP 17/12/2018 |
| AU | TRUE |
| CZ | TRUE |
| DK | TRUE |
| HK | TRUE |
| IE | TRUE |
| IT | TRUE |
| KR | TRUE |
| NL | TRUE |
| NZ | TRUE |
| SE | TRUE |
| SG | TRUE |
| US | TRUE |
| ZA | TRUE |
| AE | FALSE |
| AT | FALSE |
| CH | FALSE |
| CN | N/A |
| DE | FALSE |
| EE | FALSE |
| ES | FALSE |
| FI | FALSE |
| FR | FALSE |
| GB | FALSE |
| GR | FALSE |
| JP | FALSE |
| NO | FALSE |
| PL | FALSE |
| RU | FALSE |
| SI | FALSE |
| TR | FALSE |0 -
Hreflang tags and canonical tags - might be causing indexing and duplicate content issues
Hi, Let's say I have a site located at https://www.example.com, and also have subdirectories setup for different languages. For example: https://www.example.com/es_ES/ https://www.example.com/fr_FR/ https://www.example.com/it_IT/ My Spanish version currently has the following hreflang tags and canonical tag implemented: My robots.txt file is blocking all of my language subdirectories. For example: User-agent:* Disallow: /es_ES/ Disallow: /fr_FR/ Disallow: /it_IT/ This setup doesn't seem right. I don't think I should be blocking the language-specific subdirectories via robots.txt What are your thoughts? Does my hreflang tag and canonical tag implementation look correct to you? Should I be doing this differently? I would greatly appreciate your feedback and/or suggestions.
International SEO | | Avid_Demand0 -
Google is still indexing with https,i removed ssl for my website
My website is claydip.com. I removed ssl for my website, but when i type claydip in google search it is still displaying with https and saying no description available..i lost visitors from search..kindly help me. I moved from bluehost to deamhost. I had a ssl at bluehost, when i move to dreamhost i am not using it.
International SEO | | knextweb8190 -
International Sites - Sitemaps, Robots & Geolocating in WMT
Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint0 -
Ranking issues for UK vs US spelling - advice please
Hi guys, I'm reaching out here for what may seem to be a very simple and obvious issue, but not something I can find a good answer for. We have a .com site hosted in Germany that serves our worldwide audience. The site is in English, but our business language is British (UK) English. This means that we rank very well for (e.g.) optimisation software but optimization software is nowhere to be found. The cause of this to me seems obvious; a robot reading those two phrases sees two distinct words. Nonetheless, having seen discussions of a similar nature around the use of plurals in keywords, it would seem to me that Google should have this sort of thing covered. Am I right or wrong here? If I'm wrong, then what are my options? I really don't want to have to make a copy of the entire site; apart from the additional effort involved in content upkeep I see this path fraught with duplicate content issues. Any help is very much appreciated, thanks.
International SEO | | StevenHowe0 -
Geolocation and Indexing
Hi all, Our company owns site that have over 5 millions pages in Google index. We are locating in German, but our business aimed to US market. So, recently I checked index of our site using region targeting in US and there were only 150k of pages, but when I checked targeting in German there were almost 5 billion pages. Our server/IP locating in US, all the backlinks are from US sites. So, why there it is only small part of the site indexed in US? Regards, Dmitry
International SEO | | bubliki0 -
De-Indexing URLs from a specific Locale
Is it possible to de-index a specific URL from showing up in a specific locale? For example, if I want to de-index http://www.example.com/category/product1 from http://www.google.co.uk but not http://www.google.com, is that possible?
International SEO | | craigsmith3330