Robots.txt issue with indexation
-
Hello
i have a problem with one of the rules for robots.txt
i have a multilingual mutation of entire page on www.example.com/en/
I want to make indexable /allow/ the main page under /en/
but not indexable /disallow/ everything else under /en/*
Please help me how to write the rule.
-
Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
-
There is no option like
/en/index.html
The only adress where you can reach the english main page version is www.example.com/en/
-
Name the page you want indexing something and you can use the following:
Disallow: /en/
Allow: /en/index.html
Always test robots.txt in google webmaster tools.
Hope that helps,
Keith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to get multilingual posts indexed on Google
Last year on June I decided to make my site multi-lingual. The domain is: https://www.dailyblogprofits.com/ The main language English and I added Portuguese and a few posts on Spanish. What happened since then? I started losing traffic from Google and posts on Portuguese are not being indexed. I use WPML plugin to make it multi-lingual and I had Yoast installed. This week I uninstalled Yoast and when I type on google "site:site:dailyblogprofits.com/pt-br" I started seeing Google indexing images, but still not the missing posts. I have around 145 posts on Portuguese, but on Search Console it show only 57 hreflang tags. Any idea what is the problem? I'm willing to pay for an SEO Expert to resolve this problem to me.
International SEO | | Cleber0090 -
How to check Robots.txt File
I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
International SEO | | seobac1 -
Backlinks issue
Hi guys So I have 3 top level domains - I'm seeing backlinks that point to each other. Is this something that I need to fix? I'm not too sure if this practice is ethical, but I'm not entirely sure how all the links are pointing towards each other and if this is something I will get pulled up for how do i go about fixing it? My 3 domains are as follows: zenory.com, zenory.com.au and zenory.co.nz I would appreciate some advice around this! Cheers
International SEO | | edward-may0 -
Geotargeting Issues
I have a different problem then most. My international website (www.solmelia.com) is showing number one in english for "SOL MELIA" in the Mexican google search engine. Plus the 3rd listing on google.com.mx is our homepage in spanish but it is showing up as a 401. We need to redirect the ccTLD (www.solmelia.es) to our current spanish version that is actually a subdomain (es.solmelia.com). Please let me know how I can fix both issues.
International SEO | | Melia0 -
Geotargetting Issues
I have a different problem then most. My international website (www.solmelia.com) is showing number one in english for "sol melia" in the Mexican google search engine. Plus the 3rd listing on google.com.mx is our homepage in spanish but it is showing up as a 401. We need to redirect the ccTLD (www.solmelia.es) to our current spanish version that is actually a subdomain (es.solmelia.com). Please let me know how I can fix both issues.
International SEO | | Melia0 -
Geotargeting Issues
Hi, I am having a problem with geotargeting. I have gTLD (solmelia.com) but my international languages are in subdomains. My english version is showing in the google mexico search engine. I need to know what I can do to show our international as well as regional languages (fr.solmelia.com) in the right search engines. Also, having problem with mexico (es) and spain (es) causing dulipication issues and showing in both search engines. I dont think geotargeting in webmaster tools would be the action or setting a lang attribute. Let me know what you think I should I do.
International SEO | | Melia0 -
Geolocation and Indexing
Hi all, Our company owns site that have over 5 millions pages in Google index. We are locating in German, but our business aimed to US market. So, recently I checked index of our site using region targeting in US and there were only 150k of pages, but when I checked targeting in German there were almost 5 billion pages. Our server/IP locating in US, all the backlinks are from US sites. So, why there it is only small part of the site indexed in US? Regards, Dmitry
International SEO | | bubliki0 -
De-Indexing URLs from a specific Locale
Is it possible to de-index a specific URL from showing up in a specific locale? For example, if I want to de-index http://www.example.com/category/product1 from http://www.google.co.uk but not http://www.google.com, is that possible?
International SEO | | craigsmith3330