Robots.txt issue with indexation
-
Hello
i have a problem with one of the rules for robots.txt
i have a multilingual mutation of entire page on www.example.com/en/
I want to make indexable /allow/ the main page under /en/
but not indexable /disallow/ everything else under /en/*
Please help me how to write the rule.
-
Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
-
There is no option like
/en/index.html
The only adress where you can reach the english main page version is www.example.com/en/
-
Name the page you want indexing something and you can use the following:
Disallow: /en/
Allow: /en/index.html
Always test robots.txt in google webmaster tools.
Hope that helps,
Keith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
.fr Site Not Indexed in Search Console
Hey everyone, I am just now learning international SEO, and so appreciate any and all help! 1. I added supershuttle.fr to Search Console about a week ago and it still has 0 indexed pages. It is showing that pages were indexed in the sitemap, and when I check, there are 75 results in Google. Is this something I should be concerned with? Is there a setting that I'm not aware of in Search Console that I need to change? 2. Also, I read the below regarding the automatically translated pages. Would https://en.supershuttle.fr/ be considered an "automatically translated" page? Use robots.txt to block search engines from crawling automatically translated pages on your site. Automated translations don’t always make sense and could be viewed as spam. More importantly, a poor or artificial-sounding translation can harm your site’s perception
International SEO | | SuperShuttle0 -
Low Index: 72 pages submitted and only 1 Indexed?
Hi Mozers, I'm pretty stuck on this and wondering if anybody else can give me some heads up around what might be causing the issues. I have 3 top level domains, NZ, AU, and USA. For some od reason I seem to be having a real issue with these pages indexing and also the sitemaps and I'm considering hiring someone to get the issue sorted as myself or my developer can''t seem to find the issues. I have attached an example of the sitemap_au.xml file. As you can see there is only 1 page that has been indexed and 72 were submitted. Basically because we host all of our domains on the same server, I was told last time our sitemaps were possibly been overwritten hence the reason why we have sitemap_au.xml and its the same for the other sitemap_nz.xml and sitemap_us.xml I also orignially had sitemap.xml for each. Another issue I am having is the meta tag des for each home page in USA and AU are showing the meta tag for New Zealand but when you look into the com and com.au code meta tag description they are all different as you can see here http://bit.ly/1KTbWg0 and here http://bit.ly/1AU0f5k Any advice around this would be so much appreciated! Thanks Justin new
International SEO | | edward-may0 -
Blocking domestic Google's in Robots.txt
Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM0 -
Ranking issues for UK vs US spelling - advice please
Hi guys, I'm reaching out here for what may seem to be a very simple and obvious issue, but not something I can find a good answer for. We have a .com site hosted in Germany that serves our worldwide audience. The site is in English, but our business language is British (UK) English. This means that we rank very well for (e.g.) optimisation software but optimization software is nowhere to be found. The cause of this to me seems obvious; a robot reading those two phrases sees two distinct words. Nonetheless, having seen discussions of a similar nature around the use of plurals in keywords, it would seem to me that Google should have this sort of thing covered. Am I right or wrong here? If I'm wrong, then what are my options? I really don't want to have to make a copy of the entire site; apart from the additional effort involved in content upkeep I see this path fraught with duplicate content issues. Any help is very much appreciated, thanks.
International SEO | | StevenHowe0 -
Geo targeting issue and hosting
Hi guys and gals, this is not a problem per se, but an oddity that I would appreciate some insight on from the big juicy brains in this community. Our site had hosting in the US, and I was concerned that therefore our relevance to our own country (Australia) was diminished because of it. For one of our main keywords we were a few spots behind the competitor on the 1st page for an australian searcher, but when i searched the same keyword from Google.com with gl=us to show US only results, we outranked the competitors by a few spots. On page elements aside (if anything we had more geo identifiers on the ranking page in question) I wanted to move hosts anyway and got hosting in Australia. The next week our search traffic jumped by 25%. But it was almost all US traffic. Australian traffic was unchanged. Any idea how this could happen? It's an .AU domain, hosted in Australia, with on page clearly identifying Australia. I checked webmaster tools and our geo is properly set to Australia. I checked the keywords that the traffic increased for and they are not geo specific at all. Besides that I don't know how else to pin this down. Thanks.
International SEO | | Digital3600 -
Google Indexing Part two
Hi Everybody, I am trying to understand how does Google works, so I ve been reading and researching a lot. But I am still having a problem that I cannot solve. My website is in several languages, but its main language is Catalan. so if you get into my webite: "www.vallnord.com" the default language will be catalan. but if someone using Google.es in Spanish I would like the spanish version of the web to be the main result not the catalan. unfortunately this does not work like this. For a search query like "Esqui Andorra" the catalan version is on the 1st page and the spanish (www.vallnord.com/es) is on the 4th page. Does anybody know why is this happening or how can I solve it? Regards.
International SEO | | SilbertAd0 -
Geotargeting Issues
Hi, I am having a problem with geotargeting. I have gTLD (solmelia.com) but my international languages are in subdomains. My english version is showing in the google mexico search engine. I need to know what I can do to show our international as well as regional languages (fr.solmelia.com) in the right search engines. Also, having problem with mexico (es) and spain (es) causing dulipication issues and showing in both search engines. I dont think geotargeting in webmaster tools would be the action or setting a lang attribute. Let me know what you think I should I do.
International SEO | | Melia0 -
De-Indexing URLs from a specific Locale
Is it possible to de-index a specific URL from showing up in a specific locale? For example, if I want to de-index http://www.example.com/category/product1 from http://www.google.co.uk but not http://www.google.com, is that possible?
International SEO | | craigsmith3330