Robots.txt issue with indexation
-
Hello
i have a problem with one of the rules for robots.txt
i have a multilingual mutation of entire page on www.example.com/en/
I want to make indexable /allow/ the main page under /en/
but not indexable /disallow/ everything else under /en/*
Please help me how to write the rule.
-
Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
-
There is no option like
/en/index.html
The only adress where you can reach the english main page version is www.example.com/en/
-
Name the page you want indexing something and you can use the following:
Disallow: /en/
Allow: /en/index.html
Always test robots.txt in google webmaster tools.
Hope that helps,
Keith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Configure Robots.txt File
How to configure correctly the Robots.txt File of the website. Need proper process to follow.Because a lot of my website URLs are excluded by google with the issues in the Robots.txt file.
International SEO | | seobac1 -
Issues with Baidu indexing
I have a few issues with one of my sites being indexed in Baidu and not too sure of how to resolve them; 1. Two subdomains were redirected to the root domain, but both (www. and another) subdomains are still indexed after ~4 months. 2. A development subdomain is indexed, despite no longer working (it was taken down a few months back). 3. There's conflicting information on what the best approach is to get HTTPS pages indexed in Baidu and we can't find a good solution. 4. There are hundreds of variations of the home page (and a few other pages) on the main site, where Baidu has indexed lots of parameters. There doesn't appear to be anywhere in their webmaster tools to stop that happening, unlike with Google. I'm not the one who deals directly with this site, but I believe that Baidu's equivalent of Webmaster Tools has been used where possible to correctly index the site. Has anyone else had similar issues and, if so, were you able to resolve them? Thanks
International SEO | | jobhuntinghq0 -
Lost local organic rankings and international issues
Hi Everyone, Hoping we can get some help from our fellow Mozzers (Mozee's? Mozites?) We have 3 TLD's .com.au .co.uk & .com We noticed an issue a couple of weeks ago where we suddenly lost a lot of our search rankings for keywords on the .com.au site that we'd been top with for a long time. A lot of our Australian visitors were coming through our US site. US & UK sites got increased ranking results. We fixed up what we thought were the issues (Potentially HREF Lang issues and old sitemap issues). Google Search Console is still telling us we have some HREF Lang Errors, (but this could be waiting an updated crawl as the number is decreasing) Our main domain example.com is now showing up as first result in google.com.au search and the example.com.au doesn't show up until page 4 (prior to 2 weeks ago it was number 1) Any input would be appreciated...
International SEO | | tinyme0 -
Google does not index UK version of our site, and serves US version instead. Do I need to remove hreflanguage for US?
Webmaster tools indicates that only 25% of pages on our UK domain with GBP prices is indexed.
International SEO | | lcourse
We have another US domain with identical content but USD prices which is indexed fine. When I search in google for site:mydomain I see that most of my pages seem to appear, but then in the rich snippets google shows USD prices instead of the GBP prices which we publish on this page (USD price is not published on the page and I tested with an US proxy and US price is nowhere in the source code). Then I clicked on the result in google to see cached version of page and google shows me as cached version of the UK product page the US product page. I use the following hreflang code: rel="alternate" hreflang="en-US" href="https://www.domain.com/product" />
rel="alternate" hreflang="en-GB" href="https://www.domain.co.uk/product" /> canonical of UK page is correctly referring to UK page. Any ideas? Do I need to remove the hreflang for en-US to get the UK domain properly indexed in google?0 -
Backlinks issue
Hi guys So I have 3 top level domains - I'm seeing backlinks that point to each other. Is this something that I need to fix? I'm not too sure if this practice is ethical, but I'm not entirely sure how all the links are pointing towards each other and if this is something I will get pulled up for how do i go about fixing it? My 3 domains are as follows: zenory.com, zenory.com.au and zenory.co.nz I would appreciate some advice around this! Cheers
International SEO | | edward-may0 -
Do I have duplicate content issues to be worried about?
Hey guys, We built a website http://www.cylon.com/ targeting different regions but with the same English langauage (Ireland, England and America). The content for the most part is the same set up on 3 different subfolders. http://www.cylon.com/ - Targeting United States in WMT http://www.cylon.com/ie - Targeting Ireland in WMT http://www.cylon.com/uk - Targeting UK in WMT Do I have duplicate content issues to be worried about? If so, how do I get around this issue? Also is there anyway of finding out if Google have in some way penalised these pages for having the same content on other pages trageting different Countries? I have not received any messages from Google in WMT saying there is duplicate so I'm not sure if this is an issue. Thanks Rob
International SEO | | daracreative0 -
Understanding the "Index Status" Data Inside Google Webmaster Tools
Currently there are total 2,787 Articles added to my Blog. The Index Status shows the following report under Index Status>Advance Total Indexed = 12,505 Blocked by robots = 8,659 And when I do search for site:techmaish.com in Google.com, it shows; About 12,200 results (0.15 seconds) Now my question. 1:- Is it normal Or there is something wrong? 2:- If there is something wrong then what is that? Thanks in advance. _ Attached is the screenshot of my GWT._ 7dk.png
International SEO | | techmaish0 -
Geotargeting Issues
Hi, I am having a problem with geotargeting. I have gTLD (solmelia.com) but my international languages are in subdomains. My english version is showing in the google mexico search engine. I need to know what I can do to show our international as well as regional languages (fr.solmelia.com) in the right search engines. Also, having problem with mexico (es) and spain (es) causing dulipication issues and showing in both search engines. I dont think geotargeting in webmaster tools would be the action or setting a lang attribute. Let me know what you think I should I do.
International SEO | | Melia0