Robots.txt & Mobile Site
-
Background - Our mobile site is on the same domain as our main site. We use a folder approach for our mobile site abc.com/m/home.html We are re-directing traffic to our mobile site vie device detection and re-direction exists for a handful of pages of our site ie most of our pages do not redirect the user to a mobile equivalent page.
Issue – Our mobile pages are being indexed in desktop Google searches
Input Required – How should we modify our robots.txt so that the desktop google index does not index our mobile pages/urls
User-agent:
Googlebot-Mobile
Disallow: /m
User-agent: `YahooSeeker/M1A1-R2D2`
Disallow: /m
User-agent: `MSNBOT_Mobile`
Disallow: /m
Many thanks
-
Hi,
The fact that you do not have mobile versions for all pages of the site makes your case a bit of an odd one when it comes to best practices. I would recommend if at all possible to address this and make all pages available in mobile format also.
If for whatever reason this is not possible then I think you are better off using a meta robots tag NOINDEX to get your mobile results out of the serps completely and a rel canonical to the relevant desktop page to make extra sure there are no issues with duplicate content.
If you can get all pages into a mobile version then you will want to implement proper redirects based on user string as you are doing, and also implement rel alternate/canonical tags between the desktop and mobile pages of the site (which should make sure the right version of the site shows up in the relevant serps). For more details on the whole setup check:
http://moz.com/blog/how-to-optimize-a-mobile-site and
http://moz.com/blog/mobile-seo-process -
User-agent: Googlebot-Mobile
Disallow: /m
would't that statement prevent Googlebot-Mobile from crawling the mobile pages ? Why would you want to do that ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disallow wildcard match in Robots.txt
This is in my robots.txt file, does anyone know what this is supposed to accomplish, it doesn't appear to be blocking URLs with question marks Disallow: /?crawler=1
Technical SEO | | AmandaBridge
Disallow: /?mobile=1 Thank you0 -
Google Indexing Desktop & Mobile Versions
We have a relatively new site and I have noticed recently that Google seems to be indexing both the mobile and the desktop version of our site. There are some queries where the mobile version will show up and sometimes both mobile and desktop show up. This can't be good. I would imagine that what is supposed to happen is that the desktop version is the one that should be indexed (always) and browser detection will load the mobile version where appropriate once the user is on the site. Do you have any advice on what we should do to solve this problem as we are a bit stuck?
Technical SEO | | simonukss0 -
Sudden traffic down after mobile site
Hi to all. I have a huge problem about my web site traffic. We published our mobile site version under mobile domain in m.etilercerrental.com. And we published mobile site sitemap under main site such as sitemap-mobile.xml and notify google from webmaster tools. After a while whole site traffic gone down so badly. Almost 60% gone down which doesnt make sense. Mobile sitemap is hosted under desktop site. And we didnt add any link to switch sites between mobile and desktop version. We automaticly redirect between desktop and mobile versiyons by detecting useragent. One another this is we have announcement section in dekstop version but we dont have it in mobile. But we are not making any redirection to mobile site when user visits the announcement section even by mobile phone. Because we are not displaying it in mobile version.
Technical SEO | | gkhnrtk
One last this is we are making redirections automatically and not asking user preference such as showing a link in the bottom of page to switch sites betwenn mobile and pc version. While developing mobile site we followed instructions in this https://developers.google.com/webmasters/smartphone-sites/?hl=tr For find the details I would like to give you some site urls. Site Url : http://www.etilercarrental.com/
Sitemap Xml : http://www.etilercarrental.com/sitemap.xml
Mobile Sitemap Xml : http://www.etilercarrental.com/sitemap-mobile.xml0 -
301 Multiple Sites to Main Site
Over the past couple years I had 3 sites that sold basically the same products and content. I later realized this had no value to my customers or Google so I 301 redirected Site 2 and Site 3 to my main site (Site 1). Of course this pushed a lot of page rank over to Site 1 and the site has been ranking great. About a week ago I moved my main site to a new eCommerce platform which required me to 301 redirect all the url's to the new platform url's which I did for all the main site links (Site 1). During this time I decided it was probably better off if I DID NOT 301 redirect all the links from the other 2 sites as well. I just didn't see the need as I figured Google realized at this point those sites were gone and I started fearing Google would get me for Page Rank munipulation for 301 redirecting 2 whole sites to my main site. Now I am getting over 1,000 404 crawl errors in GWT as Google can no longer find the URL's for Site 2 and Site 3. Plus my rankings have dropped substantially over the past week, part of which I know is from switching platforms. Question, did I make a mistake not 301 redirecting the url's from the old sites (Site 2 and Site 3) to my new ecommerce url's at Site 1?
Technical SEO | | SLINC0 -
How to create site map for large site (ecommerce type) that has 1000's if not 100,000 of pages.
I know this is kind of a newbie question but I am having an amazing amount of trouble creating a sitemap for our site Bestride.com. We just did a complete redesign (look and feel, functionality, the works) and now I am trying to create a site map. Most of the generators I have used "break" after reaching some number of pages. I am at a loss as to how to create the sitemap. Any help would be greatly appreciated! Thanks
Technical SEO | | BestRide0 -
Is having no robots.txt file the same as having one and allowing all agents?
The site I am working on currently has no robots.txt file. However, I have just uploaded a sitemap and would like to point the robots.txt file to it. Once I upload the robots.txt file, if I allow access to all agents, is this the same as when the site had no robots.txt file at all; do I need to specify crawler access on can the robots.txt file just contain the link to the sitemap?
Technical SEO | | pugh0 -
How is this site doing this?
http://www.meccabingo.com It shows a splash / promotion page yet you check the cache and it's the real homepage, they are doing this so they don't lose rankings but how are they redirecting users to that but Google is caching the real homepage? is it friendly? thanks!!
Technical SEO | | AdiRste0 -
Is robots.txt a must-have for 150 page well-structured site?
By looking in my logs I see dozens of 404 errors each day from different bots trying to load robots.txt. I have a small site (150 pages) with clean navigation that allows the bots to index the whole site (which they are doing). There are no secret areas I don't want the bots to find (the secret areas are behind a Login so the bots won't see them). I have used rel=nofollow for internal links that point to my Login page. Is there any reason to include a generic robots.txt file that contains "user-agent: *"? I have a minor reason: to stop getting 404 errors and clean up my error logs so I can find other issues that may exist. But I'm wondering if not having a robots.txt file is the same as some default blank file (or 1-line file giving all bots all access)?
Technical SEO | | scanlin0