Mobile site - allow robot traffic
-
Hi,
If a user comes to our site from a mobile device, we redirect to our mobile site. That is www.mysite/mypage redirects to m.mysite/mypage. Right now we are blocking robots from crawling our m. site. Previously there were concerns the m. site could rank for normal browser searches. To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site).
From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing. Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)
-
Hi Jamie -- I noticed the mobile version of our website is appearing in the Bing search results (from a desktop) --- can you employ the same technique as Google and specify to the Bing/Yahoo! mobile bot to only crawl the mobile version of your site and restrict the normal bot from crawling mobile version? Thanks for additional guidance. I assume this would clear up this issue of our mobile site appearing in normal non-mobile search results.
- Matt
-
My preference is not to redirect to m.domain.com as there are potential issues with m.domain.com listings appearing in natural search. I would use the user agent detection, I think this gives the best user experience as well, rather than re-directing, what type of re-direct are you using anyway?
I would for best practice, still specify google mobile bot to crawl the mobile version of your website only ans restrict normal g bot form the mobile version.
Cheers,
Jamie.
-
Rather than just blocking robots from crawling your mobile site, you should configure your robots.txt file to allow the mobile googlebot to crawl that subdomain specifically and disallow it on your regular domain. This way users searching on mobile devices will be shown your mobile site. In your current set up you risk sending them to your main site from the SERPs.
If you need help getting that set up there is a tool in Google webmasters called "Generate robots.txt" located in the crawler settings under Site Configuration.
-
We don't block robots from our mobile site. Our site is designed to work well with iPad. We have found that our KWs perform slightly better when searched from a mobile device. It's a once little bonus for us. Duplicate content hasn't been a problem, I think the engines are smart enough to figure it out, even without a mobile sitemap.
-
For Google's opinion, you'll want to watch http://www.youtube.com/watch?v=mY9h3G8Lv4k, and read http://googlewebmastercentral.blogspot.com/2009/11/help-google-index-your-mobile-site.html.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My traffic keeps going down...
My website been through many changes the last 5 months. From migrating http to https, server transfer and more. Everything was double checked all the redirects are perfect (301) to the https pages (we have like 28K backlinks so everything needs to be correct), in a matter of fact the migration to ssl made in October and in November the stats went high. The problem is that at the end of November after the server migration the traffic went down and is stably down till today. We re-checked the redirects in every aspect and more in depth and finished fixing issues around end of January. Also before 1 week we stopped the functionality of a parameter (product sort) and removed it from the website entirelly and canonicaled the urls that were using it so if google tries to test those urls with the parameter to see a canonical tag and eventually drop them out of index and stop crawling them. Generally we tried to fix the crawling efficiency and speed. The technical SEO was on fire the last weeks. Still i cant see improvement in rankings, i only see drops. What is the problem here? Where should i look? Any ideas? In webmaster tools we have no 404's and no manual actions. The visibility in Sistrix goes down (check attachment), the ahrefs organic traffic goes down, semrush stats go down... My website is: https://www.pccdkeys.com ecfp87
Technical SEO | | dos06591 -
Site dropped from SERP
Hello, I've been ranking a site for the last 5 months with good success, ranking on the first page for a high traffic keyword. In the beginning of September however, my site completely dropped out of the SERPs for several of those keywords yet my site was still indexed and there was no penalty applied to my site via search console. I would assume this maybe because of the update during the time.My site came back again a week later and it was ranking much higher on the first page (#2). Today, I just checked the SERPs and my site is now gone again. It was there this morning but now as of two hours ago it is gone, as well as one of my main competitors. My site is still indexed and no penalties via search console. Does anyone know what causes these types of issues? Im assuming my site will come back in a week or so with hopefully the same or better ranking, but when I have disruptions like this it really hurts my organic traffic. Any input is appreciated. Thanks!
Technical SEO | | KathleenDC0 -
What is the process for allowing someone to publish a blog post on another site? (duplicate content issue?)
I have a client who allowed a related business to use a blog post from my clients site and reposted to the related businesses site. The problem is the post was copied word for word. There is an introduction and a link back to the website but not to the post itself. I now manage the related business as well. So I have creative control over both websites as well as SEO duties. What is the best practice for this type of blog post syndication? Can the content appear on both sites?
Technical SEO | | donsilvernail0 -
Why is my site not being indexed?
Hi, I have performed a site:www.menshealthanswers.co.uk search on Google and none of the pages are being indexed. I do not have a "noindex" value on my robot tag This is what is in place: Any ideas? Jason
Technical SEO | | Jason_Marsh1230 -
Site Link Issues
For several search terms I get site links for the page http://www.waikoloavacationrentals.com/kolea-rentals/kolea-condos/ It makes sense that that page be a site link as it is one of my most used pages, but the problem is google gave it the site link "Kolea 10A". I am having 0 luck making any sense of why that was chosen. It should be something like "Kolea Condos" or something of that nature. Does anyone have any thoughts on where google is coming up with this?
Technical SEO | | RobDalton0 -
Robots.txt
I have a client who after designer added a robots.txt file has experience continual growth of urls blocked by robots,tx but now urls blocked (1700 aprox urls) has surpassed those indexed (1000). Surely that would mean all current urls are blocked (plus some extra mysterious ones). However pages still listing in Google and traffic being generated from organic search so doesnt look like this is the case apart from the rather alarming webmaster tools report any ideas whats going on here ? cheers dan
Technical SEO | | Dan-Lawrence0 -
Robots.txt usage
Hey Guys, I am about make an important improvement to our site's robots.txt we have large number of properties on our site and we have different views for them. List, gallery and map view. By default list view shows up and user can navigate through gallery view. We donot want gallery pages to get indexed and want to save our crawl budget for more important pages. this is one example of our site: http://www.holiday-rentals.co.uk/France/r31.htm When you click on "gallery view" URL of this site will remain same in your address bar: but when you mouse over the "gallery view" tab it will show you URL with parameter "view=g". there are number of parameters: "view=g, view=l and view=m". http://www.holiday-rentals.co.uk/France/r31.htm?view=l http://www.holiday-rentals.co.uk/France/r31.htm?view=g http://www.holiday-rentals.co.uk/France/r31.htm?view=m Now my question is: I If restrict bots by adding "Disallow: ?view=" in our robots.txt will it effect the list view too? Will be very thankful if yo look into this for us. Many thanks Hassan I will test this on some other site within our network too before putting it to important one's. to measure the impact but will be waiting for your recommendations. Thanks
Technical SEO | | holidayseo0 -
Confused about robots.txt
There is a lot of conflicting and/or unclear information about robots.txt out there. Somehow, I can't make out what's the best way to use robots even after visiting the official robots website. For example I have the following format for my robots. User-agent: * Disallow: javascript.js Disallow: /images/ Disallow: /embedconfig Disallow: /playerconfig Disallow: /spotlightmedia Disallow: /EventVideos Disallow: /playEpisode Allow: / Sitemap: http://www.example.tv/sitemapindex.xml Sitemap: http://www.example.tv/sitemapindex-videos.xml Sitemap: http://www.example.tv/news-sitemap.xml Is this correct and/or recommended? If so, then how come I see a list of over 200 or so links blocked by robots when Im checking out Google Webmaster Tools! Help someone, anyone! Can't seem to understand this robotic business! Regards,
Technical SEO | | Netpace0