Why are these m. results showing as blocked?
-
If you go to http://bit.ly/173gdWK, you'll see that m. results are showing as blocked by robots.txt, but we don't have anything in our robots.txt file that specifies to block m. results. Any ideas why these URLs show as blocked?
-
Yeah, I was testing exactly the same when you posted the response. I even tried crawling a googlebot-mobile and still I get the 301 redirect. Which, for everything that I am seeing it is correct, as no matter what browser I use (desktop, mobile, spider) I always get a 301 to the www. version.
@michelleh, are you sure there's a mobile version not being redirected to the www. one?
-
(Using example.com instead of your domain in case you want anonymity later)
If you try to go to any of these m.example.com URLs on your desktop computer, you're redirected on the server to a www.example.com URL. I'm guessing Googlebot and Googlebot-Mobile cannot access m. pages (unless you're sniffing out Googlebot-Mobile specifically to serve it m.example.com pages). If you're looking at screen resolution for these redirects, you might not be catching Googlebot-Mobile, as I don't think Googlebot-Mobile gives a screen resolution in its user agent. I believe you want Googlebot indexing your www. content, and Googlebot-Mobile indexing your m. content, so you'll need to sniff out Googlebot-Mobile's user agent (see here), and redirect it to m. content.
Also of note is that I think these should be 302 temporary redirects, and not 301 permanent redirects between your www. and m. versions, as they're not really permanent, just getting a given user to the right version of the site. Also, you don't let me switch from the mobile version to the desktop version, which drives me bananas! Let users choose after the initial redirect. If you allow people to switch, but maintain 301 redirects, the browsers may cache some of the redirects which will lead to weird behavior if people hit a page that redirected before.
You don't have a robots.txt file at m.example.com/robots.txt, as that redirects to www.example.com/robots.txt even on my phone. I don't think this is the root of the problem, but once you figure things out, you can set up a robots.txt file on your m. subdomain.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt blocked internal resources Wordpress
Hi all, We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted. I've created the following new one: User-agent: *
Intermediate & Advanced SEO | | Mat_C
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-admin/admin-ajax.php However, in the site audit on SemRush, I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts. Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO? Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index? Thanks for your thoughts!2 -
UK version of site showing US Cache and meta description
Hi Fellow Moz'ers We seem to have an issue where some of our UK site is showing meta descriptions from our US site in the serp's and when you check the cache: of the site it's brining up the .com instead of the .co.uk site. example: cache:https://www.tinyme.co.uk/name-labels shows the US site We've checked the href lang tags and they look ok to me (but i'm not an expert) https://www.tinyme.co.uk/name-labels" hreflang="en-gb"/> https://www.tinyme.com/name-labels" hreflang="en-us"/> https://www.tinyme.com.au/name-labels" hreflang="x-default" /> https://www.tinyme.com.au/name-labels" hreflang="en-au"/> We've had a search around and seen people have similar issues, but cant seem to find a definitive solution.
Intermediate & Advanced SEO | | tinyme1 -
H2 header on MOZ bar showing 2,451 # of Characters
My page is Real Estate Listing Service on Wordpress. My H2 header shows 2,451 Characters.... It is picking up my property listings, is this a problem from an SEO standpoint? Any help would be greatly appreciated, and thank you inadvance If required: the website is www.latinproperties.com
Intermediate & Advanced SEO | | LatinProps0 -
Mobile homepage title is showing unrelated text
Hi all, I've read the FAQs and searched the help center. My URL is: http://www.leibish.com/I just discover it today - when searching on mobile for the brand name (search for "Leibish") I find the following title: Reply (see the attached image).Also, when searching for site:leibish.com +reply you'll find at least a few dozens indexed pages with this title.I checked for the following:1. Link bombing - check with ahrefs and majestic - everything seemed fine.2. Alt tag or other hidden text - couldn't find anything. 3. DMOZ or Yahoo directory submissions with this anchor text - yet, nothing.Besides the fact that I must fix this issue I find it fascinating from the SEO perspective. I need the big guns in here - can you help my resolve this mystery? :-)Thanks...
Intermediate & Advanced SEO | | seoperad0 -
Omitted results
Google used to display all my pages now most are under repeat the search with the omitted results included."_ what dose that mean? Dose this predict something bad?_ All pages are unique.
Intermediate & Advanced SEO | | Joseph-Green-SEO0 -
Recently created site indexed; no backlinks showing?
I launched a website for a client in mid-March. The site is already indexed, I have built quite a few links to it (links are also indexed), and ranks well for some targeted keywords. However, when I try to check backlinks to the site with Open Site Explorer, it comes back with "No Data Available For This URL". Is this something I should be worried about or merely a case of 'recency' of page creation'? I know it says that it can take 45-60 days for a site to be included in Linkscape but I'm approaching the 60 days mark and still nothing.
Intermediate & Advanced SEO | | Igor-Avidon0 -
Does using robots.txt to block pages decrease search traffic?
I know you can use robots.txt to tell search engines not to spend their resources crawling certain pages. So, if you have a section of your website that is good content, but is never updated, and you want the search engines to index new content faster, would it work to block the good, un-changed content with robots.txt? Would this content loose any search traffic if it were blocked by robots.txt? Does anyone have any available case studies?
Intermediate & Advanced SEO | | nicole.healthline0 -
10,000 New Pages of New Content - Should I Block in Robots.txt?
I'm almost ready to launch a redesign of a client's website. The new site has over 10,000 new product pages, which contain unique product descriptions, but do feature some similar text to other products throughout the site. An example of the page similarities would be the following two products: Brown leather 2 seat sofa Brown leather 4 seat corner sofa Obviously, the products are different, but the pages feature very similar terms and phrases. I'm worried that the Panda update will mean that these pages are sand-boxed and/or penalised. Would you block the new pages? Add them gradually? What would you recommend in this situation?
Intermediate & Advanced SEO | | cmaddison0