Why are these results being showed as blocked by robots.txt?
-
If you perform this search, you'll see all m. results are blocked by robots.txt: http://goo.gl/PRrlI, but when I reviewed the robots.txt file: http://goo.gl/Hly28, I didn't see anything specifying to block crawlers from these pages.
Any ideas why these are showing as blocked?
-
Hi,
Your robots.txt file is very .. steroid healthy. It has his own universe
Are you 100% sure all of the entries are legit and clean ?
First thing I would do is to check Web M;aster Tools for the mobile subdomain. If you don't have it yet, that will be a good place to start - to verify the m subdomain.
Once in WeB Master Tools - you can debug this in no time.
Cheers.
-
but, even when i search from my mobile device, I get the same results (that m. is blocked)
-
I can't submit because I haven't claimed m. in GWT
-
If you haven't already done so, I recommend testing your robots.txt file against one of your mobile pages (such as m.healthline.com/treatments) in Google Webmaster Tools. You can do this by logging into GWT, then click Health, then Blocked URLs.
If you have already tested it in GWT, can you let us know what the results said?
-
Another good article from the community
-
So after a little it or research as I never ever came past this before as all the site we do are responsive, I found this
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=72462
It seems Google wont index a website that they think is a mobile website within the main serp, and vice verse ...
Hope that helps, cause it had me puzzled
Regards
John
-
Which directory are you storing your mobile website files within ...
-
Oh, sorry, on further investigation I see its just your mobile site that are being blocked ...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Not showing up in search results for non-branded terms
Hello! Can anyone see any glaring reasons why this post: "98 Book Marketing Ideas That Can Help Authors Increase Sales" isn't on page one of Google — or even page 10! — for the term "book marketing ideas"? Many other sites with lower domain and page authority — even ones linking to this article — are ranking on the first ten pages for this term, and I can't figure out why we're not appearing anywhere. The same thing is happening for ALL of our other blog posts, and the keywords they're optimized for. According to GA, the only terms we're getting clicks from are branded keywords. This subdomain is now 2 years old, and the domain bookbub.com has been around for 5 years. Our domain authority is 61. We have the Yoast SEO plugin installed and are following all the standard SEO best practices. We have enough external links to at least be ranking within the first 10 pages of this Google search. I feel like there's something technically wrong, maybe in the code or backend, but nobody here can figure it out, and our hosting provider WP Engine has no ideas. Moz is returning crawl errors on our site, mainly "Error Code 804: HTTPS (SSL) Error Encountered" and "Error Code 803: Incomplete HTTP Response Received." I have confirmed with WP Engine that everything is set up correctly on our end, and that this is a known Moz issue. I've reached out to Moz's support team about this, and am awaiting a response. But what else am I missing? There's got to be something — I've been blogging for 10 years for different companies and my own personal websites, and I've never come across anything like this before. I'm completely stuck! I'd appreciate any insights you can offer. Thanks in advance! 🙂 EDIT: I heard back from Moz on those errors. The 804 errors are a Moz-side issue — their crawler isn't equipped to be able to handle SNI. They're looking into a resolution, and this wouldn't affect search engine crawlers. Regarding the 803 error: "When you see an 803 error, that means your site closed its TCP connection to our crawler before our crawler could read a complete HTTP response. You don't see this error when you go to the page in your browser because content-length is an outdated component for modern browsers and they will disregard this error, but the intention of our crawler is to report any errors that might be occurring. So the crawler is configured to detect and report such errors." The only thing I can think to do here is go back to WP Engine with this information, but other than that, I'm not sure what this could mean or how to fix it, or if this might be the underlying technical issue keeping us from ranking.
Intermediate & Advanced SEO | | bookbubpartners1 -
Google robots.txt test - not picking up syntax errors?
I just ran a robots.txt file through "Google robots.txt Tester" as there was some unusual syntax in the file that didn't make any sense to me... e.g. /url/?*
Intermediate & Advanced SEO | | McTaggart
/url/?
/url/* and so on. I would use ? and not ? for example and what is ? for! - etc. Yet "Google robots.txt Tester" did not highlight the issues... I then fed the sitemap through http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php and that tool actually picked up my concerns. Can anybody explain why Google didn't - or perhaps it isn't supposed to pick up such errors? Thanks, Luke0 -
Robots txt is case senstive? Pls suggest
Hi i have seen few urls in the html improvements duplicate titles Can i disable one of the below url in the robots.txt? /store/Solar-Home-UPS-1KV-System/75652
Intermediate & Advanced SEO | | Rahim119
/store/solar-home-ups-1kv-system/75652 if i disable this Disallow: /store/Solar-Home-UPS-1KV-System/75652 will the Search engines scan this /store/solar-home-ups-1kv-system/75652 im little confused with case senstive.. Pls suggest go ahead or not in the robots.txt0 -
Why some pages show schema and some don't in Google?
I notice Google displays the schema(reviews, price, availability etc.) in results only for some of our item pages in same category using same template. Any ideas why this is happening. They are created around same time - more than a year ago. Schema was also added a year ago.
Intermediate & Advanced SEO | | rbai0 -
Issue with Robots.txt file blocking meta description
Hi, Can you please tell me why the following error is showing up in the serps for a website that was just re-launched 7 days ago with new pages (301 redirects are built in)? A description for this result is not available because of this site's robots.txt – learn more. Once we noticed it yesterday, we made some changed to the file and removed the amount of items in the disallow list. Here is the current Robots.txt file: # XML Sitemap & Google News Feeds version 4.2 - http://status301.net/wordpress-plugins/xml-sitemap-feed/ Sitemap: http://www.website.com/sitemap.xml Sitemap: http://www.website.com/sitemap-news.xml User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Other notes... the site was developed in WordPress and uses that followign plugins: WooCommerce All-in-One SEO Pack Google Analytics for WordPress XML Sitemap Google News Feeds Currently, in the SERPs, it keeps jumping back and forth between showing the meta description for the www domain and showing the error message (above). Originally, WP Super Cache was installed and has since been deactivated, removed from WP-config.php and deleted permanently. One other thing to note, we noticed yesterday that there was an old xml sitemap still on file, which we have since removed and resubmitted a new one via WMT. Also, the old pages are still showing up in the SERPs. Could it just be that this will take time, to review the new sitemap and re-index the new site? If so, what kind of timeframes are you seeing these days for the new pages to show up in SERPs? Days, weeks? Thanks, Erin ```
Intermediate & Advanced SEO | | HiddenPeak0 -
What can I do to put these pages back in the top results?
Hello here, here is an interesting question for you. The following 2 webpages from our website have been ranking well on Google (usually on the 1st or 2nd page) for the past 12 years. They are among our oldest, highly relevant product pages on our site: http://www.virtualsheetmusic.com/score/Moonlight.html http://www.virtualsheetmusic.com/score/Eliza.html And we could always find them with the keyword "moonlight sonata sheet music" or "fur elise sheet music". Now, since the last November these pages don't show up anymore despite they are still present in the index. It is pretty hard to understand why those pages don't show up in the search results for those keywords as they used to, above all if you consider that those are among our best, most popular and unique product pages! But instead to struggle to understand why we lost presence (Panda? Some unknown sort of penalization?), has anyone any suggestions to help us to have those pages back in the top results? What do you suggest to do in such kind of cases? Any ideas and thoughts are very welcome! Thank you in advance.
Intermediate & Advanced SEO | | fablau0 -
C Block IP Links Strategy
Hi guys i run a web design company and have around 50 sites that i have designed most dont have links but to us i was considering adding a footer link that will link to a blog page within that site, each post on each site will have unique content about the project and about us as a design company. As you can see most of my ip address are c blocks, any advice here please, thanks in advance Example Ip list
Intermediate & Advanced SEO | | Will_Craig
abc.32.230.1
def.20.252.37
ghi.48.68.82
zz.32.229.131
zz.32.231.208
zz.32.253.87
xx.170.40.170
xx.170.40.172
xx.170.40.232
xx.170.40.247
xx.170.40.32
xx.170.43.200
xx.170.44.103
xx.170.44.105
xx.170.44.108
xx.170.44.111
xx.170.44.127
xx.170.44.137
xx.170.44.146
xx.170.44.157
xx.170.44.77
xx.170.44.81
xx.170.44.86
xx.170.44.95
xx.170.44.96 [question edited by staff to remove full IP addresses]0 -
No results for Yahoo. Can someone fill me in on something I am missing?
I have been working on a website for a little more than 2 weeks. I have seen improvements for several of my keywords in both Google and Bing SERPs (first page in Bing and working on it in Google) but have seen no improvements with Yahoo (No keywords are appearing in the top 50 results.) Am I missing something with Yahoo? I have performed onpage optimization so that titles, meta descriptions, bolded text, header tags, alt tags, anchor text, etc is all there. I have also built offpage links using blogs, other people's websites, a few directories (submitted but not displayed), social media, etc. I have never experienced this kind of issue before. Any help would be greatly appreciated.
Intermediate & Advanced SEO | | Jeffrey21320