Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Https pages indexed but all web pages are http - please can you offer some help?
Dear Moz Community, Please could you see what you think and offer some definite steps or advice.. I contacted the host provider and his initial thought was that WordPress was causing the https problem ?: eg when an https version of a page is called, things like videos and media don't always show up. A SSL certificate that is attached to a website, can allow pages to load over https. The host said that there is no active configured SSL it's just waiting as part of the hosting package just in case, but I found that the SSL certificate is still showing up during a crawl.It's important to eliminate the https problem before external backlinks link to any of the unwanted https pages that are currently indexed. Luckily I haven't started any intense backlinking work yet, and any links I have posted in search land have all been http version.I checked a few more url's to see if it’s necessary to create a permanent redirect from https to http. For example, I tried requesting domain.co.uk using the https:// and the https:// page loaded instead of redirecting automatically to http prefix version. I know that if I am automatically redirected to the http:// version of the page, then that is the way it should be. Search engines and visitors will stay on the http version of the site and not get lost anywhere in https. This also helps to eliminate duplicate content and to preserve link juice. What are your thoughts regarding that?As I understand it, most server configurations should redirect by default when https isn’t configured, and from my experience I’ve seen cases where pages requested via https return the default server page, a 404 error, or duplicate content. So I'm confused as to where to take this.One suggestion would be to disable all https since there is no need to have any traces to SSL when the site is even crawled ?. I don't want to enable https in the htaccess only to then create a https to http rewrite rule; https shouldn't even be a crawlable function of the site at all.RewriteEngine OnRewriteCond %{HTTPS} offor to disable the SSL completely for now until it becomes a necessity for the website.I would really welcome your thoughts as I'm really stuck as to what to do for the best, short term and long term.Kind Regards
Web Design | | SEOguy10 -
Question Concerning HTML5/CSS Templates & Google Mobility Issues
Hi all, Looking for some kind of solution for a responsive update for a site and I am wondering if there are any templates (not Wordpress) that are both great SEO wise and would also pass muster with the impending Google update for responsiveness? I was looking at things like Canvas and Porto ( http://themeforest.net/popular_item/by_category?category=site-templates ) but can't find any discussion on whether or not these things have been addressed with any of these templates. If any of you have suggestions or other places to look for something that could possibly fit the bill (even if temporarily) I would be very appreciative. Thank you so much in advance!
Web Design | | Pixelwik1 -
Why aren't Images in G+ product page posts showing up in SERPs for brand searches?
Before 1-2 weeks ago, our G+ posts containing links to our product pages would show up in in SERPs (when searching for our brand name) with a thumbnail of the product image. Now, they do not (see image below for visual). Our tech team confirmed there hasn't been any coding change that might be to blame and I see that this isn't happening to other sites. Any idea what may be the problem here? tcnhLgy
Web Design | | znotes0 -
Wordpress Blog Providing SEO to Main Site
Hi, I recently started a very much "learn on the job" SEO position, transitioning from a copywriting background. We currently have a wordpress blog up and running (and producing some decent quality content too I hope!) at example.com/blog/ and a sign up page located at example.com (sorry, can't put the address right now) for a site that is being custom built as it's got some nifty software linking to back end systems. My question is whether the content on the blog will bring SEO benefits to the main domain or whether it'll just be for the blog itself? If the latter, should we navigate the blog onto the a separate page of the main site? Thanks so much! I'm learning as much as I can as quickly as I can, but somethings still get me in a little bit of a tizzy.
Web Design | | LeahHutcheon0 -
Question #1: Does Google index https:// pages? I thought they didn't because....
generally the difference between https:// and http:// is that the s (stands for secure I think) is usually reserved for payment pages, and other similar types of pages that search engines aren't supposed to index. (like any page where private data is stored) My site that all of my questions are revolving around is built with Volusion (i'm used to wordpress) and I keep finding problems like this one. The site was hardcoded to have all MENU internal links (which was 90% of our internal links) lead to **https://**www.example.com/example-page/ instead of **http://**www.example.com/example-page/ To double check that this was causing a loss in Link Juice. I jumped over to OSE. Sure enough, the internal links were not being indexed, only the links that were manually created and set to NOT include the httpS:// were being indexed. So if OSE wasn't counting the links, and based on the general ideology behind secure http access, that would infer that no link juice is being passed... Right?? Thanks for your time. Screens are available if necessary, but the OSE has already been updated since then and the new internal links ARE STILL NOT being indexed. The problem is.. is this a volusion problem? Should I switch to Wordpress? here's the site URL (please excuse the design, it's pretty ugly considering how basic volusion is compared to wordpress) http://www.uncommonthread.com/
Web Design | | TylerAbernethy0 -
301 redirects from old site to new
hey all, we just did a site redesign and have less pages on the new site than the old. is it bad to redirect multiple pages from the old site to the same page on the new? for example redirect ...com/apps ...com/android ...com/mobile and point them all to....com/custom-apps thanks!
Web Design | | jaychow0 -
Branding site
We are starting a new used car site in latam, and we are in the process of developing the brand, part of the team would like to have the "auto" (spanish for car" in the name and some no. The question is, form a SEO point of view, how strong signal is to have the string auto as part as the URL, and can you have a complete unrelated url and have several URLs with the auto string on it pointing in a 301 way to the non unrelated brand name as a way around? (sorry about my english, I hope you can understand the question) PS, yes, there are a lot of queries involving auto in our target market.
Web Design | | Bligoo0 -
How to serve a Mobile & Full Site using one URL?
Hello, Does anyone know of any resources or tutorials that outline how to serve a smartphone-formatted website using the same URL as the full site? I know that one solution is using media-queries to serve a seperate CSS stylesheet, but you still have the full HTML source code. In other words, I might want to serve a smartphone & desktop user different content, but under one URL. WP Touch (Wordpress Plugin) is a perfect example of what I mean, but how is it technically achieved? It serves two different sets of HTML for smartphone & full, but using one URL http://www.bravenewcode.com/store/plugins/wptouch-pro/
Web Design | | petecampbell-bmi0