Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help, site traffic has dropped significantly since we changed from http to https
Heya, so I am just in charge of the content on the site, and the SEO content, not the actual back-end stuff. A little under 2 weeks ago we switched to https, and our site traffic has been down a lot ever since. When I SERP check our keywords, they don't seem to have dropped in rankings pages. Here is what I got when I asked our dev guy if 301 redirects were put in: I did not add any redirects so all of the content is accessible on both unless individual links get hardcoded one way or the other. The only thing in place is a Cloudflare plugin which rewrites links in cached pages to match the way its accessed, so if for example you access a page over https you don’t get the version cached with a bunch of http links since that will throw up mixed content warnings in the browser. Other than that WP mostly generates all its links to match whatever protocol you are accessing the current page with. We can make specific pages redirect one way or the other in the future if we want to though... As a startup, site traffic is a metric we track to gouge progress, and so I really need to get to the bottom of if it was the change from http to https that has causes the drop, and if so, what can we do about it? Also, in case it is relevant: the bounce rate is now sky high (ave. 15% to 64% this last week!) Any help is very welcome! Site: https://mobileday.com Thank you!
Web Design | | MobileDay1 -
Manufacturer, New Direct-to-Consumer Site (Separate Site, or Sub-Domain?)
Hi All! Working with an established manufacturer, been around for many years, it's an internationally known brand, and their products are sold by thousands on distributors. They recently started a new website (separate from their old established B2B manufacturer site) which will be used to sell direct to customer. The new site is great, with a nice responsive design, clean look, flexible, etc. The problem is, it's a new site with low Domain Authority. The manufacturer's B2B site has been around a while, very high Domain Authority. So, I'd like to be able to harness all the link equity they've build instead of trying to optimize a brand new site. The problem with this old established site is that it IS in fact old. The design is terrible, it's not responsive, old code, bad look and feel, etc. We could incorporate the new B2C site (which has its own CMS) into a sub-domain, like store.site.com. But, I'd worry that site.com's crapiness will limit growth potential for the new pages at store.site.com. Same issue were we to add the new site into a sub-folder, like site.com/store/. On the other side, we could just keep the new site, with it's own domain, sitestore.com, and have product pages and/or category pages from the manufacturer's B2B site link to the relevant pages on the new B2C site. Thanks!
Web Design | | fiberglass0 -
Google text-only vs rendered (index and ranking)
Hello, can someone please help answer a question about missing elements from Google's text-only cached version.
Web Design | | cpawsgo
When using JavaScript to display an element which is initially styled with display:none, does Google index (and most importantly properly rank) the elements contents? Using Google's "cache:" prefix followed by our pages url we can see the rendered cached page. The contents of the element in question are viewable and you can read the information inside. However, if you click the "Text-only version" link on the top-right of Google’s cached page, the element is missing and cannot be seen. The reason for this is because the element is initially styled with display:none and then JavaScript is used to display the text once some logic is applied. Doing a long-tail Google search for a few sentences from inside the element does find the page in the results, but I am not certain that is it being cached and ranked optimally... would updating the logic so that all the contents are not made visible by JavaScript improve our ranking or can we assume that since Google does return the page in its results that everything is proper? Thank you!0 -
How Can I Make My Site iPhone Friendly?
I have been looking into making my website for iphone friendly as my analytics are not great for the iphone and I know when I try to navigate around it on an iphone it can be tough. I was told that if I make changes to the layout that it would affect my layout across everything, which I did not want to do. So I have two questions: Is this correct regarding the layout? If so, if you did something like m.waikoloavacationrentals.com which would be the mobile version how would that possibly effect your rankings with regards to the traffic distribution? Any feedback would be appreciated. Also if anyone has any experience in doing this I would be interested in discussing further.
Web Design | | RobDalton0 -
Duplicate Content? Designing new site, but all content got indexed on developer's sandbox
An ecommerce I'm helping is getting a complete redesign. Their developer had a sandbox version of their new site for design & testing. Several thousand products were loaded into the sandbox site. Then Google/Bing crawled and indexed the site (because developer didn't have a robots.txt), picking up and caching about 7,200 pages. There were even 2-3 orders placed on the sandbox site, so people were finding it. So what happens now?
Web Design | | trafficmotion
When the sandbox site is transferred to the final version on the proper domain, is there a duplicate content issue?
How can the developer fix this?0 -
Site health - webmaster tools
A bit of an odd one. In Webmaster Tools, there's the option to order sites by site health. When we do this our site - http://www.neooptic.com/ - is near the bottom, despite there being little or no crawl errors. Any ideas why this could be happening?
Web Design | | neooptic0 -
Build New Site Without Losing Rankings
Good morning SEOmoz community. I have a question which I am pretty sure I already know the answer to, however i thought I would reach out to my fellow experts to see if anyone had some great advice. I would really like to give my website a makeover. i have two thoughts on this, one is to scrap the site completely and start fresh, the other would be to only change it visually, but keep all the content and on-page optimization. I am terrified of losing my rankings. I am ranked position 1 and 2 for highly competitive terms and have another 15 - 20 keywords on page 1. Any advice would be tremendously appreciated!!!
Web Design | | WebbyNabler0 -
Setup of three major retail sites.. need advice.
I recently have taken a new position responsible for three large national retail sites which are all owned by one parent organization. Through a series of acquisitions, these three major brands have been brought under one umbrella and a brand consolidation is likely not to happen within the next 2-4 years. I have a number of questions I’m hoping to get some feedback on, but first a little more background is necessary. A year ago (before my time) the three sites were over-hauled, but were designed to use one common custom CMS and all of the navigation and nearly all the content is the same (with some exceptions, such as tags, url, etc.). All of the brands have identical products and services; however, each one services a different demographic in the US. The design was intended for ease of management, but is terrible for seo. Additionally, without the geographic reference, they all compete for the same keywords. They have now begun a very large ecommerce project utilizing an ATG platform. The initial direction is to use one platform for all three brands, but keep them on separate domains and with the use of basic switching, replace nominal content such as logos and references of the brands for each of the domains. I’m concerned with this approach and would like to hear your feedback.. When optimizing a page for one keyword set, are they likely to be filtered due to dup content? The argument that management has is that all three current sites rank very well for one keyword on all three sites. They feel it won’t be an issue due to this. One option, that is currently still available, is to tri-band one ecommerce site, but it would have to be on an entirely new domain. The other three domains are very well established and are PR6s. Management, and even I, is afraid to abandon these other domains, but having a single domain would allow us to have unique content and really leverage all efforts to one domain. Thoughts? Any knowledge or thoughts what kind of impact having three domains on one ATG platform will be? Thanks much! John If you feel it will help, please message me and I can share the urls... Also, how would you handle a company blog in this case?
Web Design | | kavaliauskas0