Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will numbers & data be considered as user generated content by Google OR naturally written text sentences only refer to user generated content.
Hi, Will numbers & data be considered as user generated content by Google OR naturally written text sentences only refer to user generated content. Regards
Web Design | | vivekrathore0 -
Does stock art photo attribution negatively impact SEO by leaking Google Page Rank?
Greetings: Companies such as Shutterstock often require that buyers place credit attribution on their web pages when photos you buy from them appear on these pages.. Shutterstock requests that credit attribution links such as these be added: Songquan Deng / Shutterstock.com Do these links negatively impact SEO? Or do search engines view them as a positive? Thanks,
Web Design | | Kingalan1
Alan0 -
Does Google penalise for alot of advertising on your site?
I look after the search side of a decorating website on which we carry a large amount of advertising from external brands as that is our business model. Do you know if we would get penalised for having too much advertising - would it be deemed to affect the user experience? Many thanks for your help on this.
Web Design | | Pday0 -
On site SEO opinions
Hi all, I have been testing different configurations for my on-site SEO for a while now and I think I am pretty much there. However it is always nice to know what other SEO's think about my keyword density and usage. My site is http://www.tomlondonmagic.com I am curious as to what you think regarding landing page content, whether you need lots or text or little text? I have just removed links in the text as I feel I want to keep as much juice on my landing page as possible. Thanks all!
Web Design | | TomLondon0 -
Time On Site and SEO?
Does time on site impact rankings? If a person visits your site from the serps or directly visits it by typing in your name in the search field and then leaves within a minute, will that impact your serps? What is the best way to increase time on site?
Web Design | | bronxpad0 -
Preserve Rankings When Switching to a New Site
Hello community! I have a quick question for you regarding preserving my SERPs once I switch a development site to replace the current production site. Basically, we are switching to a new CMS and will be featuring the same content, architectural layout, URL structure, etc. Again, the only difference is that it's going to be on a new CMS. Upon switching to the new version of the site, what would be the best strategy for making sure we preserve our rankings for content already ranking highly within the search engines? Likewise, is there anything additional we may be able to do right-off-the-bat in order to assist content that may not be ranking highly in the SERPs, rank more highly?
Web Design | | NiallSmith0 -
Whats happening with Google UK?
Within the last week we have had a handful of our rankings drop dramatically down the SERPS. About 15% but this an estimate and has not been fully investigated yet. Whilst looking into possible scenarios that could be causing this i wanted to check what the SERPS looked like for the terms that we are still holding position on. Typing "extending dining tables" into Google UK today i was amazed at what i found... Ranking in position 1 and 2 is a massive UK furniture store.
Web Design | | Silkstream
But isnt that the same landing page being returned for both positions?? It appears to be a navigation problem within the site category tags causing duplicate content. However they have been rewarded with the top two positons subsequently pushing our website onto page two. I find it so frustrating that we listen to Googles best practices when it comes to pagination issues yet this is how our hard work is rewarded! Anyone else have any thoughts about this? SERPS.jpg0 -
Two sites in same industry and which shopping cart
Right. So I suspect I am going to sound paranoid here - but you'll all forgive me right?? I am sure I saw a reply to a question on the Q&A suggesting that it was a bad idea to have two sites in one industry as Google may see it as trying to get two bites of the SERP cherry... is this accurate? I have an existing asp.net site in the maternity wear industry here in Australia and am wanting to start another site to appeal to a different customer base... the market is quite broad. There will be a core list of products that are the same between the sites, but also some quite different products. Content, product descriptions and categorys will be different. I have another website that I bought with reasonable age and links in the industry that I was going to 301 to the new site to give it a kick in the juice. So, not wanting to deceive my customers in anyway, I was thinking I would call it a "division of" or "sister site to" the existing ecommerce site, with a single link back and forward between the two sites. Would there be anything wrong with this in googles eyes? Even with same contact details? They would be run on totally different platforms and hosted by totally different providers. Or would you keep them totally seperate and only have contact details in images? Or a step further and have totally different phone numbers etc? Then the shopping cart - I would love some suggestions on which opensourse cart to use, preferrably one that I can set up myself, and that has a good framework for seo. I want to use schema.org, authorship, seo friendly urls all of which I am having trouble getting out of the developer of my asp.net site.... I don't want the new site to be asp.net Thanks in advance!!
Web Design | | catfree0