Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does stock art photo attribution negatively impact SEO by leaking Google Page Rank?
Greetings: Companies such as Shutterstock often require that buyers place credit attribution on their web pages when photos you buy from them appear on these pages.. Shutterstock requests that credit attribution links such as these be added: Songquan Deng / Shutterstock.com Do these links negatively impact SEO? Or do search engines view them as a positive? Thanks,
Web Design | | Kingalan1
Alan0 -
How to link to a site without passing ANY linkjuice (other than simply nofollowing)
I have heard that there are other ways of linking to a site, to completely avoid passing any seo value I think it was even in a whiteboard friday video where I saw Rand say something about doing a 307 "temporary" redirect, or something like that? Basically, I want to let my customers compare our prices with ebay, but I don't want to have ebay outrank us (for obvious reasons) Any help?
Web Design | | TylerAbernethy0 -
Unable to set preferred domain, can I verify a site that's already redirected?
I'm in the process of trying to set a preferred domain in webmaster tools -- to set our www version as preferred vs. the non www. version. IT is already redirecting non-www to www, but I get this message when trying to change settings "Part of the process of setting a preferred domain is to verify that you own http://mnn.com/. Please verify http://mnn.com/." While we own the domain, I am not sure how we can have Google access a file at [http://mnn.com/some_file when we are forwarding all requests for non-www to our www site.
Web Design | | Aggie
Note: The apache rewrite predates me and I'm not sure how / why we have two domains set up, but I'm trying to fix the preferred domain now.Am I able to verify the non version once the redirect is in place.Any ideas??? Help???Thanks!Lisa0 -
How SEO friendly (or unfriendly) this site structure is
We have a client who wants a site structure like this http://thethomasoliverband.com/home - try to scroll down on the content and see how the url of the site changes. Would there be any problems on trying to SEO this type of structure?
Web Design | | paulct0 -
Responsive Vs Mobile Sites
I know this is some cutting edge technology, but I think that this will be a very important topic in the coming months, as html5/css3 becomses more and more the standard, or at least standardized, I think the topic of this in relation to SEO will also arise much more. My question is simple, is it better to code a responsive site, or a completely mobile site for a small company with no special needs (mobile ordering, ecommerce, etc...) I obviously know the visuall differences, and, personally, I think respomsive websites look better. From an seo perspective, my big thing is for the resizing, for example, with WordPress, when you reach the tablet size you can set the sidebar to basically display:none, can that impact your website? I would really appreciate any feedback
Web Design | | ZacharyRussell0 -
Critique a site I'm working on please
Hi guys, I have a list of things to do on this website that is longer than my arm but thought I would gather a few more thoughts on the site before moving on further with it. The site is here. It went live a couple of days ago for live testing. I have a news section coming very soon at theworkplacedepot.co.uk/news - it is in live testing now. One glaring issue for me is that some of the products simply aren't "buyable". What I mean by that is that the options are too vague and too vast - as an outsider I wouldnt have a clue what I was buying! Other issues: The banner at the bottom of the HP is a test - its design will be changing. The sitemap has some issues and will be addressed. There is no area yet to "log-in". However, critique for me pretty please! 🙂
Web Design | | MattJanaway0 -
Suggestions For My Ecommerce Site
I am starting to work on an ecommerce site that I am part owner of. My partner who is the other owner started the site a while back and because he has no internet marketing experience the site didn't come out very well. I am currently overhauling the site and here is a list of things first on my list so I can at least get started on some seo and even ppc. I would really appreciate it if you could take a look at our store and see if you think I am missing anything or you could suggest anything else that really should be done immediately or something that is wrong. The site is www.clubfitnesswarehouse.com Update magento to newest version Fix url structure to be SEO friendly for homepage, category pages, product pages etc. For an example of a site that has great seo friendly urls please refer to examples on http://www.bigfitness.com/ site. For example these pages. http://www.bigfitness.com/treadmillstore.html , http://www.bigfitness.com/bosplincy.html , Remove top navigation menu and instead create left sidebar navigation menu created to navigate site and products. For example of these types of sidebars please refer to sidebar on http://www.bigfitness.com/ or http://www.americanfitness.net/ We will NOT be using same structure as what is in top navigation currently. Categories and keywords of categories will be changed and some will stay the same structure as is currently in navigation For header we would like to feature our shipping policy, returns, privacy policy, no sales tax etc. For example I have seen that I like refer to http://www.spiderofficechairs.com/ Obviously we do not want to copy but something similar for our own site. All images on homepage must be changed to include clickable txt so google can read text not just pictures. We would also like product pages modified to make more user friendly and conversion increased. Add to cart button needs to be changed, text needs to be brighter instead of dull grey color. Add to cart button must also be moved ABOVE THE FOLD! Also on product pages we would like to add sections in addition to Product Description, of Specs, and About section for each product page. We also need the ability to change this information as needed when we need to.
Web Design | | PEnterprises0 -
Ecommerce web site with too many internal links
Hi, We're using Magento CE 1.4.0.1 for our ecommerce web site with a fairly flat navigation system i.e. 9 major categories display across the top menu that when you roll over display 2-20 sub categories (which take you to a groups of similar products) and then individual product pages. The categories and sub categories are available to click on as part of a dynamic Html menu system on each page. Each page also shows a small number of related products. This linking structure seems fairly standard and yet Seomoz throws up the error message, "Too Many On-page links" for most pages on our site. Do I need to really worry about this? Is there much can be done to improve this on an ecommerce web site with a large catalogue of products? I've looked at the Knowledge Base but I don't feel the existing responses adequately address the issue for ecommerce sites.
Web Design | | languedoc0