Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is this is Wow HIT ME IN THE Face Google bug or am I missing something?
We have a page on our site https://www.spurshelving.co.uk/shop/bigimage.aspx?m=353&i=3436 which enders happily on all browsers as far as I am aware and is reasonably well optimised. So when google sent me a link to a new test tool I just had to check it out. https://testmysite.withgoogle.com/intl/en-gb/?utm_source=awareness&utm_medium=email&utm_campaign=tmsv1awareness&utm_content=header Well the result was shocking...... The page that renders in the results is a default missing product page and not the page that the link renders on a web page. I played a little and simply used the I=3436 attribute and the page appeared no problem I then reversed the attributes so that they were i=3436&m=353 and the page again resolved totally as expected. This indicates to me that Google have an issue with aspx attributes. Now I know what to do but is this same issue an issue in spidering and indexing pages. If is is wow that is a big smack in the face. Does it also harm search results in other engines. Keen for comments here
Web Design | | Eff-Commerce0 -
May open the site's homepage...?
In mobile search results, many of my inner pages show the title tag, but the link is removed. And instead of showing the meta description, there is the following: May open the site's homepage
Web Design | | Dino64
Try anyway | Learn more Both "Try Anyway" and "Learn more" are links. Try anyway used to link to the homepage, but after we removed a redirect plugin and got rid of all the html mobile pages (now we're just using the Wordpress responsive layout), the link is now going to the correct page. So, it in fact, does NOT "open the site's homepage" any longer. But, we're still seeing this on mobile searches for our inner pages. It's been almost 3 weeks now. Any ideas on how to get this back to normal?0 -
A/B Testing.. Are you doing? how is it been? What do you think would be the best path for who is starting now?
Hey Mozers, One of my 2014 resolutions is to start doing A/B Testing, so far I have been following "best practices" and "common sense" when comes to website design, but I would like to go above and beyond. I was hoping a could get a few tips some of you that are already doing A/B testing. How is it been? Do you see a great ROI? What do you think would be best path for who is starting now? Any book or links you would recommend? Thanks
Web Design | | Felip30 -
Ecommerce Site - SEO
We have a Business Catalyst Site with the Same product Listed in 2 different catalogs. Each product page is the same page with different URLs you can see it here: http://www.yourpharmacy.co.nz/beauty/clarins-skincare/clarins-advanced-extra-firming-eye-contour-cream-20ml http://www.yourpharmacy.co.nz/clarins/clarins-advanced-extra-firming-eye-contour-cream-20ml Any suggestions welcome
Web Design | | OnlineAssetPartners0 -
Question #1: Does Google index https:// pages? I thought they didn't because....
generally the difference between https:// and http:// is that the s (stands for secure I think) is usually reserved for payment pages, and other similar types of pages that search engines aren't supposed to index. (like any page where private data is stored) My site that all of my questions are revolving around is built with Volusion (i'm used to wordpress) and I keep finding problems like this one. The site was hardcoded to have all MENU internal links (which was 90% of our internal links) lead to **https://**www.example.com/example-page/ instead of **http://**www.example.com/example-page/ To double check that this was causing a loss in Link Juice. I jumped over to OSE. Sure enough, the internal links were not being indexed, only the links that were manually created and set to NOT include the httpS:// were being indexed. So if OSE wasn't counting the links, and based on the general ideology behind secure http access, that would infer that no link juice is being passed... Right?? Thanks for your time. Screens are available if necessary, but the OSE has already been updated since then and the new internal links ARE STILL NOT being indexed. The problem is.. is this a volusion problem? Should I switch to Wordpress? here's the site URL (please excuse the design, it's pretty ugly considering how basic volusion is compared to wordpress) http://www.uncommonthread.com/
Web Design | | TylerAbernethy0 -
Sites went from page 1 to page 40 + in results
Hello all We are looking for any insight we can get as to why all (except 1) of our sites were effected very badly in the rankings by Google since the Panda updates. Several of our sites londonescape.com dublinescape.com and prague, paris, florence, delhi, dubai and a few others (all escape.com urls) have had major drop in their rankings. LondonEscape.net (now.com (changed after rank drop) ), was ranked between 4th & 6th but is now down around 400th and DelhiEscape.net and MunichEscape.com were both number 1 for several years for our main key words We also had two Stay sites number 1 , AmsterdamStay and NewYorkstay both .com ranked number 1 for years , NewYork has dropped to 10th place so far the Amsterdam site has not been effected. We are not really sure what we did wrong. MunichEscape and DelhiEcape should never have been page 1 sites ) just 5 pages and a click thru to main site WorldEscape) but we never did anything to make them number 1. London, NewYork and Amsterdam sites have had regular new content added, all is checked to make sure its original. **Since the rankings drop ** LondonEscape.com site We have redirected the.net to the .com url Added a mountain of new articles and content Redesigned the site / script Got a fair few links removed from sites, any with multiple links to us. A few I have not managed yet to get taken down. So far no result in increased rankings. We contacted Google but they informed us we have NOT had a manual ban imposed on us, we received NO mails from Google informing us we had done anything wrong. We were hoping it would be a 6 month ban but we are way past that now. Anyone any ideas ?
Web Design | | WorldEscape0 -
What is a really great bounce rate for a product or service site? What does Good look like?
I am really curious about a result I have never seen before. Our bounce rate went down a lot on a new site. So, what is good??? Recently, we took on a project with a company that offers a product they install for consumers and who had been in business for 15 plus years. The company is successful, has good customer base of those who have been made very happy, etc. It is not a repeat sale type of product, etc. One and done. Their site when we began talking was roughly a year old and was not well constructed but not terrible. Most of the issues were around I frames, use of older coding, poor SEO, etc. There was not really a way to "redesign" and we built a new site. This became a true collaboration in a B2B environment as the owner pushed us like crazy. Not the bad kind of push, the one that makes you say to your team, "Let's find a way!" The result, IMO, was a gorgeous site. But, as you know, those are a dime a dozen. But, to get to the point, when we took over the account they had a bounce rate of around 45%. I did not see this as either good or bad, but a fact and for this industry probably not bad at all. In all honesty, I was not looking at that as a first metric I wanted to move, but it was obviously at or near the top for all the reasons we know. So, this site is a local business, not an everyday product and gets about 2500 to 3000 uniques per month. If we compare to May of 2011/2012: 2011 2012 Total Visitors 1852 3,298 Uniques 1609 2,740 Pageviews 5,634 23,203 Pages/visit 3.04 7.04 Avg Duration 2.05 3.20 Yes, I am leaving off what we are getting, yes, I am leaving off the site. Please don't hate me. I am really wanting to see what others see with site changes and bounce rates first and will disclose. So, what's a great bounce rate? How do you know?
Web Design | | RobertFisher0 -
IP block in Google
Our office has a number of people performing analysis and research on keyword positions, volume, competition etc. We have 1 external static IP address. We installed the static IP so we can filter out our visits in Google Analytics. However by 10 AM we get impssible CAPTCHA's or even get blocked in Google. Do you have any experience with such an issue? Any solutions you can recommend? Any help would be appreciated! SXI5A.png
Web Design | | Partouter0