Why is site not being indexed by Google, and not showing on a crawl test??
-
On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get There was an error fetching this page. Error description For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.
-
Site last cached 2nd March
Your site is indexed.
Header's returning 200 codes.
Site can be crawled fine, Xenu finds about 27 pages.
Lynxviewer gets through the page alright.
Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.
If no joy, let me know and I'll have another look.
-
The site is www.innerloophomesreport.net, .com. Thanks.
-
Probably going to need the URL on this one.
I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?
-
Hi Robert Fisher,
This problem probably come from the headers of the file and not from the content itself. You might want to look at the headers returned by your URL using one of the following tools :
http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetesWhen you got the headers, I suggest you post it here so we can look into it.
Best regards,
Guillaume Voyer.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
App indexing with several subdomains
Hi, We are currently in the process of getting our app developed. From an SEO point of view I want the app to get indexed and am currently trying to get a step by step guide for it. I've come across this helpful guide: https://moz.com/blog/how-to-get-your-app-content-indexed-by-google but need to go a bit deeper now. Questions I have is related to the "apple-app-site-association" file As far as I could find out the JSON code of that file has the following syntax: {_ “applinks”: {__ “apps”: [],__ “details”: [__ {__ “appID”: “9JA89QQLNQ.com.my.bundle.id“,__ “paths”: [ “/myPath1”, “/myPath2/*” ]__ },__ ]__ }_ } My question is related to the key "appID" I'm wondering how the entry would look like if I have 10 country subdomains and would like to make sure that the deep link clicked in the browser is going to the right subdomain page within the app. Current website structure:
Web Design | | FashionLux
uk.example.com/english-path/
de.example.com/german-path/ Many Thanks in advance0 -
Changing website to Mobile site
Hi All, We are building a new website and would take it live in next few weeks. I want help to understand what all should we consider.. The site structure would be same with additional features but the URL path and site name is same...Should i take any precausions before shutting down the old site and getting the new one live? Thanks
Web Design | | jomin740 -
Is it bad to have /index.php at the end of a uri?
Is it bad for SEO if traffic is directed to "http://www.example.com/someuri/index.php" instead of "http://www.example.com/someuri/" and would it be works setting up a redirect rule at htaccess level?
Web Design | | NoisyLittleMonkey1 -
Having a second homepage for a site would affect my SEO?
Hello guys, One of our clients is planning to have a new landing page for any users hitting the site for the first time. (returning users will still see the current homepage based on cookies ... in other words, the site would technically have 2 home pages). According to this client, they are planning to do something like this: https://www.websitename.com/ (for returning visitors) https://www.websitename.com/newuser (for first time visitors) Our instinct is that is not great to have 2 home pages (that would affect the SEO campaign we are managing for this company) and we are not sure how to handle this. That's why we would appreciate your opinion regarding this topic: From an SEO perspective, do you think this is a good idea? If not, what would you guys do differentiate first-time visitors vs returning visitors without affecting SEO? Maybe just a pop-up? Thanks in advance for your help !
Web Design | | Robertnweil10 -
Will google penalize a website for using a table layout?
I just got a new client today and his entire website layout and structure is using tables instead of divs. This client is on a tight budget and wants to avoid unnecessary hours for re-coding the website, but at the same time he wants me to improve his SEO organically. This is the first time I've been asked to do work on an existing website that uses pure tables for the entire layout and I'm wondering if this effects the SEO in any way. So my question is, will tables effect rankings and SEO in any way?
Web Design | | ScottMcPherson0 -
For a varied product type or keywords group is it best to have several sites?
Hello everyone... Question: I have 7-8 generic keywords that I would like to rank for, is it possible for one site to rank highly for all these different keywords, or would this be best achieved by making 2 or 3 websites in total targeting different keywords (product sectors)? More info: We are in a niche industry & would like to know if it would be beneficial to have several websites made for specific product types rather than one main site? Although these sub classifications of products are nice, they are competitive as they have a high search volume Would it be better to build specific websites that only do that one type of product and have related keyword in domain, content & blogs on the site to that effect to increase relevance and positions as a result? Thanks
Web Design | | Ray_UK0 -
Will my site structure provide decent SEO?
We have an ASP.NET MVC website with a view that can dynamically display each product we offer. The product name is hyphenated in the URL, and this is what we’re using to pull the product from the database. So an example URL would be: http://www.mysite.com/Products/Florida/Sample-Product-Name We have another view that dynamically lists the products offered for each state. This page would contain links to the URL for each product offered in that state. The URL for Florida would be: http://www.mysite.com/Products/Florida We want to make sure that when we enter a new product into the database, the product is indexed by Google the next time our site is crawled. I know that Google will crawl through the links in our website, so the new product should get indexed as long as we have a link to it. In this case, the link will be on the view that lists the products for the corresponding state. I have 2 questions: 1) Is my understanding correct that Google will index the product page as long as it can find a link to it somewhere in my site? 3) To get Google to index each URL for content that is generated dynamically from a database, is having links in my site for each URL the only way to do it? Is there something we can do with the site map? Thanks in advance everyone! -Alex
Web Design | | dbuckles0