Website Server Issue?
-
I'm getting error messages that a website cannot be crawled and it might be due to the following issues:
Couldn't access the webpage because the server either timed out or refused/closed the connection before our crawler could receive a response. How to fix: Please contact your web hosting technical support team and ask them to fix the issue
Could Possibly Be:
1. DDoS protection system.
OR
2. Overloaded or misconfigured serverThey asked me to talk to my hosting company about this issue and he's at a loss (I don't think he knows everything he needs to know potentially). Have you seen these issues before? Where is the best spot to start troubleshooting this issue?
-
Hi,
Try using any other crawler to see if they provide you with the same message (ScreamingFrog), in some cases it could mean that the site is overloaded and will start blocking the crawler. We're doing something similar for ours as we have a lot of scrapers but exclude the 'good' sites.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Weird Layout on Initial Website Load?
Whenever I open my site from an uncached source, like google incognito, for a split second it displays purple links and a white background while it loads the rest of the content. I've included a screenshot. Is there any way to fix that.? The site is www.kemprugegreen.com. u8P9q
Web Design | | KempRugeLawGroup1 -
Need suggestion: What is the best internal linking structure for our website to gain in SEO & UX too?
Hi all, We have 3 different editions of our product we are selling with 20 features. 1st edition & 2nd edition comes with 15 features in which 10 are common in each edition. 3rd edition comes with all 20 features. Now what's the best way to interlink and show the navigational menu to highlight 3 editions and features as well? Much appreciated if some one refer me a website with such structure. Thanks
Web Design | | vtmoz0 -
Multiple H1 Tags for different section on one webpage in HTML5 Website? Should I have only one?
https://www.tcs.com website has multiple H1 tag on around 40 out of 1000 pages. Webpage has different section and each section is important that's why used multiple H1 tag (one h1 for each section) **I understand, google will not penalize for multiple H1. But having multiple H1, for site like tcs, is it good or should use only one H1 tag? **Pls check https://www.tcs.com to see heading tag(6 heading tag used).
Web Design | | JayprakashSEO0 -
Looking to remove SSL because it is causing very slow website download speeds. Does WP have a plugin that redirects SSL urls to non SSL urls?
After some extended debate with our web development team we are considering dropping the SSL from our website because it is adding almost 2 additional seconds to our download speeds. We know there is a SEO boost from having a SSL but we believe the extended download speeds maybe outweighing the benefit. However we are concerned about the SEO implications of having no method possible of redirect SSL to non SSL webpages. Does anybody know of a Wordpress Plugin that can force redirect SSL urls to non SSL urls?
Web Design | | RosemaryB0 -
Website organic traffic unchanged, impressions took a 98% drop in the last week.
Hi all, I have a very curious predicament and I'd be grateful if someone could shed some light on the situation. As mentioned in the title, organic traffic to our website has remained unchanged, but organic impressions have taken a 98% drop in the last week. This happened suddenly over one day; on October 22, impressions were 700, on October 23, they were 500, and on October 24 they drastically dropped to 50. The next two days they were at 22 and then up to 35. Organic traffic, however, showed the normal "weekend drop" as of October 24, and is still showing normal level (even increased a bit) continuing into this week. These are organic impressions according to Google Analytics and Google Webmaster tools. We did perform a complete site redesign a month ago. Could this be an effect from the redesign? We also noticed drop in Domain Authority, but our competitors suffered a similar (if not greater) drop as well, so we wondered if it could be due in part to the algorithm update. If anyone could shed some light on the situation I would be so appreciative! Thanks!
Web Design | | Joanne_Pendon0 -
Redirect From .aspx to .html if already indexed - Website Redesign
Hi Guys I would like to know if somebody could possibly shed some light on this for me. We are in the process of re-designing our site, but we are keeping all of our content in terms of site structure, internal linking etc. the same. Now we were wondering if it would be a SEO best practice for us to change our pages' extension from .aspx to .html and just put a re-direct from the aspx to the html pages. Or should we keep everything as is, and maybe just revise our on-page seo efforts as well as do some more link-building. I just have to note that we are currently ranking very well for top positions and obviously all these pages are already nicely indexed. And then another question I have is with regards to our mobi site of this same website.Our dev team created it using Responsive Web Design, but they decided to implement techniques that show and hide content based on what device you are viewing it on. So when viewing it on your desktop, it will show content as per normal, but when viewing it on a mobile device it will hide this content and show the content formatted for that specific mobile device. So we are obviously sitting with a case of dup content here.Is this technique acceptable, or is there a workaround/different way of implementing this? Thanks In Advance Dave
Web Design | | DavidZA10 -
Should I use the google mod_pagespeed in my apache server?
Anyone already use it? There is some speed benefit? http://code.google.com/speed/page-speed/docs/module.html
Web Design | | Naghirniac0 -
Is the server or host a metric for SERPs? Amazon Webstore a plus?
Hello Friends, I have been pondering a question that I bet Ben or some of the other clever folks this list will know. Do the search engines give a preference to sites hosted and served from larger companies like Amazon? With an Amazom Webstore, it would have our domain name but Amazon would supply much of the structure and back end like card processing and check out. Could having amazon as the server be an indicator of better results in the SERPS similar to having a stronger Domain Authority? Are there any draw backs for optimization when using an Amazon Webstore? Thank you. QP
Web Design | | QuietProgress0