404 Error on Spider Emulators
-
I recently began working at a company called Uncommon Goods. I ran a few different spider emulators on our homepage (uncommongoods.com) and I saw a 404 Error on SEO-browser.com as well as URL errors on Summit Media's emulator and SEOMoz's crawler. It seems there is a serious problem here. How is this affecting our site from an SEO standpoint? What are the repercussions?
Also, I know we have a lot of javascript on our homepage..is this causing the 404? Any advice would be much appreciated.
Thanks!
-Zack
-
Hey Zack,
It seems your website is now returning a 200 so you apparently managed to fix the problem.
Was the problem coming from the server configuration as I suggested?
Best regards,
Guillaume Voyer. -
Hi Zack,
Yes, having the home page return a 404 error is a HUGE problem. It actually tells the engines that the page doesn't exist so they will stop crawling it and eventualy drop it from their index even if it returns content.
You should solve this problem ASAP!
Best regards,
Guillaume Voyer. -
Hi Guillaume,
Your comments about Javascript on the client side make complete sense to me now and I will examine our Resin config w/ my IT team. Thanks for explaining. Also, as per Beneeb's advice above, I'm going to try making some changes to robots.txt.
From a bigger picture perspective though, do you think this 404 Error is even that big of a deal? Are we likely to be penalized for this in terms of Page Rank, Domain Authority, etc..??
Thanks for your help!
-Zack
-
Hi Zack,
The 404 error has nothing to do with the robots.txt file, it has to do with your server configuration as I said in my answers bellow.
About the robots.txt file, I would remove the Disallow: line if you don't need to block anything.
Best regards,
Guillaume Voyer. -
Hi Beneeb,
That tool is awesome! It definitely helps, thanks! I'm going to show that report to my IT guys today. I think your guess is a very good one. Hopefully I can persuade them to make the changes and we'll see if it resolves the error.
Best Regards,
-Zack
-
Hi Zack,
To be honest with you, it was just a guess. I used a robots.txt syntax checker and saw several issues. You can check out that same tool here & run your current robots.txt file through it:
http://tool.motoricerca.info/robots-checker.phtml
I hope that gets you pushed in the right direction. I'm very new to SEO, but I've worked in the technical support world forever. So, my suggestion is only worth what you paid for it.
-
Hi Beeneeb,
Thank you for your insight. I think this makes sense as I see there is some redundancy in robots.txt as it is now. I'm curious however, why do you think that changing robots.txt will resolve the 404 error?
Best Regards,
-Zack
-
Hi Zack,
Quick followup : Your website will always return 500 to HTTP/1.0 queries. With HTTP/1.1, homepage returns 404 and subpages returns 200.
I saw the website was running on a Resin server rather than a Apache server, then, you might want to look into your Resin server's configuration.
Best regards,
Guillaume Voyer. -
Hi Zack,
Actually, when I use this http header tool and that I input http://www.uncommongoods.com/ I see that the header returned is in fact a 500 Internal Server Error.
The HTTP Header is returned by the server even before the browser can kow that their is javascript on that page so it has nothing to do with javascript.
You'll have to look at the server side as an Internal Server Error and the HTTP Header are returned by the server in opposite to the javascript that is executed client send.
Best regards,
Guillaume Voyer. -
Hi Zack,
Looking at your robots.txt file, you have several errors. I would replace your current robots.txt file with the following:
User-Agent: *
Disallow:Sitemap: http://www.uncommongoods.com/sitemap.xml
(not sure why the message truncated your sitemap file, but you get the picture)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Thousands of 404-pages, duplicate content pages, temporary redirect
Hi, i take over the SEO of a quite large e-commerce-site. After checking crawl issues, there seems to be +3000 4xx client errors, +3000 duplicate content issues and +35000 temporary redirects. I'm quite desperate regarding these results. What would be the most effective way to handle that. It's a magento shop. I'm grateful for any kind of help! Thx,
Technical SEO | | posthumus
boris0 -
Many errors in Search Console (strange parameters)
Hello, I have many strange parameters in my search console that make many 404 pages, for example: mywebsite.com/article-name/&ct=ga&cd=CAIyGjk4YjY4ZDExNTYxOTgzZTk6Y29tOmVuOlVT&usg=AFQjCNFvpYpYpYf9DoyRBBu8jbiQB8JcIQ mywebsite.com/article-name/&sa=U&ved=0ahUKEwj1zMLR0JbLAhUGM5oKHejjBJAQqQIILSgAMAk&usg=AFQjCNEBNFx3dG5B0-16X6eXTS7k-Srm6Q Can someone tell me how to solve it?
Technical SEO | | JohnPalmer0 -
How to inform Google to remove 404 Pages of my website?
Hi, I want to remove more than 6,000 pages of my website because of bad keywords, I am going to drop all these pages and making them ‘404’ I want to know how can I inform google that these pages does not exists so please don’t send me traffic from those bad keywords? Also want to know can I use disavow tool of google website to exclude these 6,000 pages of my own website?
Technical SEO | | renukishor4 -
Why can't I redirect 302 errors to 301's?
I've been advised by IT that due to the structure of our website (they don't use sub-folders) it's not possible to change 302's to 301's. Is this correct, or am I being fobbed off?
Technical SEO | | lindsaytuerena0 -
Redirecting 404
Hi. I'm working on a wordpress site, which got some old deleted pages indexed and now shows a 404 (also in the results) As these old pages earlier got content and probably also some links pointing towards it, what would then be best practice to do? Should i make an 301 redirect? Make the 404 noindex?
Technical SEO | | Mickelp0 -
Help with strange 404 Errors.
For the most part I have never had trouble tracking down 404's. Usually it's simply a broken link, but lately I have been getting these strange errors http://gridironexperts.com/http%3A/www.nfl.com/gamecenter?game_id=29528&season=2008&displayPage=tab_gamecenter/ What does; %C2%94 repersent? The error always points to NFL.com, but we don't link to them...like ever? Can I just 404: http://gridironexperts.com// to fix the problem, as all 404's start with this weird %C2%94 error. Is this error even on my site? Is in the backend...virus? thanks -Mike
Technical SEO | | MikePatch0 -
4XX Errors - Adding %5c%5c to Links
Hi all 😃 Hope someone can help me with this. The internal links on my hubby's business site occasionally break and add %5c%5c%5c endlessly to the end of the url - like this: site.com/about/hours-of-operation/\\\\\\\\% I cannot for the life of me figure out why it is doing this and while it has happened to me from time to time, I can't recreate it. My crawl diagnostics here in my SEOMox campaign show 19-20 urls doing this - it's nuts. Any insight? Thank you!! Jennifer ~PotPieGirl
Technical SEO | | potpiegirl0 -
SEOMoz Crawling Errors
I recently implemented a blog using WordPress on our website. I didn't use WordPress as the CMS for the rest of our site just the blog portion. So as an example I installed Wordpress in http://www.mysite/blog/" not in the root. My error report in SEOMoz went from 0 to 22e. The Moz bot or crawler that SEOMoz uses is reporting a ton of 4xx errors to strang links that shouldn't exist anywhere on the site. Example: Good link - http://www.mysite/products.html Bad link reported by SEOMoz - http://www.mysite/blog/my-first-post/products.html I've also noticed that my page speed as become much slower as reported by Google. Does anybody know what could be happening here? I know that typically it's better to install WordPress in the root and use it to control the entire site but I was under the gun to get a blog out. Thanks
Technical SEO | | TRICORSystems0