404s in Google Search Console and javascript
-
The end of April, we made the switch from http to https and I was prepared for a surge in crawl errors while Google sorted out our site. However, I wasn't prepared for the surge in impossibly incorrect URLs and partial URLs that I've seen since then.
I have learned that as Googlebot grows up, he'she's now attempting to read more javascript and will occasionally try to parse out and "read" a URL in a string of javascript code where no URL is actually present. So, I've "marked as fixed" hundreds of bits like
/TRo39,
category/cig
etc., etc....But they are also returning hundreds of otherwise correct URLs with a .html extension when our CMS system generates URLs with a .uts extension like this:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.html
when it should be:
https://www.thompsoncigar.com/thumbnail/CIGARS/90-RATED-CIGARS/FULL-CIGARS/9012/c/9007/pc/8335.utsWorst of all, when I look at them in GSC and check the "linked from" tab it shows they are linked from themselves, so I can't backtrack and find a common source of the error.
Is anyone else experiencing this? Got any suggestions on how to stop it from happening in the future? Last month it was 50 URLs, this month 150, so I can't keep creating redirects and hoping it goes away.
Thanks for any and all suggestions!
Liz Micik -
Hi Liz,
What I would do as well is go with a solution around your robots.txt to make sure the crawlers will respect it and don't go on a hunch trying to find new URLs that are embedded somewhere else. Usually it's something you shouldn't worry about too much, it's just the crawler doing a good job trying to find more content/URLs on your site.
Martijn.
-
Google Search Console 404s can be very irritating.
What does your robots.txt file currently say?
You can remove all instances of .html in your htaccess file to help solve for this issue. More on that here: https://alexcican.com/post/how-to-remove-php-html-htm-extensions-with-htaccess/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fixing 404s
One of our sites is littered in 404s and I have the lucky task of clearing them up. I'm using a mixture of 410s and 301s. Party central, I know. Wondered what ways people measure the success of clearing these 404s up on your site? Whether this is CTR or rankings... do you look at site benefits outside of your console errors coming down? Basically trying to understand how much this is worth doing and how to measure it's progress in real terms. Thanks!
Algorithm Updates | | Fubra0 -
Meta Descriptions - Google ignores what we have
Hi I still write meta descriptions to help with CTR. I am currently looking at a page where the CTR needs improving. I check the meta on Google SERPs & it isn't pulling through the meta description we have - but other info on the page. This isn't ideal - why does this happen? Will Google just make the decision and are descriptions not worth writing?
Algorithm Updates | | BeckyKey0 -
Celebrity Profile On The Side of Google For High Profile Person
Hello! When I google "Justin Timberlake" I see web search results and a sidebar. See image below: http://screencast.com/t/qwYeiFZQRzT How does one get their results to display like this? Is this something that Google creates automatically or is it something the celebrity initiates/creates on their behalf. Does the celebrity have any options to choose from as to what displays on this sidebar? What is this called? I look forward to your response. qwYeiFZQRzT
Algorithm Updates | | InternetRep0 -
Google Algorithm change this month - theories ?
Hi Moz fans! If you've not had the chance to check out moz cast go check it out it seems Google has been busy and so much so it broke Moz cast. There has been some discussion on seo round table about the changes and something seems to be going on. I wanted to find out how you guys are all finding it have you had a change in rankings? Any theories on what you think Google is up to? Personally I've seen rather a few of my sites go up in the rankings last week or so. As always look forward to hearing your thoughts and feelings on it Moz.
Algorithm Updates | | GPainter3 -
Google places - are you still registering your companies?
Is Google plus taking over from Google Places? I have our company and website registered on Google places and Google Plus. Now we have a new website ( same company different product) Should I register a new listing on Google places? And what about Google Plus?
Algorithm Updates | | Realtor1010 -
Rank Tracking & Personalized Search
How effective is rank tracking when google tends to deliver personalized search? I tend to clear out my browser of all info, cookies and cache so I can get the best results but how effective are rank tracking algo's in delivering accurate results. I run various apps and tests and I get different results.
Algorithm Updates | | bronxpad0 -
Related Searches in Google
Hello, We're helping a client remove/minimize some negative information about their brand in Google's search results. Just curious about your take on if the related searches that appear at the bottom of Google search results can in any way be influenced or if it is more a combination of so many factors that any one person or organization wouldn't be able to change very easily? I've heard the related results could be influenced if enough queries generated overtake the "negative" queries done initially but I feel like that is venturing into black hat land a bit. thanks -Mike
Algorithm Updates | | mattmainpath0 -
Google results on an Ipad 2
Has anyone else seen different google organic results for a site when viewing on an Ipad compared to computer browser ? I've just checked a site and were no1 on google when searched on the Ipad 2 but when searched on my Macbook we are page 2 ? Could this just be different data centers or do google serve up different results to the 2 devices ? Would be really interested to know if anyone else has seen this. JP
Algorithm Updates | | Prongo0