Crawl anamoly issue on Search Console
-
Has anyone checked the crwal anamoly issue under the index section on Search console? We recently move to a new site and I'm seeing a huge list of excluded urls which are classified as crawl anamoly (they all lead to 404 page). Does anyone know that if we need to 301 redirect all the links? Is there any other smarter/ more efficiently way to deal with them like set up canonical link (I thought that's what they're used for isn't it?)
Thanks!
-
Did you keep the old url in webmaster tools? With sitemap of old domain that redirects? You should make sure that was submitted for crawl so google see's the 301's. Also make sure you redirected all versions http, https, www, etc of old domain* to new https//: version.
Are the pages in fact 404 pages? are they pages in your sitemap? be careful too that they are not bad internal links. Did you crawl site with moz?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Back links issue and how to resolve it
Hi there! We have a client who has been generating back links from external sites over a period of two years with all the same anchor text which all link back to the home page. This anchor text is also their main search phrase they wish to score highly on. In total, they have roughly 300 domain names linking to their site. Over 50 of these domain names all have the same anchor text. These links have been generated through articles and blogs. So roughly 20% of the total number of links all have the same anchor text. Over the past 6 months the client has noticed a steady drop in their rankings for this term. From the back link analysis we have done, we believe it is this which is causing the problem. Does any one else agree? For the remedy, do we go in and see if we can change the anchor text or disavow them through Google webmaster tools? Suggestions? Thanks for your help! P 🙂
White Hat / Black Hat SEO | | Globalgraphics0 -
"Google chose different canonical than user" Issue Can Anyone help?
Our site https://www.travelyaari.com/ , some page are showing this error ("Google chose different canonical than user") on google webmasters. status message "Excluded from search results". Affected on our route page urls mainly. https://www.travelyaari.com/popular-routes-listing Our canonical tags are fine, rel alternate tags are fine. Can anyone help us regarding why it is happening?
White Hat / Black Hat SEO | | RobinJA0 -
Googlebot crawling AJAX website not always uses _escaped_fragment_
Hi, I started to investigate googlebot crawl log of our website, and it appears that there is no 1:1 correlation between a crawled URL with escaped_fragment and without it.
White Hat / Black Hat SEO | | yohayg
My expectation is that each time that google crawls a URL, a minute or so after, it suppose to crawl the same URL using an escaped_fragment For example:
Googlebot crawl log for https://my_web_site/some_slug Results:
Googlebot crawled this URL 17 times in July: http://i.imgur.com/sA141O0.jpg Googlebot crawled this URL additional 3 crawls using the escaped_fragment: http://i.imgur.com/sOQjyPU.jpg Do you have any idea if this behavior is normal? Thanks, Yohay sOQjyPU.jpg sA141O0.jpg0 -
Active, Old Large site with SEO issues... Fix or Rebuild?
Looking for opinions and guidance here. Would sincerely appreciate help. I started a site long, long ago (1996 to be exact) focused on travel in the US. The site did very well in the search results up until panda as I built it off templates using public databases to fill in the blanks where I didn't have curated content. The site currently indexes around 310,000 pages. I haven't been actively working on the site for years and while user content has kept things somewhat current, I am jumping back into this site as it provides income for my parents (who are retired). My questions is this. Will it be easier to track through all my issues and repair, or rebuild as a new site so I can insure everything is in order with today's SEO? and bonus points for this answer ... how do you handle 301 redirects for thousands of incoming links 😕 Some info to help: CURRENTLY DA is in the low 40s some pages still rank on first page of SERPs (long-tail mainly) urls are dynamic (I have built multiple versions through the years and the last major overhaul was prior to CMS popularity for this size of site) domain is short (4 letters) but not really what I want at this point Lots of original content, but oddly that content has been copied by other sites through the years WHAT I WANT TO DO get into a CMS so that anyone can add/curate content without needing tech knowledge change to a more relevant domain (I have a different vision) remove old, boilerplate content, but keep original
White Hat / Black Hat SEO | | Millibit1 -
Controlling crawl speed/delay through dynamic server-code and 503's
Lately i'm experiencing performance trouble caused by bot traffic. Although Googlebot is not the worst (it's mainly bingbot and ahrefsbot), they cause heavy server load from time to time. We run a lot of sites on one server, so heavy traffic on one site impacts other site's performance. Problem is that 1) I want a centrally managed solution for all sites (per site administration takes too much time), which 2) takes into account total server-load in stead of only 1 site's traffic and 3) controls overall bot-traffic in stead of controlling traffic for one bot. IMO user-traffic should always be prioritized higher than bot-traffic. I tried "Crawl-delay:" in robots.txt, but Googlebot doesn't support that. Although my custom CMS system has a solution to centrally manage Robots.txt for all sites at once, it is read by bots per site and per bot, so it doesn't solve 2) and 3). I also tried controlling crawl-speed through Google Webmaster Tools, which works, but again it only controls Googlebot (and not other bots) and is administered per site. No solution to all three of my problems. Now i came up with a custom-coded solution to dynamically serve 503 http status codes to a certain portion of the bot traffic. What traffic-portion for which bots can be dynamically (runtime) calculated from total server load at that certain moment. So if a bot makes too much requests within a certain period (or whatever other coded rule i'll invent), some requests will be answered with a 503 while others will get content and a 200. Remaining question is: Will dynamically serving 503's have a negative impact on SEO? OK, it will delay indexing speed/latency, but slow server-response-times do in fact have a negative impact on the ranking, which is even worse than indexing-latency. I'm curious about your expert's opinions...
White Hat / Black Hat SEO | | internetwerkNU1 -
Potential spam issue - back links
Hi - we have a client whom we work with for SEO. During a review we noticed in Webmaster Tools, there was an IP address with over 30,000 links to our clients site. The IP address is 92.60.0.123. From looking up the IP address details, it looks like it is based in Europe - but we are unable to establish what it is, where the links are and who created it. We are concerned it could be a potential spammer trying to cause an issue with the SEO campaign. Is there any way of finding out any more details apart from the basic information about the location of the IP address? Also - if we submit a disavow via webmaster tools, we are unsure what issue it will have on the clients site if we do not know what it is and the type of links it is creating. Any ideas? Thanks for your help! Phil.
White Hat / Black Hat SEO | | Globalgraphics0 -
Page not being indexed or crawled and no idea why!
Hi everyone, There are a few pages on our website that aren't being indexed right now on Google and I'm not quite sure why. A little background: We are an IT training and management training company and we have locations/classrooms around the US. To better our search rankings and overall visibility, we made some changes to the on page content, URL structure, etc. Let's take our Washington DC location for example. The old address was: http://www2.learningtree.com/htfu/location.aspx?id=uswd44 And the new one is: http://www2.learningtree.com/htfu/uswd44/reston/it-and-management-training All of the SEO changes aren't live yet, so just bear with me. My question really regards why the first URL is still being indexed and crawled and showing fine in the search results and the second one (which we want to show) is not. Changes have been live for around a month now - plenty of time to at least be indexed. In fact, we don't want the first URL to be showing anymore, we'd like the second URL type to be showing across the board. Also, when I type into Google site:http://www2.learningtree.com/htfu/uswd44/reston/it-and-management-training I'm getting a message that Google can't read the page because of the robots.txt file. But, we have no robots.txt file. I've been told by our web guys that the two pages are exactly the same. I was also told that we've put in an order to have all those old links 301 redirected to the new ones. But still, I'm perplexed as to why these pages are not being indexed or crawled - even manually submitted it into Webmaster tools. So, why is Google still recognizing the old URLs and why are they still showing in the index/search results? And, why is Google saying "A description for this result is not available because of this site's robots.txt" Thanks in advance! Pedram
White Hat / Black Hat SEO | | CSawatzky0 -
Penguin issues
Hello everyone, I run about 10 sites and pretty much every single one got hit by Penguin (the traffic plummeted on 24th April). I have never done reciprocal links (except 1 domain upto 2005 or so), I have never bought links, I have never spammed message boards or anything like that (except 1 different domain got hit by negative SEO by someone else) and I have never employed anyone to do any of the above. The way I have created sites for the last 10 years is to try to make them useful and let the links build naturally which more or less worked until April this year. I've been tearing my hair out ever since. The only thing you can say about all of them (apart from that I own them but I've been careful with whois etc) is that the link profile is 100% natural apart from the 2 provisos above. Since April I've hired people but I'm down $20K but not any better in the rankings. A few of the sites are: short-hairstyles.com was number 1 for short hairstyles and short haircuts for years then Penguin came and its dropped off for both. It had 10000 or so spammy message board links posted by someone as negative seo I have got some removed but google webmaster tools still reports them as there. There are tentative signs of recovery (maybe) but no traffic increase. 1001-hairstyles.com has been there or there abouts for 10 years for the keyword hairstyles and hair styles until April. A site ourlipsaresealed.skyblogs.be has 30000 links to it (there are only 40000 total) with the anchor text haarstijls which is dutch for hairstyles, I don't think its malicious just they set a template and do a new page every day and they also link in the same way to a competitor who wasn't affected. An seo firm have been working on this one for a few months, the traffic increased 50% a couple of weeks ago but bombed the day after to worse than before. Prom-hairstyles.org when the same way as above in April. The only back link oddity is a site polyvore.com links to it about 400 times (out of 1000 or so total) they are using our pictures to sell their prom dresses (with out permission) but mostly deep link. Most of the other sites went in a similar way but have no obvious backlink anomalies. Do I use the link disavowel tool? I am a bit wary of it because if you watch matt cutts video he keeps reiterating that the tool is for people who have used dodgy link practises in the past and want to do a clean up but that isn't me so am I owning up to something I haven't done by using it? Are the search results as strange in everybody's niche? In mine there is some real dross as well as loads of pinterest and other user generated stuff. Sorry to go on for so long and thanks for getting this far. Ian
White Hat / Black Hat SEO | | jwdl0