Bot Crawling issues
-
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site.
Some more examples are shown below with other websites:
cache:www.vastuwebsite.com ("NOT" showing when Google visited this site)
cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site)
cache:www.shubhavaastu.com ("NOT" showing when Google visited this site)
cache:www.subhavastu.com (Showing when Google visited this site)
To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404.
what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods.
If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google.
Require experts analyzation on this doubt. Thanks in Advance.
-
Dear SeoMAN,
Thanks for your answer. I just tried at google, its shows normal, I just checked google search console and webmaster tools, there is nothing serious shows. Can it be possible to try to find out what was the main cause.
Is I taken the right image, is I searched exactly what you are asking about?
Many thanks.
-
Dear Vijay, Thank You.
You are an expert in this field, I respect your thought, but I am thinking there should be some other reason, that might cause of this problem. I am MOZ PRO subscriber, if there is, canonical issues, I will get such notifications. Long back, there are some canonical issues, we fix them.
Site is totally free from malware and viruses, I just now also observed Google webmaster tools and checked other ways, there is nothing viruses found in my site.
Can it be possible to find any other issues which caused this problem.
Many thanks.
-
You are asking quite a lot of questions here, basically if your site has been around for a reasonable amount of time and your page is accessible and can be crawled by search engines you should be able to Google your domain name (including the TLD) like "mydomain.com", if that doesn't show up you either have a serious technical error or you have had a penalty.
As for telling whether it is bot or manual Google search console should give you a bit more information. Personally as far as I can see a penalty is a penalty and needs to be overcome one way or another, whether it is automatic or manual is a bit beside the point!
-
It can be due to a lot of SEO/technical issues, Here are a couple of observations/suggestions:
- Your site works with https:// and with http:// both and also with www. and without www. , this might be an issue for duplicate content. Please make sure that you have only one version with or without www. or https://
- Check your website for malware and viruses just as a precaution.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile Usability Issues after Mobile Frist
Hi All A couple months ago we got an email from google, telling us - Mobile-first indexing enabled for https://www.impactsigns.com/ Ran the test on MOZ, Mobile usability shows 100% Last week got an email from google - New Mobile usability issues detected for impactsigns.com Top new issues found, ordered by number of affected pages: Content wider than screen Clickable elements too close together I can not seem to figure out what those issues are, as all content is visible. How important are these 2 issues? Since we are now on the mobile first side?
Algorithm Updates | | samoos0 -
Site:www Issue - Homepage of the website is not showing in Google
Hello everyone, When I have manually search site:www.blinds4uk.co.uk in google.co.uk to know about webpages status, home page of the website is not showing in google search engine result pages. Please let me know, what is the reason behind this? because website crawling and indexing is good. Many Thanks.
Algorithm Updates | | Kuldeep-Sharma0 -
Anyone else notice a global Google cache issue?
Im noticing a huge % of pages on my sites and those picked at random from Google searches, high traffic pages from big sites like booking.com have not been cached since 22nd Oct by Google. Anyone else noticed this or got insight on it? Andy
Algorithm Updates | | AndyMacLean0 -
Struggling with Google Bot Blocks - Please help!
I own a site called www.wheretobuybeauty.com.au After months and months we still have a serious issue with all pages having blocked URLs according to Google Webmaster Tools. The 404 errors are returning a 200 header code according to the email below. Do you agree that the 404.php code should be changed? Can you do that please ? The current state: Google webmaster tools Index Status shows: 26,000 pages indexed 44,000 pages blocked by robots. In late March, we implemented a change recommended by an SEO expert and he provided a new robots.txt file, advised that we should amend sitemap.xml and other changes. We implemented those changes and then setup a re-index of the site by google. The no of blocked URLs eventually reduced in May and June to 1,000 for a few days – but now the problem has rapidly returned. The no of pages that are displayed in a google search request of www.google.com.au where the query was ‘site:wheretobuybeauty.com.au’ is 37,000: This new site has been re-crawled over last 4 weeks. About the site This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster tools robots.txt file has been modified several times: Firstly we had none Then we created one but were advised that it needed to have this current content: User-agent: * Disallow: Sitemap: http://www.wheretobuybeauty.com.au/sitemap.xml
Algorithm Updates | | socialgrowth0 -
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Algorithm Updates | | WebRiverGroup0 -
SERPS Issue. Why is this happening?
All of our pages have been indexed by Google and we are ranking very well thus far. Here is an example of a search where, based on my searches, we rank well. Search for "Indiana ERISA Bond". You can search for any state name followed by "ERISA Bond" and we rank really well (Top of page 1). The pages that should show up for you are here: 1. https://www.suretegrity.com/surety-bonds/indiana/erisa-bond 2. https://www.suretegrity.com/surety-bonds/indiana/erisa My issue is with a different product we offer. Search for "Georgia Mortgage Lender Bond". The pages I would hope to come up, in order of priority, are: 1. https://www.suretegrity.com/surety-bonds/georgia/mortgage-lender 2. https://www.suretegrity.com/surety-bonds/georgia/mortgage-broker-banker-lender When I do my searches I show up at the bottom of page 2 and the URL that is being found is #2 above and not #1. I have spent a LOT of time thinking about this and am in need of some direction. What am I doing wrong? Thanks, Alex
Algorithm Updates | | dbuckles0 -
Large site with faceted navigation using rel=canonical, but Google still has issues
First off, I just wanted to mention I did post this on one other forum so I hope that is not completely against the rules here or anything. Just trying to get an idea from some of the pros at both sources. Hope this is received well. Now for the question..... "Googlebot found an extremely high number of URLs on your site:" Gotta love these messages in GWT. Anyway, I wanted to get some other opinions here so if anyone has experienced something similar or has any recommendations I would love to hear them. First off, the site is very large and utilizes faceted navigation to help visitors sift through results. I have implemented rel=canonical for many months now to have each page url that is created based on the faceted nav filters, push back to the main category page. However, I still get these damn messages from Google every month or so saying that they found too many pages on the site. My main concern obviously is wasting crawler time on all these pages that I am trying to do what they ask in these instances and tell them to ignore and find the content on page x. So at this point I am thinking about possibly using robots.txt file to handle these, but wanted to see what others around here thought before I dive into this arduous task. Plus I am a little ticked off that Google is not following a standard they helped bring to the table. Thanks for those who take the time to respond in advance.
Algorithm Updates | | PeteGregory0 -
Issue: Having to Fight Product Marketing to Use Keywords
This is sort of a "DUH!" moment to me. I know everyone has come across this at some point in time and am interested in hearing how others deal with this. A little background: I was researching keywords for new menus and pages. Sometimes, people (product marketing in my case) do not give me a heads up on changes they want to make to pages and it is always a fight with them to change it. This is pretty normal for me and I am use to it. It is one of those things that they don't want to discuss it with you because they know you are going to critique their work. and, yes, change it for the good of the company. I had a co-worker say to me:
Algorithm Updates | | SmartBear
"We may have to start making [pages] meaningful to the human visitor than satify the bot army". My response was:
"What better way to make it [web page] meaningful to users by knowing which terms they search on the most in our industry? Keyword research is not just for Search Engines, it is actual live data as to what most people are searching. That is why I put such a high precedence on it and report on trends. You can bet that if 100,000 people are searching for [keyword], that is what they want to see when they search for it." Anyways, that is how I handled this particular event. I have several responses when these comments pop up from time to time. Usually it is something to the fact that they are not the ones who will get fired if leads drop via organic search, so we better try this. But today, I was feeling kind of spunky and decided to take another route. What are some of your responses to these types of remarks? Hopefully this will make for a good discussion.0