Is googlebot the slowest bot?
-
This morning, I wrote a breaking news story about a "Wolf of Wall Street"
It was published at 12:05:49
Googlebot, which used to be on my site within a minute or less, didn't bother to visit for 53 minutes. And now, 32 minutes later, even though it has been crawled, this story doesn't even show up in google search.
Except that it is in the top 10 stories today, at #2, so the headline appears in every page on the site, so every page that has been crawled today, around 10 minutes after it was published, contains that text, so they show up. EINnews, which also crawls our pages is listed for the headline text.
Finally, the page turns up in search results 4 hours later, and says that it is 4 hours old.
Does anyone else see this slow motion mode?
If you do see this, what is wrong with the site that causes this recalcitrant behavior?
The headline of the story is "A 'Wolf of Wall Street' Raided By FBI In Florida"
and the link is http://shar.es/1bW5Sw
-
Your gap will disappear if you get back into the News index. Best of luck!
-
Thank you for looking Ryan.
Google ignores out news map because they removed us from new for a reason they didn't disclose, about 2 years ago.
So I haven't been checking on that map.
I've been trying to find time to change over from my custom-built CMS to Wordpress, and thought I'd reapply after I did that, but I'm 6 months behind my schedule to get that done. (had problems with the page design and the data conversion)
Yes, we're much smaller than the others, but 4hours for a page to show up in the index must mean something else is going on, and I can't work out what that could be.
I'll see if I can get my redesign back on track, and that will make the site more mobile-friendly.
Have you seen anything like that 4 hour gap before? I will track the next few stories I publish too, and report back.
-
Hi Alan,
Are you still pushing Google News tagged XML sitemaps when publishing articles as well? Looking at the ones currently on your site I don't see any new ones referenced since October 2014. And it looks like there's a lot of current mapping that could be updated.In general the site seems a little low in the loop of the major news cycle and would have a lower crawl/index priority on big stories behind the CNNs, Foxs, and Yahoos of the world.
It also doesn't seem to be in the Google News index: https://encrypted.google.com/search?hl=en&q=site%3Anewsblaze.com#hl=en&tbm=nws&q=wolf+of+wall+street+site:newsblaze.com
Google's Guideline to Google News Publisher inclusion is straightforward, and fairly thorough. https://support.google.com/news/publisher/answer/40787?hl=en If you get included via those means, you should see your news articles appearing very quickly.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO: High intent organic revenue down in Europe
Our team is stumped and we are hoping some of you might have some insight! We are seeing a drop in Europe organic revenue and we can't seem to figure out what the core cause of the problem is. What's interesting, the high intent traffic is increasing across the business, as is organic-attributed revenue. And in Europe specifically, other channels appear to be doing just fine. This seems to be a Europe high-intent SEO problem. What we have established: Revenue was at a peak in Q4 2017 and Q1 2018 Revenue dips in mid-late Q2 2018 and again in Q4 2018 where it has stayed low since Organic traffic has gone up, conversion rate has gone down, purchases have gone down Paid search traffic has gone up, conversion rate has gone down slightly, submissions have gone up Currency changes are minimal We cannot find any site load issues What we know happened during this time frame (January 2018 onward): Updates to the website (homepage layout, some text changes) end of April 2018 GDPR end of May 2018 Google Analytics stops being able to track Firefox Europe is a key market for us and we cant figure out what might be causing this to happen - again, only in Europe - beyond GDPR and the changes we've made on our site is there anything else major that we're missing that could be causing this? Or does anyone have any insights as to where we should look? Thank you in advance!
Algorithm Updates | | RS-Marketing0 -
International Homepage Advice
Hello, colleagues! We have a conundrum. A client website has a good subdirectory strategy for localized/translated content for its various international markets, but nothing currently "lives" at the root. In my mind, this presents a challenge to search engines (note that we have had some trouble getting proper visibility overall, which is why I'm asking this question). I'm looking for any links or just plain old good advice on why it's important to have a global homepage. Should that global homepage be in English? Most enterprise sites I've worked with do have a homepage that's in English, with the ability to select a country from a drop down in a nav across the site. Any advice, best practices, etc. about why a global homepage is important and what language it could/should be in would be really helpful. Hreflang tags would make sense, I guess, but each country has slightly different offerings so I'm not sure that it makes complete sense. In other words, one country's homepage may have completely different content than another's. Thank you!
Algorithm Updates | | SimpleSearch0 -
Our Sites Organic Traffic Went Down Significantly After The June Core Algorithm Update, What Can I Do?
After the June Core Algorithim Update, the site suffered a loss of about 30-35% of traffic. My suggestions to try to get traffic back up have been to add metadata (since the majority of our content is lacking it), as well ask linking if possible, adding keywords to alt images, expanding and adding content as it's thin content wise. I know that from a technical standpoint there are a lot of fixes we can implement, but I do not want to suggest anything as we are onboarding an SEO agency soon. Last week, I saw that traffic for the site went back to "normal" for one day and then saw a dip of 30% the next day. Despite my efforts, traffic has been up and down, but the majority of organic traffic has dipped overall this month. I have been told by my company that I am not doing a good job of getting numbers back up, and have been given a warning stating that I need to increase traffic by 25% by the end of the month and keep it steady, or else. Does anyone have any suggestions? Is it realistic and/or possible to reach that goal?
Algorithm Updates | | NBJ_SM2 -
Bot Crawling issues
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site. Some more examples are shown below with other websites: cache:www.vastuwebsite.com ("NOT" showing when Google visited this site) cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site) cache:www.shubhavaastu.com ("NOT" showing when Google visited this site) cache:www.subhavastu.com (Showing when Google visited this site) To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404. what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods. If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google. Require experts analyzation on this doubt. Thanks in Advance.
Algorithm Updates | | SubhaVaastu0 -
What does it mean to build a 'good' website.
Hi guys. I've heard a lot of SEO professionals, Google, (and Rand in a couple of whiteboard Friday's) say it's really important to build a 'good' website if you want to rank well. What does this mean in more practical terms? (Context... I've found some sites rank much better than they 'should' do based on the competition. However, when I built my own site (well-optimised (on-page) based on thorough keyword research) it was nowhere to be found (not even top 50 after I'd 'matched' the backlink profile of others on page 1). I can only put this down to there being 'good quality website' signals lacking in the latter example. I'm not a web developer so the website was the pretty basic WordPress site.)
Algorithm Updates | | isaac6630 -
Is anyone else's ranking jumping?
Rankings have been jumping across 3 of our websites since about 24 October. Is anyone seeing similar? For example ... jumps from position 5 to 20 on one day, then back to 5 for 3 days and then back to 20 for a day I'm trying to figure out if it's algorithm based or if my rank checker has gone mad. I can't replicate the same results if I search incognito or in a new browser, everything always looks stable in the SERPs if I do the search myself
Algorithm Updates | | Marketing_Today0 -
Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem. To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board. Context that might be helpful: I did check technical settings, and the sites are crawlable. The sites do appear in search but seem to be losing organic search traffic. Thanks for any help you can provide!
Algorithm Updates | | geodigitalmarketing0 -
Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level? BT
Algorithm Updates | | achituv0