I'm pulling my hair out trying to figure out why google stopped crawling.. any help is appreciated
-
This is going to be kind of long, simply because there is a background to the domain name that is not typical to anybody in the world really and I'm not sure if its possible that it was penalized or ranked lower because of that or not. Because of that I'm going to include it with the hopes that giving the full picture some nice soul in the world who has more knowledge in this than me see's something or knows something and can point me in the right direction.
Our site has been around for a few years, at one point the domain was seized by homeland security ICE, and then they had to give it back in Dec. which sparked a lot of the SOPA PIPA stuff and we became the poster child so to speak. The site had previously been up since 2008, but due to that whole mess the site was down for 13 months on the dreaded seized server with a scary warning graphic and site title which caused quite obviously a bunch of 404 errors and who knows what else damage to anything we'd had before that as far as page rank and incoming links. we had a lot of incoming links from high quality sites. We were advised upon getting the domain back to pretty much scrap all the old content that was on the site prior and just start fresh.. which we did. Googlebot started crawling slowly, but then as we started getting back into the swing of things people started linking to us,some with high page rank, we were getting indexed quite frequently and ranking high on search results in our niche.. Then something happened on March 4th, we had arguably our best day with google traffic, we'd been linked back by places like Huff Post etc for content in our niche.. and the next day literally it was a freefall. Darn near nothing. I've attached a screen shot from webmaster tools so you can see how drastic it was.
I went crazy, trying to figure out what was wrong, searching obsessively through webmaster tools looking for any indication of a problem, searched the site on google site:dajaz1.com and what comes up is page 2 page 3 page 45 page 46. It's also taken to indexing our category and tag pages and even our search pages. I've now set those all to noindex follow but when I look at where the googlebots are at on the site, they're on the categories, pages, author pages, and tags. Some of our links are still getting indexed, but doing a search just of our site name and we're ranking below many of the media sites that have written about our legal issues, when a month ago we were at least top result for our own name. I've racked my brain trying to figure out the issue. I've disabled plugins, I'm on fetch as google bot all the time making sure our stuff is at least coming out as 200 (we had 2 days where we were getting 403 errors due to a super-cache issue, but once fixed googlebot returned like it never left) I've literally watched 1000 videos, read 100 forums, added in SEO plugins, tried to optimize the site to the point I'm worried I'm over doing it.. and still they've barely begun to crawl. As you can see there is some activity in the last 2-3 days, but even submitting a new site map once I changed the theme out of desperation it's only indexed 16.
I've looked for errors all through webmaster tools and I can't find anything to tell me why that happened, how to fix it, and how to get googlebot to like us again. I'm pulling my hair out here. The links we have incoming are high quality links like huffington post , spin, complex, etc. Those haven't slowed down at all, we do outgoing links to sites we trust and are high quality as well. I've got interns working on how they're writing titles and such, I've gone through and attempted to fix duplicate pages and titles.. I've been going through and re-writing meta description tags What am I missing? I'm pulling my hair out trying to figure out what the issue is. Eternally grateful for any help provided. jnzb6.png
-
Glad to hear there's at least some progress. So often in these cases, I'm seeing that there are multiple things going on, and all you can do is fix them one at a time until the situation gets clearer. Give it a little time and then make the next set of changes - hopefully, it's all cumulative. Just make sure you measure progress and give things time to work. Where I see the most problems is when someone makes a change, it doesn't work in 3-4 days and then they reverse it or pile on new changes. Half the time, Google hasn't even re-cached 90% of the pages at that point.
-
Well google made a strong return yesterday.. so that definitely helped. Not sure if they're indexing us (though my sitemap no longer says 16 indexed and now 2000 indexed. And still not positive how many of those are pages and categories, but it looks like some of the posts are starting to make it under site:dajaz1.com
Thank you so much for your help. I'm not sure the issue is fixed ( I currently have all SEO plugins disabled and I'm too scared to turn them on) but at least google is crawling again and the next few days should tell the tale.
Another thing I did was got rid of the sub categories and turned them into tags. Hopefully this will help with the site structure, even if we take an initial hit with 404's at this point I don't want the tags indexed anyway since I want google to focus on the posts.
-
Sorry - my browser auto-completed the "/350" paginated URL. That change looks good, actually - it'll simplify the crawl path and should prevent those weird, empty results pages. Not sure why those are happening (and you may still need to fix something), but sticking to "Older" / "Newer" should help. I doubt this is the whole problem, but it's definitely an issue.
-
SORRY: SEE NEXT COMMENT
The current and prev/next link are just showing up as blank for me now (no articles). Something's off in the database, I suspect. It's almost like the "newest" articles are blank.
-
I removed the pagi navigation and just left it to the older and newer links to take care of that issue. Lets see if that works at all. I removed all the drop down navigation, though to be honest I'd added that in after this all started hoping to set some type of site structure.. which I clearly didn't do a good job of.
I'm ok with radical at this point, I just want our posts to start getting indexed again. I really appreciate the suggestions..
-
The category/tag issue is tricky - sometimes, it does make sense to NOINDEX/FOLLOW, if those categories and tags spin out into a ton of internal search pages. My worry is that you're noindex'ing main navigation links - which is really a mixed signal. Those pages are important enough to get top billing, but not to index (?)
I have a feeling the best answer is a balance, but that takes a really deep knowledge of the site to sort out. If it's only been a week, it's really tough to tell. On a big site and deeper pages, it can take weeks to sort out, I'm afraid to say. NOINDEX is far from instantaneous.
Your main categories could potentially get ranked over your blog posts in some cases, but again, that's partially the site architecture. If you don't want them to rank, why are they main navigation links? Another, more radical options would be to kill the drop-down menus and add a layer (each main-nav home page would have left-side links or something to that effect) - that would drive more link-juice to the main sections and articles, and less to the sub-categories.
There's something very wrong with your article/blog navigation. If I click the ">>", I jump to page 417, but then there are dozens of empty pages. This is part of why all those paginated versions are getting indexed. I don't know if this is a WP issue or a quirk of your implementation (maybe a bunch of blank entries in the database), but it's creating dozens or hundreds of empty pages for Google to crawl. Since that's available right from the home-page, it's definitely a problem.
-
Thanks for the response. Honestly I don't know why it's indexing the pages like that. I was told to make the categories and tags noindex, follow because google is indexing us, but it's indexing our pages, categories and tags. We've even had search results pages show up.. But it doesn't seem to like our actual posts.
I've turned the wp seo plugin off because honestly I'm just confused and not sure what to do. The site was innocent, so I would think that it wouldn't be penalized for it. It was (and still sorta is) a case that's mainstream media and they are fully aware of the situation all the way up the food chain at google. We were very active in the anti SOPA/PIPA and used by companies like google and the eff as an example of why it was a bad law. I can't see google penalizing us for that.
searching site:dajaz1.com none of our blog posts come up. I don't understand why it's not ranking them, but it will rank our pages with numbers behind them, or categories and tags nor do I understand how to fix it.
we've been noindex,follow on the categories, tagged, pages, search, and author pages for about a week now, and it's not indexing new stuff, and it's still not indexing our actual blog posts.
So if II should put them to index our categories in the nav bar, what's to stop them from indexing them over our blog posts.. because our posts are not ranking at all. (but our category pages are? )
That scroller at the top I think you're talking about is a gawd awful Nike Ad. We're on a ad network so we're stuck with it.. however that occurred after all this ranking stuff started.
I'm confused about the site speed, it loads very quickly for me, scores a 86/100 on google page test, 87/100 on webpagetest.org.. but yet shows slow load in analytics, webmaster tools, and occasionally people will tell me it loads slow for them. Unless it's that ad, in which case there's not much I can do about it. We're in a contract with the ad network (which is very a reputable and well known one)
-
Unfortunately, it can be very difficult to separate a bad history from a penalty from a large-scale technical problem, especially on large sites. I've seen many people assume they got hit by Panda when it was really a link-based penalty, and vise-versa. The site's history makes this go from difficult to nearly impossible, at least without a very deep dive, but I'll see what I can see.
Alan's right on one thing - Google Webmaster Tools has huge gaps in what they warn you about, and it's typically only manual penalties. Many sites have massive problems that never trigger a warning from Google.
I notice that you're NOINDEX'ing even high-level pages (in the navigation), such as:
http://dajaz1.com/music/alternative/
That seems like a bad message to Google - if it's important enough to appear in navigation, it's important enough to index. That's a pretty extreme culling of pages.
The paginated content is a bit of a mess, such as:
In some cases, these don't even seem to return any results, so I'm not sure how they got crawled in the first place. The trick with META NOINDEX here is that, until Google re-crawls, they won't process the tag. This gets tricky, but I'd recommend a couple of possibilities:
(1) If the page returns no results, 301-redirect to the last page of search that has results.
(2) If none of these pages have search value, you could block "/page" as a folder in Google Webmaster Tools. This is a bit dangerous, so I'd want to make sure none of these pages had search value.
Are you getting any page load (speed) warning? Hitting your site intially is massive - about 4MB, by my count, with a ton of JavaScript, most of which just fuels the top, rotating images (which are loading very slowly on my machine). It seems like overkill, both from an SEO and usability standpoint, and is probably in your way to recovery. I'd seriously consider stripping down the size of the code and pruning back some of the active elements for a while.
If you can re-open the important paths, get rid of the thin content (this is going to be complicated and probably involves multiple steps), and speed up the site, you'll know enough to see if this is a technical issue (such as Panda).
There is certainly weak ranking even on your indexed pages, which could indicate a penalty, but it's really tough to tell. Too much of your content is competitive or uses shared phrases or videos, so it's hard to see whether a search for:
"Dwayne Wade 70-Foot Buzzer Beater" ...has you in 6th place for competitive reasons or because your site has been devalued. I don't think it's a penalty, at least in this case. It's a YouTube video and there are other, similar videos for a fairly recent, competitive term, so this may be an accurate ranking (in Google's POV).
The history is a lot tougher, and Q&A just isn't adequate to comment on a situation that complicated, as there are not only SEO but legal ramification. Honestly, I'd have to know a lot more details on that. If you suspect the history has hit you permanently, there may come a time when you have to completely re-brand and re-launch under a new domain.
I suspect, though, that cleaning up the crawl problems, removing the thinnest content, speeding up the site, and generally fixing some technical issues could help quite a bit. It's going to be a difficult process, though. The thing about changes like the Panda Update, is that it's not just one factor. I can't point to one thing and say "fix this" - you have to aggressively attack multiple factors, since Google is wrapping multiple signals into Panda and won't tell you which one is the problem.
I should say that I'm not saying this is Panda, but that it's a Panda-like situation - you've got a lot of crawl/index issues that are going to cause you problems. The question is whether those are compounded by your history (and, unfortunately, they probably are). The combination means that you have to be even more aggressive with the clean-up.
-
well, I'm sorry I wasted your time, like you, I'm looking for answers, and thats all. Don't worry, I won't respond to any of your threads again, I did the best I could.
If you look back through the things I said, you may find something useful. Don't underestimate your own powers of investigation and testing.
I can almost guarantee you this - there is no simple answer. Nobody has it, other than google. The only way to work out what it is - is to ask, listen, experiment and test. There are a lot of smart people in SEOMoz, and if they haven't responded, it is because they don't have "the answer" you want.
And just remember, nobody responding to you here, if they do respond, is making any money from giving you suggestions, nor are they trying to wind you up. This is a voluntary thing of people helping people. The only possible reward is a thumbs up on a post, but I see I didn't give you anything that was even remotely useful.
I hope you figure it out.
-
When I post something on the internet, freaked out or not, that's never an invite to go looking for my personal contact information to contact me outside of this medium.
Anyway.. thanks for your help.. My situation might not be like yours, I was looking for actual advice tips and knowledge on things that might really be wrong. What I got was google hates your guts, but hey I'm in the same boat maybe they just manually told you to screw off.
How does that help me in any way when what I'm looking for are things on my site that I can use to improve my situation. I know you're well meaning, but you're not giving me any answers here. I'm trying to improve the SEO on my site.. not get phone calls from guys telling me their story in the middle of the night because I made a mistake in putting my url. I'll ask my question in another thread elsewhere.
Thanks
-
it was the only number I could find - and you seemed to be pretty well freaked out about the situation.
I am in Australia right now, and I didn't know you were on the east coast.
Since the timezone changed, I messed up the time calculation, sorry.
-
Melissa.
If anyone knew the answer, they would be going around selling the ability to fix panda and other problems, but I don't think there is anyone outside of google who knows the answer. We are on our own. The same thing that happened to you happened to us over 12 months ago and I'm still struggling to find a cure.
Working on a drastic change next week, but that change will not help you - your site is much too small for this change.
Not a lot of this makes any sense.
Linking to other sites - so is everyone else and they are not affected.
short content - so are many others and we're not hearing from them
some racy content - maybe, but look at all the others.
You have many incoming links from big sites. Maybe their algorithm is so paranoid about new links that it has hit you.
Here are just about my final ideas for you, after this, I can't think of anything.
1. Do you have any copyright content that someone could have reported you for?
2. Are you linking out to any bad sites - check all of your links - including the ones in your footers and sidebars.
3. check your pages with "fetch as googlebot" in case they can't find what they want
4. Check to see if you've been linkbombed with a ton of bad links, by someone who doesn't like you.
5. Register your writers with a google profile and link from their profile on their site to their google profile
You said you checked WMT many times, so you didn't get an unnatural links message
Two possible reasons nobody else has answered - they don't have an answer - or maybe they are too frightened to connect with you, because of the takedown.
-
Yeah well, I prefer to think it's not that since that would be a pretty big deal not to mention there is nothing going on on that page that would be worthy of a penalty... having said that I'm pretty disappointed with Seomoz in that nobody else is responding to this. certainly not worth the money per month..
And the phone call from you at 130am to the number listed on our DMCA page was really REALLY creepy..
-
Melissa, I doubt you'd get anything in webmaster tools.
That is because it would be a manual penalty, and I have no idea whether they would admit to it or not.
-
Anyone? I've posted this here and on google's webmaster forums and so far all the responses I've gotten are that google has pretty much told me to screw off and marveling at how drastic a fall it was.. but nobody can give me any real indication as to why. It's very frustrating.. granted we don't write big long drawn out paragraphs on a lot of our content, and it's content elsewhere, we're usually first or close with a lot of things, and we've earned enough respect by being quick with the exclusives that we're linked in Huff Post's from around the web, linked to as a source from Spin and MTV... we ranked high on things like the superbowl, whitney houstons death.. so the sudden fall is shocking as we're certainly not a scrape copy and paste site.. We get scraped and copied and pasted a lot.
-
Well considering that the domain was seized for 13 months by the government and then when they had to give the domain back the timing was such that it kicked the ball rolling on a internet uproar that cost them 2 bills. I'd say we've got some enemies..
But in that case wouldn't we get some sort of notification from Google in webmaster tools? There is nothing showing there..
-
It is possible that one or more people, or a group, has reported you for something.
Some time ago, someone emailed us to say they had reported us to google for something. I don't know what it was, but about a week later, our ranking dropped.
When you have two million pages, its a bit hard to work out what might have caused it.
-
exactly.. I need to know what is causing it.
-
That is part of the problem - other sites ranking above you.
But thats not the reason it is happening.
The reason is something else - this is just the fallout from that.
I was just searching for one of our stories that I wrote myself.
Everyone that links to us - and even sites that copied the story, with my byline on it - they are all ranking, but we're not.
Its killing us.
-
Actually we do get original stuff, and post exclusives all the time. The problem is there are a lot of blogs who then turn around and repost the same thing we post.. Not that they don't link back or credit, because the bigger and reputable blogs like Spin and MTV etc do.. it's the trillion of blogs that copy and paste blog.
As for the length of content we're working on that, but with the type of blog we have it's often difficult to write out paragraphs on a freestyle that isn't attached to any project.
I need to figure out what the issue is.. it's going to make me crazy until I do.
-
Melissa,
That is a dramatic drop, much worse than ours was.
The issue of google indexing index pages and showing them and tag pages in SERPS, but not the original, is typical.
For some reason I don't see yet, google hates your guts.
Your stories are showing up in Australian results, but back a page or two.
If you search for
Drake Announces Second Leg of Club Paradise Tour
you won't find it until you get to the last result and then click the link that says they will show you the results they hid.
This means they hate something on your site and until you fix it, you are out of luck.
I know, because I've been trying to fix this same thing for more than 12 months.
From looking at just a few of your pages, it appears that you have nothing that is 100% original and unique to your site. That probably means they see you as an aggregator or content farm.
I could be wrong, but thats what it looks like.
Show me something that is 100% unique to you.
Some of your content is very thin, too, 20-30 words and a video.
That probably makes your content look like it is mostly duplicate, with just a few words changed.
Your sitemap looks OK.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Mobile site crawl returns poorer results on 100% responsive site
Has anyone experienced an issue where Google Mobile site crawl returns poorer results than their Desktop site crawl on a 100% responsive website that passes all Google Mobile tests?
Intermediate & Advanced SEO | | MFCommunications0 -
Should I redirect a domain we control but which has been labeled 'toxic' or just shut it down?
Hi Mozzers: We recently launched a site for a client which involved bringing in and redirecting content which formerly had been hosted on different domains. One of these domains still existed and we have yet to bring over the content from it. It has also been flagged as a suspicious/toxic backlink source to our new domain. Would I be wise to redirect this old domain or should I just shut it down? None of the pages seem to have particular equity as link sources. Part of me is asking myself 'Why would we redirect a domain deemed toxic, why not just shut it down.' Thanks in advance, dave
Intermediate & Advanced SEO | | Daaveey0 -
Prevent Google from crawling Ajax
With Google figuring out how to make Ajax and JS more searchable/indexable, I am curious on thoughts or techniques to prevent this. Here's my Situation, we have a page that we do not ever want to be indexed/crawled or other. Currently we have the nofollow/noindex command, but due to technical changes for our site the method in which this information is being implemented if it is ever displayed it will not have the ability to block the content from search. It is also the decision of the business to not list the file in robots.txt due to the sensitivity of the content. Basically, this content doesn't exist unless something super important happens, and even if something super important happens, we do not want Google to know of its existence. Since the Dev team is planning on using Ajax/JS to pull in this content if the business turns it on, the concern is that it will be on the homepage and Google could index it. So the questions that I was asked; if Google can/does index, how long would that piece of content potentially appear in the SERPs? Can we block Google from caring about and indexing this section of content on the homepage? Sorry for the vagueness of this question, it's very sensitive in nature and I am trying to avoid too many specifics. I am able to discuss this in a more private way if necessary. Thanks!
Intermediate & Advanced SEO | | Shawn_Huber0 -
Establishing if links are 'nofollow'
Wonder if any of you guys can tell me if there is any other way to tell google links are nofollow other than in the html (ie can you tell google to nofollow every link in a subdomain or something). I'm trying to establish if a couple of links on a very high ranking site are passing me pagerank or not without asking them directly and looking silly! Within the source code for the page they are NOT tagged as nofollow at present. Hope that all makes sense 😉
Intermediate & Advanced SEO | | mat20150 -
Is ok to add 'no follow' to every outbound link?
How do you handle outbound links from your site?.. do you no follow them all to be on the safe side?
Intermediate & Advanced SEO | | nick-name1230 -
Google Crawl Rate and Cached version - not updated yet :(
Hi, Ive noticed that Google is not recognizing/crawling the latest changes on pages in my site - last update when viewing Cached version in Google Results is over 2 months ago. So, do I Fetch as Googlebot to force an update? Or do I remove the page's cached version in GWT remove urls? Thanks, B
Intermediate & Advanced SEO | | bjs20100 -
Why is google automatically showing my competitor's result even when customer types in our brand name on the query?
This is little weird. We run a website specific to mobile phones called as 91mobiles.com. The site has gained lot of user interest and trust in the last 2 years [pre-dominantly indian users]. Lot of our users type mobile phone model and 91mobiles as the query. Example "Sony Xperia P 91mobiles". Google is showing gsmarena.com results on top and then shows our own results below! What's annoying is the fact that google also bolds the term gsmarena [denoting that it's a synonym]. Any idea why this is happening? We are very sure that we are not doing anything wrong..We have worked really hard for the last 2 years to reach where we are..and it's kind of hard to see gsmarena siphoning away our traffic for no reason at all [even when customer types in 91mobiles a part of the query to quality it]...Can some experts here demystify this? Thanks.
Intermediate & Advanced SEO | | Gaadi0 -
Google +one button - help needed
Can someone from this wonderful community answer the question. The link is http://www.seomoz.org/q/google-one-button-help-needed-2
Intermediate & Advanced SEO | | seoug_20050