Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to handle outdated, unoptimized blog posts receiving little or zero traffic?
I'm doing some overdue spring cleaning on our WP blog. Some big visual updates are in the works, but currently I'm working on pruning and updating some poopy and outdated content. Many of the older posts weren't written with SEO in mind and were posted to the blog merely as an extension of our monthly enewsletter. Here's an example: http://pq.systems/2FkQyVG This post needs a lot of work to meet our new standards. The content is thin, readability is weak, kw targetting is non-existant, the visuals suck, zero links, and the charting software mentioned has since been replaced with another solution that we are currently promoting. There are quite a few other posts with similar issues... Any thoughts on the best way to handle these posts? From poking around similar Q&A threads, it seems my options are: Create new updated post, remove old post, 301 redirect from old to new Create new updated post, add blurb & link pointing to new post at the top of old post Edit/update old post, add "This post was updated....etc" blurb to top of old post Any other options or opinions on which solution I should go with would be much appreciated!
On-Page Optimization | | SamKlep0 -
Stagnant Traffic
The traffic on my site (http://www.tbreak.com) has been stagnant over the last few months. We're a news posting site and posting a good 4-8 posts per day and using Yoast plugin to make sure they are optimized, but traffic has not grown at all. What could be the reason for that?
On-Page Optimization | | tbreak.ae0 -
Can I have schema.org links as relative on my site? Getting an html validation error.
I'm getting an html validation error on relative schema.org links "Bad value //schema.org/Organization for attribute itemtype on element div: The string //schema.org/Organization is not an absolute URL." This is my code for https site: <code class="input">e itemtype="//schema.org/Organization"><a itemprop="url" class="navbar-brand" …<="" code=""></a></code>
On-Page Optimization | | RoxBrock0 -
How to deal with 404 errors from deleted categories?
I was reorganizing my blog and deleted several categories. Of course now Google Webmasters is coming up with a bunch of 404 errors. How do I fix this?
On-Page Optimization | | blogger20130 -
400 error - Phone number link.
I am getting 400 errors for all my pages that have a phone number with a link to Skype etc on click, is this a genuine issue or am I ok? How do I resolve this? Any bright ideas, here is an example of the issue - http://www.arts1.co.uk/5-reasons-to-choose-arts1 There are pages of these and I am not sure what to do? Many Thanks James Grimsey
On-Page Optimization | | jamesgrimsey0 -
Unable to see internal link numbers on Opensiteexplorer - Need help
I'm Anuj, a regular user of SEOMOZ. I need some SEO guidance from SEO experts. I'm trying to optimize a webstore for few keywords. I am facing some issues on SEO I was using https all over the webstore and was advised by the community members to not have https through out the site (Due to various reasons). The internal links were not showing up in opensiteexplorer & Google Webmaster Tools too when the site was with https (They were just showing 1 or 2). After changing the pages from https to http, I'm now able to see all the internal links of my website on GWT. Unfortunately, the internal link count on opensiteexplorer shows a very small fraction when compared to the # of internal links shown on GWT. The link update from Opensiteexplorer was on 27th FEB 2013. I had done the https to http (for all pages) somewhere between 17-24th of JANUARY 2013. I wanted to know if I have missed something as I am unable to see those numbers on Opensiteexplorer or will it take time for opensiteexplorer to show the internal link numbers ?
On-Page Optimization | | Pepperjet0 -
Why are my tags causing duplicate error?
Hi, When I run an SEO moz crawl it is picking up duplicate errors from my tags. Currently I have these set as domain url/tag/name of the tag. Is this incorrect? Is there anything I can do to stop this being picked up as duplicate pages? Thanks, Eliz
On-Page Optimization | | econley0 -
Right way to block google robots from ppc landing pages
What is the right way to completely block seo robots from my adword landing pages? Robots.txt does not work really good for that, as far I know. Adding metatags noindex nofollow on the other side will block adwords robot as well. right? Thank you very much, Serge
On-Page Optimization | | Kotkov0