Recovering from Blocked Pages Debaucle

[TheCraig]

Hi, per this thread: http://www.seomoz.org/q/800-000-pages-blocked-by-robots We had a huge number of pages blocked by robots.txt by some dynamic file that must have integrated with our CMS somehow. In just a few weeks hundreds of thousands of pages were "blocked." This number is now going down, but instead of by the hundreds of thousands, it is going down by the hundreds and very sloooooowwwwllly. So, we really need to speed up this process. We have our sitemap we will re-submit, but I have a few questions related to it: Previously the sitemap had the <lastmod>tag set to the original date of the page. So, all of these pages have been changed since then. Any harm in doing a mass change of the <lastmod>field? It would be an accurate reflection, but I don't want it to be caught by some spam catcher. The easy thing to do would be to just set that date to now, but then they would all have the same date. Any other tips on how to get these pages "unblocked" faster? Thanks! Craig</lastmod></lastmod>

[TheCraig]

Hey Dan,

I am actually not so concerned about the pages being indexed.  I don't really think they were ever de-indexed.  Unless I am wrong, I think they were de-ranked.

I know others have said that when they "disallowed" large portions of their sites, their pages dropped in the rankings, and did not necessarily disappear.  This is more what I want to see recovery from.

Thanks!

Craig

[evolvingSEO]

Craig

D'you have screaming frog? BEST way to make sure you're all set is - run a crawl with Screaming Frog. By default it will acknowledge robots.txt and not crawl anything being blocked. Set the user agent to Googlebot.

If it crawls all the pages you want it to just fine, than you are all set!

-Dan

[TheCraig]

Thanks for jumping in Dan.  The number of blocked pages, over a month later is still way up there.  It really has barely gone done.  As of today it is at 904,000.

So, we still wait and hope that:

A.  That many pages aren't actually blocked (whatever blocked actually means.)

B.  The rate at which that number falls will begin to increase.

Thanks for your answer!

Craig

[evolvingSEO]

Hey There

I see this question is a bit old ... are you still have these issues? If so, when you say "going down" do you mean according to the numbers showing in Webmaster Tools?

I do know that quite often there can be a delay in the data in Webmaster Tools (especially the indexation report which you may be referring to).

I don't think there's any harm in updating the dates to reflect the most recent version of the page, so long as they are accurate.

Let me know if that helps or if you're all set.

-Dan