Robots.txt was set to disallow for 14 days
-
We updated our website and accidentally overwrote our robots file with a version that prevented crawling ( "Disallow: /") We realized the issue 14 days later and replaced after our organic visits began to drop significantly and we quickly replace the robots file with the correct version to begin crawling again. With the impact to our organic visits, we have a few and any help would be greatly appreciated -
Will the site get back to its original status/ranking ?
If so .. how long would that take?
Is there anything we can do to speed up the process ?
Thanks
-
Thank you for the response.
We have been watching over the past week and there has been a very small change in the number of indexed urls in GSC and no change in the stats on the MOZ dashboard.
Is that normal? How often does MOZ update the stats?
-
This is commonly done intentionally, when launching a site on a new domain. Once the disallow is removed, the general practice is to request reindexing of the root domain page (and possibly some key pages with paths not likely to be found through navigation) in GSC, and also submitting (or re-submitting) your sitemaps directly in GSC (even though they also may/should be in your robots.txt file).
I'm not sure how long you can expect the search engines to take, since your situation is a bit unique where the site was indexed, and then disallowed temporarily. Just guessing based on launching brand new domains, the process should be quick to be indexed (perhaps a few days) but might be slower on regaining previous ranking positions (unsure on timing of this).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Large robots.txt file
We're looking at potentially creating a robots.txt with 1450 lines in it. This will remove 100k+ pages from the crawl that are all old pages (I know, the ideal would be to delete/noindex but not viable unfortunately) Now the issue i'm thinking is that a large robots.txt will either stop the robots.txt from being followed or will slow our crawl rate down. Does anybody have any experience with a robots.txt of that size?
Intermediate & Advanced SEO | | ThomasHarvey0 -
URL Parameter Setting Recommendation - Webmaster Tools, Breadcrumbs & 404s
Hi All, We use a parameter called "breadCrumb" to drive the breadcrumbs on our ecommerce product pages that are categorized in multiple places. For example, our "Blue Widget" product may have the following URLs: http://www.oursite.com/item3332/blue-widget
Intermediate & Advanced SEO | | Doug_G
http://www.oursite.com/item3332/blue-widget_?breadCrumb=BrandTree_
http://www.oursite.com/item3332/blue-widget_?breadCrumb=CategoryTree1_
http://www.oursite.com/item3332/blue-widget_?breadCrumb=CategoryTree2_ We use a canonical tag pointing back to the base product URL. The parameter only changes the breadcrumbs. Which of the following, if any, settings would you recommend for such a parameter in GWT: Does this parameter change page content seen by the user? Options: Yes/No
How does this parameter affect page content? Options: Narrows/Specifies/Other Currently, google decided to automatically assign the parameter as "Yes/Other/Let Googlebot Decide" without notifying us. We noticed a drop in rankings around the suspected time of the assignment. Lastly, we have a consistent flow of products that are discontinued that we 404. As a result of the breadcrumb parameter, our 404s increase significantly (one for each path). Would 800 404 crawl errors out of 18k products cause a penalty on a young site? We got an "Increase in '404' pages' email from GWT, shortly after our rankings seemed to drop. Thank you for any advice or suggestions! Doug0 -
YOAST SEO: How to set rel=cannonical tags to the original article post
Hi Mozers, Can anyone tell me how to set the rel="canonical tags via SEO YOAST? I have an article posted on my blog that was published first on another blog and i need to reference this entry somehow, I have been told to use rel="canonical - if so I would appreciate some insight on how to do this exactly! Thanks very much in advance
Intermediate & Advanced SEO | | edward-may0 -
We have set up our H1 to contain the product name - it used to be our Company name all the time - would this affect our sales
We noticed recently on our site - that our H1 tag was the Company Name - we changed this to be the product name - our products would be searched for by all or part of the description. Our sales have dropped of the days since we changed it, could it be a result of this change ? Is it best to have the H1 tag as the product name ?
Intermediate & Advanced SEO | | CostumeD0 -
Google Webmaster Tools Set-up
Hello! My URL of www.morganlindsayphotography.com I have set up in google webmaster tools. I also have added the instance of morganlindsayphotography.com (non www version) My blog is located at www.morganlindsayphotography.com/blog/ My question is do I add a sitemap to the "<a>www.morganlindsayphotography.com/blog/</a>" as well as the <a>www.morganlindsayphotography.com</a> ? Thank you!
Intermediate & Advanced SEO | | 393morgan0 -
How to make Google index your site? (Blocked with robots.txt for a long time)
The problem is the for the long time we had a website m.imones.lt but it was blocked with robots.txt.
Intermediate & Advanced SEO | | FCRMediaLietuva
But after a long time we want Google to index it. We unblocked it 1 week or 8 days ago. But Google still does not recognize it. I type site:m.imones.lt and it says it is still blocked with robots.txt What should be the process to make Google crawl this mobile version faster? Thanks!0 -
Use Canonical or Robots.txt for Map View URL without Backlink Potential
I have a Page X with lots of unique content. This page has a "Map view" option, which displays some of the info from Page X, but a lot is ommitted. Questions: Should I add canonical even though Map View URL does not display a lot of info from Page X or adding to robots.txt or noindex, follow? I don't see any back links coming to Map View URL Should Map View page have unique H1, title tag, meta des?
Intermediate & Advanced SEO | | khi50 -
Disallow my store in robots.txt?
Should I disallow my store directory in robots.txt? Here is the URL: https://www.stdtime.com/store/ Here are my reasons for suggesting this: SEOMOZ finds crawl "errors" in there that I don't care about I don't think I care if the search engines index those pages I only have one product, and it is not an impulse buy My product has a 60 day sales cycle, so price is less important than features
Intermediate & Advanced SEO | | raywhite0