Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. How long could Google take to crawl/index the new pages and rank the keywords used within those pages?
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. The 3 locations old domains were redirected to their sites within our main brand domain. How long could Google take to crawl/index the new pages and rank the keywords used within those pages? And possibly increase our domain authority hopefully? We didn't want our brand spread out over multiple websites/domains on the internet. This also allowed for more content to be written on pages, per each of our locations service's, as well.
Web Design | | BurgSimpson0 -
Interlinking using Dynamic URLs Versus Static URLs
Hi Guys, Could you kindly help us in choosing best approach out of mentioned below 2 cases. Case. 1 -We are using: We interlink our static pages(www.abc.com/jobs-in-chennai) through footer, navigation & by showing related searches. Self referential Canonical tags have been implemented. Case. 2 -We plan to use: We interlink our Dynamic pages(www.abc.com/jobs-in-chennai?source=footer) through footer, navigation & by showing related searches. Canonical tags have been implemented on dynamic urls pointing to corresponding static urls Query 1. Which one is better & expected to improve rankings. Query 2. Will shifting to Case 2 negatively affect our existing rankings or traffic. Regards
Web Design | | vivekrathore0 -
What do you use for test rendering your dev site?
I'm redesigning our company ecommerce site and need to test render an infinite scroller to ensure that it is as SEO friendly as possible. My problem is that I cannot view it in Webmaster Tools since I am blocking the site from crawlers using robots.txt. I know I could simply unblock Google temporarily but I really would rather not make my dev site available to search engine crawlers.
Web Design | | bearpaw0 -
Should I Use An Animated Javascript Responsive Site
Hi, hope someone might be able to help me with this. I am setting my son up with a website for his small painting and decorating company. However, I am a wordpress stalwart and he has seen a theme which is a javascript animated responsive theme from template monster. Its not my choice just he is adamant that he wants it. However, I am slightly concerned that Google cannot index as well with these kind of sites as they would with a standard HTML site. I would be grateful if someone could confirm if they can be indexed etc. The content appears in what I can only describe as lightboxes. Thanks
Web Design | | denismilton0 -
Using a query string for linked, static landing pages - is this good practice?
My company has a page with links for each of our dozen office locations as well as a clickable map. These offices are also linked in the footer of every page along with their phone number. When one of these links is clicked, the visitor is directed to a static page with a picture of the office, contact information, a short description, and some other information. The URL for these pages is displayed as something like http:/example.com/offices.htm?office_id=123456, with seemingly random ID numbers at the end depending on the office that remain static. I know first off that this is probably bad SEO practice, as the URL should be something like htttp://example.com/offices/springfield/ My question is, why is there a question mark in the page URL? I understand that it represents a query string, but I'm not sure why it's there to begin with. A search query should not required if they are just static landing pages, correct?. Is there any reason at all why they would be queries? Is this an issue that needs to be addressed or does it have little to no impact on SEO?
Web Design | | BD690 -
The impact of using directories without target keyword on our Rankings
Hello all, I have a question regarding a website I am working on. I’ve read a lot of Q en A’s but couldn’t really find the best answer. For one of our new websites we are thinking about the structure of this website and the corresponding URL-structure. Basically we have a main product (and a few main keywords) which should drive the most traffic to our website, and for which we want to optimize our homepage. Besides those main keywords, we have an enormous base of long-tail keywords from which we would like to generate traffic. This means we want to create a lot of specific pages which are optimized. My main question is the following: We are thinking of two options: Option 1: www.example.com/example-keyword-one Option 2: www.example.com/directory/example-keyword-one With option 1 we will link directly from our homepage to the most important pages (which represent our most important keywords). All the pages with the long tail content will be linked from another section on our website, which is one click away from our homepage (specifically a /solutions page which is linked from the footer). All the pages with long-tail content will have this structure www.example.com/example-keyword-one so the URLs will not contain the directory /solutions With option 2 we will use more subdirectories in our URLs. Specifically, for all the long tail content we would use URLs like this: www.example.com/solutions/example-keyword-one
Web Design | | NielsB
The directories we want to use wouldn't really have added value in terms of SEO, since they don’t represent important keywords. So what is the best way to go? Option 1, straightforward, short URL’s which don’t really represent the linking structure of our website, but only contain important keywords. Or option 2, choose for more directories in our URLs which represent the linking structure of our website, but contain directories which don’t represent important keywords. Would the keyword ‘solutions’ in the directory (which doesn’t really relate to the content on the page) have a negative impact on our rankings for that URL?0 -
How to fix and issue with robot.txt ?
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site Oct 26, 2012
Web Design | | skehoe
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. The site has dropped out of Google search.0 -
Correct Canonical Reference
Aloha, This is probably a noob question, but here we go: I got a CMS e-commerce, which does not allow static "rel=canonical" declaration in the header and can only work with third-party modules (xml packages) that append "rel=canonical" to all pages dynamic pages within the URL. As a result, I have pages I'm declaring incomplete rel="canonical" as such: Instead of: rel="canonical" src="www.domainname.com/category.aspx" I get: rel="canonical" src="/category.aspx" Coincidentally (or not), after the implementation of the canonical tag, pages that were continuously increasing in rankings started dropping, and, within a week, disappeared from the index completely. Could the drop be a result of my canonical links pointing to incomplete URLs? If so, by fixing this issue, do I stand a chance of recovering my pages' SERPs?
Web Design | | dimanyc0