Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will interlinking using dynamic parameters in url help us in increasing our rankings
Hi, Will interlinking our internal pages using dynamic parameters(like abc.com/property-in-noida?source=footer) help us in increasing our rankings for linked pages OR we should use static urls for interlinking Regards
Web Design | | vivekrathore0 -
Moz webfonts do not render correctly ...
Actually I have problems with the rendering of webfonts on all browsers. As moz uses 100% Webfonts I hardly can read text. Anybody had this problem before? please help! THX 🙂 attached a screen to see what it looks like... screenshot-problem-webfonts.png
Web Design | | inlinear0 -
Using a lightbox
Using a lightbox type thing for our new website (you click on an image and then get a bigger version of image). Is there any best practices SEO wise for this and any no no's I should be aware of? It's all hard coded so not going the normal way via wordpress of using thumbnails then the larger original image as you click on the thumbnail, using two separate images- this sound to you guys?
Web Design | | Jon-C0 -
How does using a CMS (i.e. Wordpress/Drupal) affect backlinks and SEO?
So I need to build a website with over 100 pages in it. Elements of the design will probably be moved around and or tested so I need to use a CMS. It's pretty much a review site so while the content will remain static I'd like to employ A/B testing to mess with conversion rates. Wordpress has a plugin for that even. So I'm just wondering, since CMS pages are pretty much created on spot and not retrieved from a library, how this affects backlinks and anchor text? How exactly does the external website point to yours if the URL is dynamically generated? Or am I misunderstanding something? Please recommend any extra resources as well if you can.
Web Design | | seochump0 -
How to best correct cannibalization?
I apologize if this has already been answered, but after reading several posts on cannibalization, I can't seem to find what I am looking for. The site in question is www.urbanitystudios.com and in particular the term "western wedding invitation". We rank in the top 30 for this term in Google, but Google has indexed a particular product, versus our western wedding invitation collection page. The product that is indexed for this term: http://www.urbanitystudios.com/Designs/western-wedding-invitations-p-1527.html The page that we would rather be indexed: http://www.urbanitystudios.com/Designs/western-wedding-invitations-c-95_179_181.html After running an onpage report in SEOmoz tools for the collection page, we recieve an A grade, but get a warning on the cannibalization line item. As you can see, we name each product within that collection as "Western Wedding Invitation-x" (and have done this for other product categories...not good). After a good head slap, we realized that we are confusing Google as to what should be the main page. If we rename our products, the product's URL will change-Do we do a 301 for those products? If we rename our products, do we take out the words "Western Wedding Invitation" entirely or can we say "x-Western Wedding Invitation"? Or. because cannibalization is deemed a "low priority" in the reports, do we let things be and work on getting links to the collections page vs the individual product? Any insight would be most appreciated.
Web Design | | UrbanityStudios0 -
Why or why not use Dot Net Nuke (DNN)?
I have development team pitching my client on using Dot Net Nuke (DNN) for the CMS. I have no experience with DNN and not sure why the developers want to use it. They do want to use Server SQL and may build some software for a different part of the project in .NET. What SEO and UX questions should I ask about using DNN?
Web Design | | Joshen0 -
Is there any difference in using an underscore vs. a dash in the directory portion of the url?
A friend who is a software developer asked this question regarding the directory portion of the url: Is it better to use dashes or underscores? I know in the domain name Matt Cutts recommends dashes, but what about the directory extension?
Web Design | | RobertFisher0 -
Using Wordpress as CMS for large Websites
Is Wordpress good enough to be used as a full fledge CMS for a large website. In particular, I'm talking about a news website. We have been online since 2002 but pretty soon we will have digitized our print newspaper archives of about 60 years. So, my question is, is it OK to use Wordpress for the entire website and if so what are some of the important things that need to be kept in mind. Cheers!
Web Design | | RishadShaikh590