Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should i be using shortcodes for my my page content.
Hello, I have a question. Sorry if this is been answered before. Recently I decided to do a little face lift to my main website pages. I wanted to make my testimonials more pretty. Found this great plugin for testimonials which creates shortcodes. I love how it looks like, but just realised that when I use images in shortcodes, these are not picked up by search engines 😞 only text is. Image search ability is pretty important for me and I'm not sure if I should stick with my plain design and upload images manually with all alt tags and title tags or there is a way to adjust shortcode so it shows images to search engines. You can see example here. https://a-fotografy.co.uk/maternity-photographer-edinburgh/ Let me know your thoughts guys. Regards, Armands
Web Design | | A_Fotografy1 -
What is your opinion in the use of jquery for a continuous scroll type of page layout?
So, I'm in 2 minds about this; let me start with a bit of background info. Context
Web Design | | ChrisAshton
We have a new client who is in the final days of their new site design and were when they first contacted us. Their design essentially uses 5 pages, each with several pages worth of content on each, separated with the use of jquery. What this means is a user can click a menu item from a drop-down in the nav and be taken directly to that section of content like using internal anchor links as if it were a separate page, or they can click the top-level nav item and scroll through each "sub-page" without having to click other links. Vaguely similar to Google's "How Search Works" page if each sector of that page had it's own URL, only without the heavy design elements and slow load time. In this process, scrolling down to each new "sub-page" changes the URL in the address bar and is treated as a new page as far as referencing the page, adding page titles, meta descriptions, backlinks etc. From my research this also means search engines don't see the entire page, they see each sub-page as their own separate item like a normal site. My Reservations I'm worried about this for several reasons, the largest of them being that you're essentially presenting the user with something different to the search engines. The other big one being that I just don't know if search engines really can render this type of formatting correctly or if there's anything I need to look out for here. Since they're so close to launching their new site, I don't have time to set up a test environment and I'm not going to gamble with a new corporate website but they're also going to be very resistant to the advice of "start the design over, it's too dangerous". The Positives
For this client in particular, the design actually works very well. Each of these long pages is essentially about a different service they offer and the continuous scrolling through the "sub-pages" acts as almost a workflow through the process, covering each step in order. It also looks fantastic, loads quickly and has a very simple nav so the overall user experience is great. Since the majority of my focus in SEO is on UX, this is my confusion. Part of me thinks that obscuring the other content on these pages and only showing each individual "sub-page" to search engines is an obvious no-no, the other part of me feels that this kind of user experience and the reasonable prevalence of AJAX/Paralax etc means search engines should be more capable of understanding what's going on here. Can anyone possibly shed some light on this with either some further reading or first-hand experience?0 -
How to know if a wordpress theme is coded correctly for Seo
Hi, So I am curious if there is a tool to see if a site is coed properly for Google? I am running Avada, a standalone theme, yet I am also using a cache plugin. But when I search my code, its all like on one huge line. So I am curious if there is a way to verify or check if a theme is coded correctly? Thank you
Web Design | | Berner1 -
Is there any negative SEO effect when using Wordpress for your Blog?
I have a site entirely done in html, no CMS used. The blog page however, is wordpress. Wondering if this will effect us negatively in terms of SEO, having the blog that is linked to our site, a wordpress site. My gut is absolutely not, but the questions was asked....what do you think?
Web Design | | cschwartzel0 -
What backup application to use?
I'm just looking for a backup application in Wordpress. I've found BackupWordpress y BackWPup. What of these ap. is the best in terms of performance and functionality? Is there any plugin better?Thanks in advanced.
Web Design | | aalcocer20030 -
Search directory - How to apply robots
Hi. On the site I'm working on, we use a search directory to display our search results. It displays as follows - Mydomain.com/search-results/# With the dynamic search results appearing after the hash tag. Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav. How should I handle this? Thanks.
Web Design | | Blenny0 -
The use of foreign characters and capital letters in URL's?
Hello all, We have 4 language domains for our website, and a number of our Spanish landing pages are written using Spanish characters - most notably: ñ and ó. We have done our research around the web and realised that many of the top competitors for keywords such as Diseño Web (web design) and Aplicaión iPhone (iphone application) DO NOT use these special chacracters in their URL structure. Here is an example of our URL's EX: http://www.twago.es/expert/Diseño-Web/Diseño-Web However when I simply copy paste a URL that contains a special character it is automatically translated and encoded. EX: http://www.twago.es/expert/Aplicación-iPhone/Aplicación-iPhone (When written out long had it appears: http://www.twago.es/expert/Aplicación-iPhone/Aplicación-iPhone My first question is, seeing how the overwhelming majority of website URL's DO NOT contain special characters (and even for Spanish/German characters these are simply written using the standard English latin alphabet) is there a negative effect on our SEO rankings/efforts because we are using special characters? When we write anchor text for backlinks to these pages we USE the special characteristics in the anchor text (so does most other competitors). Does the anchor text have to exactly I know most webbrowsers can understand the special characters, especially when returning search results to users that either type the special characters within their search query (or not). But we seem to think that if we were doing the right thing, then why does everyone else do it differently? My second question is the same, but focusing on the use of Capital letters in our URL structure. NOTE: When we do a broken link check with some link tools (such as xenu) the URL's that contain the special characters in Spanish are marked as "broken". Is this a related issue? Any help anyone could give us would be greatly appreciated! Thanks, David from twago
Web Design | | wdziedzic0 -
What value could you expect from the use of schema.org metadata?
Google , Yahoo and Bing have now teamed up to develop unified Meta data standards for all types of content. I personally see this as the next step to larger world-wide knowledge availability. Why is this a big deal?
Web Design | | andrewwolf
If everything comes together, and every page had perfect markup, the answers to users questions could be answered and validated against the information found on every website that had that answer. Example: Who is the author of "The catcher in the Rye" 11345 Websites Result : J. D. Salinger
301: Websites Resulted : Jake Salinger Best Result: J.D. Salinger -> Contact J.D Salinger -> Bio etc. Information would become the link , user's navigation intent becoming the Anchors. Here is the URL of the different types of metadata schema's found on google's new schema.org http://schema.org/docs/schemas.html Creative works: CreativeWork, Book, Movie, MusicRecording, Recipe, TVSeries ... Embedded non-text objects: AudioObject, ImageObject, VideoObject Event Organization Person Place, LocalBusiness, Restaurant ... Product, Offer, AggregateOffer Review, AggregateRating What are the benefits of the different categories of metadata being used? Can anyone site specific case studies done on the enhanced SERP results shown as a result of this metadata?0