Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Benefit to Hosting Site on a Dedicated Server?
Our annual hosting plan expires soon. Our website is hosted on a shared server. Is there an SEO benefit to hosting our site on a dedicated server. Could this result in faster download times which is a ranking factor? Our traffic is currently low (only about 20 visits per day). Thanks!!
Intermediate & Advanced SEO | | Kingalan1
Alan0 -
News articles on our website are being indexed, but not showing up for search queries.
News articles on distributed.com are being indexed by Google, but not showing up for any search queries. In Google Search, I can copy and paste the entire first paragraph of the article, and the listing still won't show up in search results. For example, https://distributed.com/news/dtcc-moves-closer-blockchain-powered-trades doesn't rank AT ALL for "DTCC Moves Closer to Blockchain-Powered Trades", the title of the article. We've tried the following so far: re-submitted sitemap to search console checked manual actions in search console checked for any no-index/no-follow tags Please help us solve this SEO mystery!
Intermediate & Advanced SEO | | BTC_Inc0 -
Pages canonicaled to another appearing before the canonical on google searches
Hello, When I do this google search, this page(amandine roses category) appears before the one it is canonical-ed to(this multi-product version of amandine roses). This happens often with this multi-product template, where they don't rank as well as their category version(that are canonical to the multi-product version). Can someone maybe point us in the right direction on what the issue may be? What can be improved?
Intermediate & Advanced SEO | | globalrose.com0 -
Not appearing when www.domainname.com.au is searched
We are working on an SEO campaign that has an issue, of which we are unsure of the cause. When the domain is searched for in Google as domainname.com.au. The search results show the website just fine. See here for the display of this. http://imgur.com/a/Vcbqs However, when the domain is searched for in Google as www.domainname.com.au. The search results do not display our website. See here to see this issue. http://imgur.com/a/jX692 We are unsure why our website does not appear in Google's listings when the www. is added in search. If anyone can help us here to explain and help us resolve this, that would be much appreciated.
Intermediate & Advanced SEO | | Gavo0 -
What are the search pros and cons of a transcript of a video in scroll box
This is being considered for fairly templated pages as a means of adding unique text to the page, without giving up a ton of real estate. The text is all visible in view source, but only the first few lines in the scroll box. The box itself is just above the fold. How, do you think this would affect search? Any other ideas for handling this. I'm limited by what the client will accept. Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Block Googlebot from submit button
Hi, I have a website where many searches are made by the googlebot on our internal engine. We can make noindex on result page, but we want to stop the bot to call the ajax search button - GET form (because it pass a request to an external API with associate fees). So, we want to stop crawling the form button, without noindex the search page itself. The "nofollow" tag don't seems to apply on button's submit. Any suggestion?
Intermediate & Advanced SEO | | Olivier_Lambert0 -
Zip Code Blocks the Search Engines!
I have a site where when you visit the product pages, it asks for your zip code. This is obviously blocking the bots from crawling the site. I know you can basically tell the bots how to ignore the zip code feature but I am not exactly sure how to do this. Any help would be appreciated
Intermediate & Advanced SEO | | lhawk0 -
How do I link a profile picture to my search results?
How do you get an icon which links to a google profile to display in google search results? Link this.... https://skitch.com/edwardrobertshaw/f12f8/central.ly-google-search
Intermediate & Advanced SEO | | ed1234560