Block search bots on staging server
-
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
-
Robots.txt should be sufficient, but if you want to completely block them, here is a useful resource:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages Disappearing from Search
Hi, We have had a strongly ranking site since 2004. Over the past couple of days, our Google traffic has dropped by around 20% and some of our strong pages are completely disappearing from the rankings. They are still indexed, but having ranked number 1 are nowhere to be found. A number of pages still remain intact, but it seems they are increasingly disappearing. Where should we start to try and find out what is happening? Thanks
Intermediate & Advanced SEO | | simonukss0 -
Adding some unique text to search
I do not know how to easily explain that but I will try 🙂 We are running the yellow pages in our country, where most (all) of the companies are listed. Each company has their keywords that are used in our search. All of those search keywords are indexed by Google and it goes something like that domain.com/search/keyword So far this page only lists the companies that match the search query. So that's not much content. I'm thinking about adding a sentence that would go something like this:
Intermediate & Advanced SEO | | FCRMediaLietuva
The search query lists the companies of our country that do KEYWORD. So far COUNT was found. That seems pretty short and I am not sure if that does a lot of help for Google. But I cannot think of more text. Any ideas? Or should I give up on the idea of adding anything at all?0 -
Can bots identify shmushed keywords?
I remember reading some years ago that domains and pages that have smushed keywords, such as cheapbaseballs.com/redbaseball.html could be identified by Google as "cheap baseballs" and "red base ball". Is this still correct?
Intermediate & Advanced SEO | | CFSSEO0 -
International SEO and Server Hosting
I am quite new to international SEO. I have a customer who wants to use the same website content on various domains targeting different countries. Such as; xxxx.hk - same content targeting Hong Kong xxxx.co.uk - same content targeting UK xxxx.de - same content targeting Germany I found that it could be possible with the Google suggested hreflang without any duplicate content problem. Is that true? Could someone explain this for me. Another question; if above is true do we need to make other adjustments as well. Such as; Any adjustments in Google Webmaster Tools (for each domain) Server location, does that really make a difference? Can we host each of the domains on the same server or should we seperate them and host each one on the country it is targeting ? Thanks in advance!
Intermediate & Advanced SEO | | stradiji0 -
Ranking for local searches without city specific keywords?
Hey guys! I had asked this question a few months ago and now that we are seeing even more implicit information determining search results, I want to ask it again..in two parts. Is is STILL best practice for on-page to add the city name to your titles, h1s, content etc? It seems that this will eventually be an outdated tactic, right? If there is a decent amount of search volume without any city name in the search query (ie. "storefont signs", but no search volume for the phrase when specific cities are added (ie. "storefront signs west palm beach) is it worth trying to rank and optimize for that search term for a company in West Palm Beach? We can assume that if there are 20,000 monthly searches for the non-location specific term that SOME of them would be fairly local, so do we optimize the page without the city name and trust Google to display results with a local intent...therefore showing our client's site in the SERPS when someone searches "sign company" and they are IN West Palm Beach? If there is any confusion, please just ask me to clarify! I think this would be a great WhiteBoard Friday topic for Rand!
Intermediate & Advanced SEO | | RickyShockley0 -
Duplicate title tags in a pagination case (not search results)
Hello fellows I working on a site that have duplicate pages' titles tags, the case is the following: there is a category page that is split in couple pages via pagination (1,2,3 ....) all these pages have the same title tag for example: looking at this page for example, it has a pagination at the bottom, all the pages in the pagination (let's say page 5 for example) have the same title tag for the rest (including the first page). How should I deal with this case? should I simply add to the title tag its corresponding page number? e.g, full interviews - page 1? or I should I use canonical tag? I'm really confused. Your help is really appreciated.
Intermediate & Advanced SEO | | MHD0 -
Having Content be the First thing the bots see
If you have all of your homepage content in a tab set at the bottom of the page, but really would want that to be the first thing Google reads when it crawls your site, is there something you can implement where Google reads your content first before it reads the rest of your site? Does this cause any violations or are there any red flags that get raised from doing this? The goal here would just be to get Google to read the content first, not hide any content
Intermediate & Advanced SEO | | imageworks-2612900 -
Does Blocking ICMP Requests Affect SEO?
All in the title really. One of our clients came up with errors with a server header check, so I pinged them and it times out. The hosting company have told them that it's because they're blocking ICMP requests and this doesn't affect SEO at all... but I know that sometimes pinging posts, etc... can be beneficial so is this correct? Thanks, Steve.
Intermediate & Advanced SEO | | SteveOllington0
