How did my dev site end up in the search results?
-
We use a subdomain for our dev site. I never thought anything of it because the only way you can reach the dev site is through a vpn. Google has somehow indexed it. Any ideas on how that happened? I am adding the noindex tag, should I used canonical? Or is there anything else you can think of?
-
Personally, I'd still recommend using robots.txt to disallow all crawlers, even if more steps are taken.
-
Don't use tool removal, it can go bad indeed. Now, are you sure that there are no external links coming from anywhere?
For now I'd recommend putting noindex, nofollow on that dev subdomain and do manual recrawl through GWT.
-
It just uses internal links. Do you think I should try the webmaster tools removal? That seems like it could go wrong.
-
I never used screaming frog, does it check both external and internal links?
-
I have ran screaming frog to see if there are any links to any pages and but couldn't see any. Even if Google did try to follow it the firewall would stop them. It is so strange.
-
Then my first assumption is that it's linked from somewhere - read my comment a little above.
-
Then there is a leak somewhere - Google bots can "see" your subdomain.
Or it's been simply linked from somewhere. Then Google will try to follow the link and that would make it indexed.
-
They are telling me that there are no holes, and I have tried getting to the pages but can not do it unless I am on my vpn.
-
We never updated the robots.txt because the site was behind a firewall. If you click on any of the results it will not load the page unless on my VPN.
-
Robots.txt won't help anyhow. Bots still can see that there is such directory, they just won't see what's inside of those directories/subdomains.
-
Hi there.
If what you say is true, then there are only two answers: you got a leak somewhere or your settings/configuration is messed up.I'd say go talk to your system admin and make sure that everything what's supposed to be closed is closed, IPs, which are supposed to be open for use are open and those IPs only.
-
Have you updated the dev sites robots.txt to disallow everything? It is up to the bot to listen, but that combined with removing all of the dev URLs from Google Webmaster tools should do the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Image Search - Is there a way to influence the related icons at the top of the image search results?
Google recently added related icons at the top of the image search results page. Some of the icons may be unrelated to the search. Are there any best practices to influence what is positioned in the related image icons section? Thank you.
Intermediate & Advanced SEO | | JaredBroussard1 -
Site Structure - Is it ok to Keep current flat architecture of existing site pages and use silo structure on two new categories only?
Hi there, I have a site structure flat like this it ranks quite well for its niche site.com/red-apples.html site.com/blue-apples.html The site is branching out into a new but related lines of business is it ok to keep existing site architecture as above while using a silo structure just for the two new different but related business? site.com/meat/red-meat.html site.com/fish/oceant-trout.html Thanks for any advice!
Intermediate & Advanced SEO | | servetea0 -
Why Did My Site Go Limp On Me?
One of my clients was once in the #1 position for "Philadelphia interior designer" and other related terms, but her site has dropped significantly. Still it is on the first page, but far from its former glory. http://www.interiorsbydonnahoffman.com is the site. What really confuses me is why in her home turf search of "Bucks County Interior Designer" a competitor, http://www.miriamansellinteriors.com, is above her in the SERPS. According to OSE her competitor has a PA of 32 vs my client's 39. My client has 35 Linking Root Domains (and some of high quality) compared to just 11 for the competition. In all aspects her competitor looks weaker and less relevant to me. Her site has been weak in the SERPs since May/June. We are redesigning her site- she has a high bounce rate compared to my other interior design clients, something like 55%. Any insights from y'all?
Intermediate & Advanced SEO | | dfhytrwy0 -
Site duplication issue....
Hi All, I have a client who has duplicated an entire section of their site onto another domain about 1 year ago. The new domain was ranking well but was hit heavily back in March by Panda. I have to say the set up isn't great and the solution I'm proposing isn't ideal, however, as an agency we have only been tasked with "performing SEO" on the new domain. Here is an illustration of the problem: http://i.imgur.com/Mfh8SLN.jpg My solution to the issue is to 301 redirect the duplicated area of the original site out (around 150 pages) to the new domain name, but I'm worried that this could be could cause a problem as I know you have to be careful with redirecting internal pages to external when it comes to SEO. The other issue I have is that the client would like to retain the menu structure on the main site, but I do not want to be putting an external link in the main navigation so my proposed solution is as follows: Implement 301 redirects for URLs from original domain to new domain Remove link out to this section from the main navigation of original site and add a boiler plate link in another area of the template for "Visit xxx for our xxx products" kind of link to the other site. Illustration of this can be found here: http://i.imgur.com/CY0ZfHS.jpg I'm sure the best solution would be to redirect in URLs from the new domain into the original site and keep all sections within the one domain and optimise the one site. My hands are somewhat tied on this one but I just wanted clarification or advice on the solution I've proposed, and that it wont dramatically affect the standing of the current sites.
Intermediate & Advanced SEO | | MiroAsh0 -
301 entire site
Is there a good 301 code snippet to change just the root domain but keep the ending extensions? I just bid on a domain that I think would be much better for me moving forward, but do not want to have to try going through thousands of pages to do their 301 individually My site is almost 4 yrs old. Well established and has a large fanbase. Several of our social networks are under the name of the new branded domain, hence part of the desire to switch.
Intermediate & Advanced SEO | | Atomicx0 -
Duplicate title tags in a pagination case (not search results)
Hello fellows I working on a site that have duplicate pages' titles tags, the case is the following: there is a category page that is split in couple pages via pagination (1,2,3 ....) all these pages have the same title tag for example: looking at this page for example, it has a pagination at the bottom, all the pages in the pagination (let's say page 5 for example) have the same title tag for the rest (including the first page). How should I deal with this case? should I simply add to the title tag its corresponding page number? e.g, full interviews - page 1? or I should I use canonical tag? I'm really confused. Your help is really appreciated.
Intermediate & Advanced SEO | | MHD0 -
Alternative links in the search results.
Hello, This is a short question Please look at this SERP screenshot: http://imgur.com/1EMen Who do they get the other links under their results. Cornel
Intermediate & Advanced SEO | | Cornel_Ilea0 -
Any SEO suggestions for my site?
Site in question: http://bit.ly/Lcspfp Does anyone have any suggestions for any on-site SEO that would benefit my website? Any recommendations, big or small are appreciated.
Intermediate & Advanced SEO | | RichardTaylor1