Robots.txt question
-
What is this robots.txt telling the search engines?
User-agent: * Disallow: /stats/
-
Oh - and it's affect the domain negatively.. when cleaning up your site directories via robots.txt. Its actually better as I explained below
-
Hey Mark,
It's good practice to disallow access to any folder/content you don't want indexed as well as anything that has any security involved (login's, databases etc).
It will also keep the most important pages from the domain in front of the search spiders eyes, while keeping poor content out of the indes. This helps the domain on a site authority level provide valuable content and information to users.
Lower ranking pages, can cause the domain to be pulled down by serarch engines (Google and Bing have attested to this already) as they want businesses to focus on high value content - which leads to better user experience.
Cheers!
-
Thanks- wanted to make sure all was copacetic there. I'm assuming that it's good practice to disallow access to stats and won't impact the site negatively?
-
Assuming that this is the entire contents of this file: It says that no robot (search engine spider, other crawler, etc.) should visit or index anything in the /stats/ directory or any directories inside of it.
More info available here: http://www.robotstxt.org/robotstxt.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mod rewrite question
Sorry in advance if this isn't the best place to ask this question. Google Webmaster Tools has recently identified a ton of "Not Found" pages, which are actual pages with some digits appended at the end. For example, suppose an actual page on my blog is: (A) http://www.example.com/blog/2012/09/my-post-title/ This page works just fine. However, GWT has identified the following page as a "not found" page: (B) http://www.example.com/blog/2012/09/my-post-title/9157586677/1846732913010 This appears to be happening to hundreds of posts on my site. In each case, the "9157586677" portion of the URL is identical, but the remaining 13 digits change from page to page. I haven't been able to determine exactly what is causing this to happen - it's probably a social plug-in for Wordpress, or perhaps Disqus, but I'm not sure which one. I'll go through a process of elimination to narrow it down over the coming week. As a quick fix, I'd like to create a ModRewrite rule so that requests for (B) get 301 redirected to (A). Since there are hundreds of posts, I need to do this in a way that works regardless of what's in the "/2012/09/my-post-title/" part of the URL. Unfortunately, mod-rewrite is outside of my area of expertise. Can somebody please suggest how I can handle this? Thanks in advance. PS - As for tracking down the cause, I've looked at the source of the pages in the "Linked From" area of GWT and the Not Found link is nowhere to be found. That is why I assume the bad link is being generated by some javascript that is a part of one of my plug-ins. Update: It seems like Disqus is the source of these phantom links. There's considerable discussion here. I'll continue searching for a long-term solution. Meanwhile, I'd still appreciate help with the mod-rewrite question above. Thanks again.
Technical SEO | | ahirai0 -
Popup Question
Hi Everyone, I have a question. Your input will be very much appreciated. My company's new website design is using a popup. I have some reservation about it and I want to know what your thoughts are. Ok, some information on what this popup is like. When a user clicks on a subcategory page, there's a popup that would ask for size, color, etc - it's like a form and those are the criteria. If nothing is selected, the product list on the subcategory page doesn't load - so the only thing is showing is the the H1 and description but everything else is empty. When a user does select a criteria the landing page is no longer the subcategory but another page with that ID. So basically the user never really land on the subcategory page but to another page with a different query string. Is this bad for SEO? Would you recommend to keep the popup? Thanks,
Technical SEO | | truckguy770 -
Panda recovery timeframe question
Site was hit by Panda Aug. 22nd. Lost 90% of Google traffic. I know 🙂 We think we found a reason and made few changes to landing pages structure. Updated sitemaps submitted. When can we expect effect (if any) - few days or after next Panda data refresh? Thank you!P.S. What is also interesting, similar traffic loss from Bing/Yahoo happened at exactly the same date. Does that mean Bing is "stealing" search results from Google when can't provide their own relevant results? 🙂
Technical SEO | | LocalLocal0 -
Webmaster Tools Site Map Question
I have TLD that has authority and a number of micro-sites built off of the primary domain. All sites relate to the same topic, as I am promoting a destination. The primary site and each micro-site have their own CMS installation, but the domains are mapped accordingly. www.regionalsite.com/ <- primary
Technical SEO | | VERBInteractive
www.regioanlsite.com/theme1/ <- theme 1
www.regioanlsite.com/theme2/ <- theme 2
www.regionalsite.com/theme3/ <- theme 3 Question: Should my XML site map for Webmaster Tools feed all sites off of the primary domain site map or are there penalties for this? Thanks.0 -
Question on 301s
Hi Everyone, I have a questions on 301 redirects, i hope someone can give me some help on this. There was some 301 redirects made on some of the URLs at the beginning of the year, however we are now re-structuring the whole website, which means the URLs which had been given a 301 redirect are now getting another 301. The question is, should i delete the first 301 redirect from the htaccess file? Kind Regards
Technical SEO | | Paul780 -
How does robots.txt affect aliased domains?
Several of my sites are aliased (hosted in subdirectories off the root domain on a single hosting account, but visible at www.theSubDirectorySite.com) Not ideal, I know, but that's a different issue. I want to block bots from viewing those files that are accessible in subdirectories on the main hosting account, www.RootDomain.com/SubDirectorySite/, and force the bots to look at www.SubDirectorySite.com instead. I utilized the canonical meta tag to point bots away from the sub directory site, but I am wondering what will happen if I use robots.txt to block those files from within the root domain. Will the bots, specifically Google bot, still index the site at its own URL, www.AnotherSite.com even if I've blocked that directory with Disallow: /AnotherSite/ ? THANK YOU!!!
Technical SEO | | michaelj_me0 -
Using robots.txt to deal with duplicate content
I have 2 sites with duplicate content issues. One is a wordpress blog. The other is a store (Pinnacle Cart). I cannot edit the canonical tag on either site. In this case, should I use robots.txt to eliminate the duplicate content?
Technical SEO | | bhsiao0 -
Question concerning a 302 Redirect
Hi! I've already done some research on redirects, but I still have a question concerning a 302 redirect implemented at the homepage of a website. The Website www.domainA.com has a 302 redirect to www.domainA.com/content/.... Also all subsequent pages have the /content/ directory in their URLs: e.g domainA.com/content/products First thing I was wondering about, was the use of a redirect to a new site using an additional directory /content/... Why would anyone do this? Would it be enough to replace the 302 with a 301 redirect, or would you recommend to change the entire structure and eliminate this /content/ directory? The most logical structure would be www.domainA.com/products/.., and not www.domainA.com/content/products, right? Second thing: Given that 302 means temporary redirect, what are the actual implications when redirecting from domainA.com to domainA.com/content? I've heard that 302 redirects don't pass linkjuice and are detrimental for the site's rankings... What are the actual implications concerning the example above (302 redirect from domainA.com to domainA.com/content ? Would be great to get some advice about the first problem and maybe some insights about the second one concerning 302s in general. Thanks in advance! Cheers, Chris
Technical SEO | | adwordize0