Robots.txt question
-
What is this robots.txt telling the search engines?
User-agent: * Disallow: /stats/
-
Oh - and it's affect the domain negatively.. when cleaning up your site directories via robots.txt. Its actually better as I explained below
-
Hey Mark,
It's good practice to disallow access to any folder/content you don't want indexed as well as anything that has any security involved (login's, databases etc).
It will also keep the most important pages from the domain in front of the search spiders eyes, while keeping poor content out of the indes. This helps the domain on a site authority level provide valuable content and information to users.
Lower ranking pages, can cause the domain to be pulled down by serarch engines (Google and Bing have attested to this already) as they want businesses to focus on high value content - which leads to better user experience.
Cheers!
-
Thanks- wanted to make sure all was copacetic there. I'm assuming that it's good practice to disallow access to stats and won't impact the site negatively?
-
Assuming that this is the entire contents of this file: It says that no robot (search engine spider, other crawler, etc.) should visit or index anything in the /stats/ directory or any directories inside of it.
More info available here: http://www.robotstxt.org/robotstxt.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I Block https URLs using Host directive in robots.txt?
Hello Moz Community, Recently, I have found that Google bots has started crawling HTTPs urls of my website which is increasing the number of duplicate pages at our website. Instead of creating a separate robots.txt file for https version of my website, can I use Host directive in the robots.txt to suggest Google bots which is the original version of the website. Host: http://www.example.com I was wondering if this method will work and suggest Google bots that HTTPs URLs are the mirror of this website. Thanks for all of the great responses! Regards,
Technical SEO | | TJC.co.uk
Ramendra0 -
Pageing page and seo meta tag questions
Hi if i am using paging in my website there is lots of product in my website now in paging total paging is 1000 pages now what title tag i need to add for every paging page or is there any good way we can tell search engine all page or same ?
Technical SEO | | constructionhelpline0 -
Header Tag Question
While reviewing code on a site, I found the following: <h1 class="<a class="attribute-value">logo</a>"> <a id="<a class="attribute-value">logo</a>" href="[http://siteexampleh1.com](view-source:http://dmbinc.com/)"><span>Example of most important content on this page- Companyspan>a> h1> Is this the correct way to place code for an h1 tag? The content is cached within the page and is hidden to the viewer. The content that is assigned as the h1, is a logo. Majority of code I have been reviewing does not use this setup. The code would instead read as ( This is heading 1 ). Can anyone provide insights on this? Thanks!
Technical SEO | | jfeitlinger0 -
Robots.txt file
How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line? Do i need to install a Robots.txt file on each page?
Technical SEO | | gimes0 -
Popup Question
Hi Everyone, I have a question. Your input will be very much appreciated. My company's new website design is using a popup. I have some reservation about it and I want to know what your thoughts are. Ok, some information on what this popup is like. When a user clicks on a subcategory page, there's a popup that would ask for size, color, etc - it's like a form and those are the criteria. If nothing is selected, the product list on the subcategory page doesn't load - so the only thing is showing is the the H1 and description but everything else is empty. When a user does select a criteria the landing page is no longer the subcategory but another page with that ID. So basically the user never really land on the subcategory page but to another page with a different query string. Is this bad for SEO? Would you recommend to keep the popup? Thanks,
Technical SEO | | truckguy770 -
Another 301 redirect question - penalty?
Good Morning, We have 2 sites have images and minimal text on them. The images have links that point to a 3<sup>rd</sup> site that facilitates eCommerce. Question: If we 301 redirect these sites permanently to yet a 4<sup>th</sup> site… 1) Does it violate any G’s guidelines 2) Should we delete the links embedded in the images (as they point to the 3<sup>rd</sup> site) Thanks
Technical SEO | | Prime850 -
Summarize your question.Sitemap blocking or not blocking that is the question?
Hi from wet & overcast wetherby UK 😞 Ones question is this... " Is the sitemap plus boxes blocking bots ie they cant pass on this page http://www.langleys.com/Site-Map.aspx " Its just the + boxes that concern me, i remeber reading somewherte javascript nav can be toxic. Is there a way to test javascript nav set ups and see if they block bots or not? Thanks in advance 🙂
Technical SEO | | Nightwing0 -
Robots.txt question
Hello, What does the following command mean - User-agent: * Allow: / Does it mean that we are blocking all spiders ? Is Allow supported in robots.txt ? Thanks
Technical SEO | | seoug_20050