Disallow: /sr/ and Disallow: /si/ - robots.txt
-
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.
If any of you know what these directives do, then please let me know.
-
Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.
I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
-
Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.
However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
Rich snippets tool/ schemer.org update?
We have a domain in the netherlands that shows stars (rich snippets) in adwords and in the organic search results. As we do the the in rich snippets test tool we see some errors as shown in the attachments, the errors are shown in review aggregate - but stars are stil shown in paid and organic search. Do we need to follow-up these errors? Do they influence ranking - or the stars in the near future? Z0fCKRW
Web Design | | remkoallertz1 -
Requirements for mobile menu design have created a duplicated menu in the text/cache view.
Hi, Upon checking the text cache view of our home page, I noticed the main menu has been duplicated. Please see: http://webcache.googleusercontent.com/search?q=cache:http://www.trinitypower.com&strip=1 Our coder tells me he created one version for the desktop and one for the mobile version. Duplicating the menu cannot be good for on page SEO. With that said, I have had no warnings reported back from Moz. Maybe the moz bots are not tuned to looks for such a duplication error. Anyway, the reason the coder created a different menu for mobile in order to support the design requirements. I did not like the look and feel of the responsive version created based on the desktop version. Hi solution to this problem is to convert the Mobile version menu into ajax. what do you guys think? Thanks, Jarrett
Web Design | | TrinityPower0 -
Can multiple domains compete with one another if they have the same/similar content?
Could an ecommerce site with a .co.nz nd .com.au domain compete with one another and hard organic rankings if the content of the pages are the same? All the links would be identical (apart from .co.nz or .com.au) the product descriptions, pages titles etc... would all be identical or similar (our page titles are ever so slightly different). Could this be hurting us? Thanks in advance ^ Paul
Web Design | | kevinliao0 -
Question #2: All of my INTERNAL links in OSE are being indexed from http://www.e.com/default.asp, and all my EXTERNAL links are linked to http://www.e.com/ am I getting a fraction of the link juice because of that?????
Hey guys, sorry for the really long question, but it appears that I am losing between 50 and 75 % of my link juice to my internal pages. In OSE all main category links (left sidebar) are being indexed from the URL that includes default.asp, even though NONE of my external links include that: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-Thread-s%2F78.htm If you check the PA for http://www.uncommonthread.com/: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2F You see that it is practically double the PA of http://www.uncommonthread.com/default.asp: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FDefault.asp **Also, non of my internal menu links are being indexed. ** Look at the menu on this page: http://www.uncommonthread.com/Sulky-Thread-s/78.htm and then look at the OSE information here for the "invisible thread" item from the menu on the page above^^^: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-monofilament-s%2F54.htm Thanks SOOO much! Pre-thumbs and thanks to anyone that can lend me a seconds worth of advice! Thanks again for your time, Tyler A.
Web Design | | TylerAbernethy0 -
After a website redesign, what is the impact and is it a good practice to use /v2/ naming convention?
Hi mightyful SEOMoz community. We just launched a redesign of our commercial website from https://www.data-field.com to https://www.data-field.com/v2/ All URLs from previous website were 301 permanent redirect to the appropriate page in the new website, and the root domains ( /, /v2/ ) send the users to their own language content /v2/en/, /v2/fr/, /v2/zh/ Up to here everything is fine. But then I setup the usual "Share" buttons, only to find that they were displaying a "0" count. Then I realized that it was because of the root URL change from / to /v2/ My question is the following: 1. Is using /v2/ a good practice? 2. If yes, then should I link the Social tool to https://www.data-field.com/ ( only ) instead of linking it to the actual page in the address bar? Thanks for your answers.
Web Design | | NicolasE0 -
RSS Submissions Positive/Negative/Neutral Impact...
We are looking to push our site content and blog content out to the masses... There are several sites and services that accept RSS feeds or submit RSS feeds to 50+ RSS sites. Have you seen an positive or negative impact of submitting feeds to these RSS directories? I am primarily looking at this for getting or content out and builing inbound links... Any thoughts or feedback would be appreciated... C
Web Design | | hireawizseo0 -
Does anyone think the <figcaption>attribute from HTML5 will have any influence for image search?</figcaption>
There is a <figure>element that is supposed to provide better descriptions of image on the web in HTML5 - do you think that will replace the importance of the "Alt" tag? Link to figcaption description </figure>
Web Design | | RankSurge2