Disallow: /sr/ and Disallow: /si/ - robots.txt
-
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember.
If any of you know what these directives do, then please let me know.
-
Thanks Tomas and Mike - good advice - I have done that and found legacy stuff they've since moved away from - there is indeed no current use for the directives.
I wonder whether there's any resource on the web that lists all robots.txt directives - and interprets them - if not then perhaps it would an idea for Moz?
-
Have a look at your site through http://web.archive.org/. You'll be able to see what the directories were used for.
However, if there's no use for them on the current site then what's the purpose of keeping these disallows in the robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do search engines see copy/keywords when it appears only at the bottom of a page?
My client is looking to improve their SEO, and to date I've written meta data and made some initial recommendations. Thing is, on some of their pages, the body copy appears at the bottom of the page, past links and big, splashy images. My question is, will search engines even see that copy to crawl it for keywords? Thanks!
Web Design | | MarcieHill0 -
Tips on finding the right Senior Designer / Design Director
Hello Everyone, I manage a fairly large educational website that we are looking to completely redesign to improve the overall site user experience and usability. In the past I, a non-designer, business person, would just roughly draw how I thought the site should look and our developer (who is good, but not a web designer) would just try his best to make everything look profession. Over the years, it has become painfully obvious that we need to invest in more design expertise and move towards a modern, smartly designed website. So my question: Where are the best places to find good freelance designers? I have of course conducted web searches, browsed elance, and asked my network for referrals. However, I am finding that most of the really good ones, ones who have to ability to take charge and lead us through this entire process, and who have at least a basic understanding of SEO principles, work for larger integrated development shops who also expect their people to develop the new site as well. We already have a developer and are primarily looking for the design expertise. Does anyone in the Moz community have any suggestions or even referrals? Thanks! Eric
Web Design | | Eric_R0 -
Rich snippets tool/ schemer.org update?
We have a domain in the netherlands that shows stars (rich snippets) in adwords and in the organic search results. As we do the the in rich snippets test tool we see some errors as shown in the attachments, the errors are shown in review aggregate - but stars are stil shown in paid and organic search. Do we need to follow-up these errors? Do they influence ranking - or the stars in the near future? Z0fCKRW
Web Design | | remkoallertz1 -
Is it bad to have /index.php at the end of a uri?
Is it bad for SEO if traffic is directed to "http://www.example.com/someuri/index.php" instead of "http://www.example.com/someuri/" and would it be works setting up a redirect rule at htaccess level?
Web Design | | NoisyLittleMonkey1 -
Which Hat of SEO (Black/White) Goes with this Green Lace Dress?
Doing some research (honestly) I came across this page: http://www.modcloth.com/th/green-lace-dress That page ranks very well for the phrase "green lace dress" - #2 in Google for me. I'm sure there's a good amount of links coming into it, but on the surface there's only 1 mention of green on the page and 2 mentions of lace. Looking at the code, you can see ModCloth is using the Quick View links below the products to display a more detailed description of the product - wherein lies many more instances of green and lace. So I ask: Which color of SEO hat goes with this green lace dress - Black or White? Is it good SEO to only show the product descriptions when the user initiates the Quick View? Or is it shady SEO to hide so much text from the immediate view of the user? Please select one and explain why.
Web Design | | rball10 -
Is it OK to 301 Redirect http://homepage.com to http://blog.homepage.com?
I don't have a homepage built yet, and I suspect that my blog will be my primary SEO draw in the long term, so I'd like to do a 301 redirect (for the techies, I'm doing it in Django with http.HttpResponsePermanentRedirect ) I just wanted to confirm that this wouldn't cause any problems as Google gets used to my page (it's only been up for a few days) Thanks! -matt
Web Design | | 49wetnoodles0 -
How do search engines interpret <hgroup>...</hgroup> tags?
Hi there. I'm building an HTML 5 site and through research of new HTML 5 elements I've seen little conclusive information about the interpretation of the new <hgroup>tag, in terms of SEO application and interpretation. In particular does Google interpret the nested heading tags as individual elements or does it combine them into one entity? For example, say I have: <hgroup> # Article Heading ## Article Sub-heading </hgroup> How is this interpreted by Google and what would be some good SEO best practices regarding the <hgroup>tag in HTML5: Is it interpretted as a single tag (" Article Heading: Article Sub-heading ") or two separate heading tags (one and one )? Also, how much does the ordering of the tags matter (say for example I wanted something like the following for visual purposes? <hgroup> ## Article Sub-heading # Article Heading </hgroup> One last thing: is it safe to assume that it is indeed OK to have multiple tags on a single page (as referenced by Matt Cutts a while back in a Webmaster Video)? Thanks! </hgroup> </hgroup>
Web Design | | LMDNYC2 -
Hosting/design company that is both cheap and has a nice partner package. Any ideas?
I need to signon with a hosting/design company that is both cheap and has a nice partner package. Any ideas?
Web Design | | christinarule0