Robots.txt query
-
Quick question, if this appears in a clients robots.txt file, what does it mean?
Disallow: /*/_/
Does it mean no pages can be indexed? I have checked and there are no pages in the index but it's a new site too so not sure if this is the problem.
Thanks
Karen
-
Thank you so much, that is a great help!
-
That blocks all spiders from viewing those pages. I am not sure what and who did the /* /_/, but unless there is something there they don't want indexed then it is not necessary to keep it.
One thing you mind want to keep in mind as well, just because you block it on robots txt, doesn't mean a spider can't still go there.
Sometimes they don't listen to the robots txt(looking at you baidu)
-
User-agent: *
Thanks for your response.
-
What is the user agent?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Handling Pages with query codes
In Moz my client's site is getting loads of error messages for no follow tags on pages. This is down to the query codes on the E-commerce site so the URLs can look like this https://www.lovebombcushions.co.uk/?bskt=31d49bd1-c21a-4efa-a9d6-08322bf195af Clearly I just want the URL before the ? to be crawled but what can I do in the site to ensure that these errors for nofollow are removed? Is there something I should do in the site to fix this? In the back of my mind I'm thinking rel-conanical tag but I'm not sure. Can you help please?
Technical SEO | | Marketing_Optimist1 -
The use of robots.txt
Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta0 -
Blocked URL's by robots.txt
In Google Webmaster Tools shows me 10,936 Blocked URL's by robots.txt and it is very strange when you go to the "Index Status" section where shows that since April 2012 robots.txt blocked many URL's. You can see more precise on the image attached (chart WMT) I can not explain why I have blocked URL's ? because I have nothing in robots.txt.
Technical SEO | | meralucian37
My robots.txt is like this: User-agent: * I thought I was penalized by Penguin in April 2012 because constantly i'am losing visitors now reaching over 40%. It may be a different penalty? Any help is welcome because i'm already so saturated. Mera robotstxt.jpg0 -
Robots.txt - What is the correct syntax?
Hello everyone I have the following link: http://mywebshop.dk/index.php?option=com_redshop&view=send_friend&pid=39&tmpl=component&Itemid=167 I want to prevent google from indiexing everything that is related to "view=send_friend" The problem is that its giving me dublicate content, and the content of the links has no SEO value of any sort. My problem is how i disallow it correctly via robots.txt I tried this syntax: Disallow: /view=send_friend/ However after doing a crawl on request the 200+ dublicate links that contains view=send_friend is still present in the CSV crawl report. What is the correct syntax if i want to prevent google from indexing everything that is related to this kind of link?
Technical SEO | | teleman0 -
Canonical solution for query strings?
Greetings, The Hotel company where I'm employed uses query strings in it's url's to track customers. The query strings are integrated into our property management system, and they help identify who we need to pay commissions to, so they aren't going anywhere. While I understand that session variables could have been a better solution, I sort of inherited this problem. The issue I'm running into is that my Webmaster tools picks up these query strings as actual url's. So for instance: www.url.com/index.php?P_SOURCE=WBFQ Seems like a duplicate page of my root, and like wise for all my other pages that use our booking widget. So, Is there a canonical solution to this issue? or would 301/302's be the only solution. Also, we may have 10 different but specific query strings to put into our urls. Would the 301/302 approach cause any server issues for say 10 pages? So 10 pages x 10 access codes = a lot of redirects. Thanks in advance, Cyril
Technical SEO | | Nola5040 -
Should I add my blog posts to my sitemap.txt file?
This seems like it should be an obvious no, just because of the amount of work that would entail, and then remembering to do it every time I make a post, but since I couldn't find anything on Google about it and have never heard anyone mention it, I figured I'd ask.
Technical SEO | | UnderRugSwept0 -
Search engines have been blocked by robots.txt., how do I find and fix it?
My client site royaloakshomesfl.com is coming up in my dashboard as having Search engines have been blocked by robots.txt, only I have no idea where to find it and fix the problem. Please help! I do have access to webmaster tools and this site is a WP site, if that helps.
Technical SEO | | LeslieVS0 -
Query String Redirection
In PHP, I'm wanting to store a session variable based upon a link that's clicked. I'm wanting to avoid query strings on pages that have content. My current workaround is to have a link with query strings to a php file that does nothing but snags the variables via $_GET, stores them into $_SESSION, and then redirects. For example, consider this script, that I have set up to force to a mobile version. Accessed via something like a href="forcemobile.php?url=(the current filename)" session_start(); //Location of vertstudios file on your localhost. Include trailing slash $loc = "http://localhost/web/vertstudios/"; //If GET variable not defined, this page is being accessed directly. //In that case, force to 404 page. Same case for if mobile session variable //not defined. if(!(isset($_GET["url"]) && isset($_SESSION["mobile"]))){ header("Location: http://www.vertstudios.com/404.php"); exit(); } //Snag the URL $url = $_GET["url"]; //Set the mobile session to true, and redirect to specified URL $_SESSION["mobile"] = true;header("Location: " . $loc . $url); ?> Will this circumvent the issue caused by using query strings?
Technical SEO | | JoeQuery0