Question about Syntax in Robots.txt
-
So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file?
Currently I have-
Disallow: /attachment_idWhere "attachment_id" is the parameter. Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do
Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first.
Thanks!
-
That's excellent Chris.
Use the Remove Page function as well - it might help speed things up for you.
-Andy
-
I don't know how but I completely forgot I could just pop those URL's in GWT and see if they were blocked or not and sure enough, Google says they are. I guess this is just a matter of waiting.... Thanks much!
-
I have previously looked into both of those documents and the issue remains that they don't exactly address how best to block parameters. I could do this through GWT but just am curious about the correct and preferred syntax for the robots.txt as well. I guess I could just look at sites like Amazon or other big sites to see what the common practices are. Thanks though!
-
Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do
It can take Google some time to remove pages from the index.
The best way to test if this has worked is hop into Webmaster Tools and use the Test Robots.txt function. If it has blocked the required pages, then you know it's just a case of waiting - you can also remove pages from within Webmaster Tools as well, although this isn't immediate.
-Andy
-
Hi there
Take a look at Google's resource on robots.txt, as well as Moz's. You can get all the information you need there. You can also let Google know about what URLs to exclude from it's crawls via Search Console.
Hope this helps! Good luck!
-
Im not a robots.txt expert by a long shot, but I found this, which is a little dated, which explained it to me in terms i could understand.
https://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/
there is also a feature in Google Webmaster tools called URL parameters that lets you block URLs with set parameters for all sorts of reason to avoid duplicate content etc. I havn't used it myself but may be work looking into
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to answer questions when there no questions for my keyword
Hello, Let's say I want to rank on "Alsace bike tour" whatever tool I use Moz keyword explorer, google suggest , keyword.io, answer the public ... there are not questions... so... what do I need to answer ? I imagine that for google there are some questions more relevant than others ? Should I answer do I need to bring my own bike or where will I go... ? and will google give me "points " for answering those questions even though people don't have questions... For the keyword title tag, it is easy, people ask the character limit, title tag generator and so on but for may keywords like that ones I am targeting people have NO Questions ! Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Canonical Tag Question Regarding Two State Pages
Hi Moz Fam! Question... we have two pages for each state. Both pages are used in our PPC campaigns. One of the two state pages is fully optimized for organic search. The page that's optimized for organic has 1,500-2,000 words, the other one has thin content. All the content is unique, nothing duplicate. We call one set of state pages that I optimized my SEO state pages, then the other ones are our PPC state pages. Should I be setting a canonical tag to one of these pages to let Google know which one is the "master" page? (My SEO state page is the master) I've never used them, so I'm not sure what the right answer is for this. Thanks!
Intermediate & Advanced SEO | | LindsayE0 -
Rel=canonical Question
Alright, so let's say we've got an event coming up. The URL is website.com/event. On that page, you can access very small pages with small amounts of information, like website.com/event/register, website.com/event/hotel-info, and website.com/event/schedule. These originally came up as having missing meta descriptions, and I was thinking a rel=canonical might be the best approach, but I'm not sure. What do you think? Is there a better approach? Should I have just added a meta description and moved on?
Intermediate & Advanced SEO | | MWillner0 -
Search Results Pages Blocked in Robots.txt?
Hi I am reviewing our robots.txt file. I wondered if search results pages should be blocked from crawling? We currently have this in the file /searchterm* Is it a good thing for SEO?
Intermediate & Advanced SEO | | BeckyKey0 -
To redirect or not to redirect, that is the question
I work for a software company that is redeveloping the website (same domain.) We have tons of content in the form of articles and documents for support, how to use the product better, case studies, and blog posts. I've downloaded a landing page report and many of these have low impressions and little or no clicks (some ranked high other very low.) Should I redirect all this content to the new site where some of it won't exist or forget about it because of the lack of juice? Is there a rule-of-thumb threshold for redirecting for content?
Intermediate & Advanced SEO | | Nobody15969167212220 -
Robots.txt: how to exclude sub-directories correctly?
Hello here, I am trying to figure out the correct way to tell SEs to crawls this: http://www.mysite.com/directory/ But not this: http://www.mysite.com/directory/sub-directory/ or this: http://www.mysite.com/directory/sub-directory2/sub-directory/... But with the fact I have thousands of sub-directories with almost infinite combinations, I can't put the following definitions in a manageable way: disallow: /directory/sub-directory/ disallow: /directory/sub-directory2/ disallow: /directory/sub-directory/sub-directory/ disallow: /directory/sub-directory2/subdirectory/ etc... I would end up having thousands of definitions to disallow all the possible sub-directory combinations. So, is the following way a correct, better and shorter way to define what I want above: allow: /directory/$ disallow: /directory/* Would the above work? Any thoughts are very welcome! Thank you in advance. Best, Fab.
Intermediate & Advanced SEO | | fablau1 -
Another Guest Blogging Question!
If you had 1 blog with good mozbar stats but hosted in the US and another with lets say 75% of the mozbar stats of the first but hosted in the UK, and your website is hosted in the UK which one would benefit SEO the most?
Intermediate & Advanced SEO | | activitysuper0 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0