Block session id URLs with robots.txt

[Mat_C]

Hi,

I would like to block all URLs with the parameter '?filter=' from being crawled by including them in the robots.txt.

Which directive should I use:

User-agent: *
Disallow: ?filter=

or

User-agent: *
Disallow: /?filter=

In other words, is the forward slash in the beginning of the disallow directive necessary?

Thanks!

[Mat_C]

Hi Martijn,

Thanks for the answer. Regarding the forward slash in the beginning, is it necessary to use this?

In the robots text from Zalando for example, you can see that they don't use it for a lot of filters.

[Martijn_Scheijbeler]

Uhh, that's not what the requester is looking for and could actually cause tons of problems if you would apply this on a site that you're unaware of. I would always go with the most limiting robots.txt that you can and in this case, I would go with: /?filter=

[jasongmcmahon]

Hi,

The following should suffice as it will black any URL with a "?" in it

User-agent: *
Disallow: /*?