Limit Pages Crawled for Cookies

It is possible to limit which pages on the website should be crawled for cookies. This can be useful, for example, on an online shop where the cookies set are the same across all instances of a given page type.

In the settings tool under the tab Settings > Crawler limits, enter a RegEx expression to specify which pages should have limited cookie scanning. You then set the Crawl pages value, which defines how many matching pages should be scanned for cookies. If you set this to 10, the first 10 matching pages found will be scanned and the rest will be ignored.

To limit all pages under, for example, “shop/”, enter the expression:

shop\/

If you need to limit pages that contain a query in the URL, you can use the expression:

\?

We recommend that you test the RegEx to ensure it works as intended. You can validate it on regex101.