Pre-processor directives for filters

[ameshkov]

Based on this discussion:
gorhill/uBlock#3331

We should provide multiple pre-processor directives that can be used by filters maintainers to improve compatibility with different ad blockers.

Syntax

!#if condition
Anything goes here
!#include URL_or_a_relative_path
!#endif

• !#if, !#endif -- filters maintainers can use these conditions to supply different rules depending on the ad blocker type.
• condition -- just like in some popular programming languages, pre-processor conditions are based on constants declared by ad blockers. Ad blocker authors define on their own what exact constants do they declare.
• !#include -- this directive allows to include contents of a specified file into the filter.

Conditions

When an adblocker encounters an !#if directive, followed eventually by an !#endif directive, it will compile the code between the directives only if the specified condition is true. Condition supports all the basic logical operators.

Example:

!#if (adguard && !adguard_ext_safari)
||example.org^$third-party
!#endif

Including a file

The !#include directive supports only files from the same origin to make sure that the filter maintainer is in control of the specified file. The included file can also contain pre-processor directives (even other !#include directives).

Ad blockers should consider the case of recursive !#include and implement a protection mechanism.

Examples

Filter URL: https://example.org/path/filter.txt

!
! Valid (same origin):
!#include https://example.org/path/includedfile.txt
!
! Valid (relative path):
!#include /includedfile.txt
!#include ../path2/includedfile.txt
!
! Invalid (another origin):
!#include https://example.com/path/includedfile.txt

Remarks

• If included file is not found or unavailable, the whole filter update should fail.
• A conditional directive beginning with a #if directive must explicitly be terminated with a #endif directive.
• Whitespaces matter. !#if is a valid directive, while !# if is not.

AdGuard-specific

• Any mistake in a pre-processor directive will lead to AdGuard failing the filter update in the same way as if the filter URL was unavailable.
• Pre-processor directives can be used in the user filter (or in the custom local filters). Same-origin limitation should be disabled for local filters.

What constants should we declare

• adguard -- Declared always. Lets maintainers know that this is one of AdGuard products. Should be enough in 95% of cases.

Product-specific constants for some rare cases (or not so rare, thx Safari):

• adguard_app_windows -- AG for Windows
• adguard_app_mac -- AG for Mac
• adguard_app_android -- AG for Android
• adguard_app_ios -- AG for iOS
• adguard_ext_chromium -- AG browser extension for Chrome
• adguard_ext_firefox -- AG browser extension for Firefox
• adguard_ext_edge -- AG browser extension for Edge
• adguard_ext_safari -- AG browser extension for Safari
• adguard_ext_opera -- AG browser extension for Opera
• adguard_ext_android_cb -- AG content blocker for Samsung/Yandex

[ameshkov]

@gorhill check this out plz, would you like to change something?

As I understand, @mat41997 and @hawkeye116477 are okay with this solution.

[ameshkov]

@mat41997 frankly, I don't understand how do you manage to live with a single file.

Most of the large filters I know split the rules into multiple files just because it's easier to support it this way:
https://github.com/easylist/easylist/tree/master/easylist
https://github.com/easylist/easylistgermany/tree/master/easylistgermany

[gorhill]

@ameshkov Looks all good to me, thanks for fleshing out this. I started implementing the !#include directive (I just need to remove the space) because it solves an immediate issue, for the other directives I will personally go on a per-need basis.

For the !#include directive, you state:

!#include relative_path_to_a_file.txt

I think for simplicity it should be the full path, but for the same origin as the main list. Personally I accept full URL, but the origins have to match. If no full URL is given, I treat as full path to which the main origin is prepended. Relative paths can be rather complicated to deal with, and having to specify full path shouldn't be such a burden on filter list maintainers, while keeping the code simpler.

[ameshkov]

Relative paths can be rather complicated to deal with, and having to specify full path shouldn't be such a burden on filter list maintainers, while keeping the code simpler.

As I understand, this is almost enough to support relative paths:

I treat as full path to which the main origin is prepended

XHR will understand a path like that properly: "https://filters.adtidy.org/android/filters/../../windows/filters/2.txt"

[hawkeye116477]

@mat41997 frankly, I don't understand how do you manage to live with a single file.

Most of the large filters I know split the rules into multiple files just because it's easier to support it this way:
https://github.com/easylist/easylist/tree/master/easylist
https://github.com/easylist/easylistgermany/tree/master/easylistgermany

@ameshkov We want to have one list with rules for Adblock and rules which Adblock have problems with it, because of no scroll for example. So rules should be commented to Adblock not use it. But for beginning it can be just !#include relative_path_to_a_file.txt And I hope that soon this will be extended also to this what I talk 😄 .

[gorhill]

We want to have one list with rules for Adblock and rules which Adblock have problems with

@hawkeye116477 Having to comment every single filter does not make sense to me. The !#include directive is what works best and what is most sound design-wise. Having rules for specific purpose in one separate file is what works best, and this is actually already like this for POL. The only change required by POL filter list maintainers is to merely add a one line !#include directive in the main list, with no further changes -- the uBO-specific filter list is already a separate list. Can't be simpler than this.

We want to have one list

Who is "we"?

[jspenguin2017]

I propose !#include to just try to load the supplement filter and just move on if it fails to load, and !#require to load the supplement filter and abort if it fails to load.

[mikhaelkh]

1. I hope you could handle the situation when list a includes itself or there exists a cycle: a0 includes a1, a1 includes a2, ..., an includes a1
2. Will this raise the amount of net requests to the server? What about update time for adblock users?

[ameshkov]

1. That's explicitly mentioned in the description:

Ad blockers should consider the case of recursive !#include and implement a protection mechanism.

2.Will this raise the amount of net requests to the server? What about update time?

Sure, by the number of include directives.