BE: Full text search support

[germanosin]

• Breaking change? (if so, please describe the impact and migration path for existing application instances)

What changes did you make? (Give an overview)

Is there anything you'd like reviewers to focus on?

How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

• No need to
• Manually (please, describe, if necessary)
• Unit checks
• Integration checks
• Covered by existing automation

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (e.g. ENVIRONMENT VARIABLES)
• My changes generate no new warnings (e.g. Sonar is happy)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged

Check out Contributing and Code of Conduct

A picture of a cute animal (not mandatory but encouraged)

[yeikel]

@germanosin It seems that this change introduced #1344

[th0ger]

What is actually the functionality implemented here? (No linked issue, no docs added.)

We upgraded to 1.4.2 and set

  fts:
    enabled: true

and I've tried to use wildcard * in topic search, but no results are returned.

[th0ger]

@germanosin Here is my test:
"Apply full text search" button enabled.
Creating topic foo-bar and consumer foo-bar.

Topic search:

1. bar foo => Found (Ok)
2. bar-foo => Found (not anticipated)
3. bar*foo=> Not found (Ok)
4. foo*bar => Not found (not anticipated)

Consumer search:

1. bar foo => Found (Ok)
2. bar-foo => Found (not anticipated)
3. bar*foo=> Found (not anticipated)
4. foo*bar => Found (Ok)

So, multi word search works.
Hyphen -is treated as whitespace in query, how to avoid?
Wildcard * seems to be treated a literal character in Topic search - expected wildcard behavior.
Wildcard * seems to be ignored in Consumer search - expected wildcard behavior.

I have not created a bug, because there is no "by design" documentation to lean on.

[germanosin]

@germanosin Here is my test: "Apply full text search" button enabled. Creating topic foo-bar and consumer foo-bar.

Topic search:

1. bar foo => Found (Ok)
2. bar-foo => Found (not anticipated)
3. bar*foo=> Not found (Ok)
4. foo*bar => Not found (not anticipated)

Consumer search:

1. bar foo => Found (Ok)
2. bar-foo => Found (not anticipated)
3. bar*foo=> Found (not anticipated)
4. foo*bar => Found (Ok)

So, multi word search works. Hyphen -is treated as whitespace in query, how to avoid? Wildcard * seems to be treated a literal character in Topic search - expected wildcard behavior. Wildcard * seems to be ignored in Consumer search - expected wildcard behavior.

I have not created a bug, because there is no "by design" documentation to lean on.

The full-text search algorithm is fairly straightforward. It splits names into parts based on spaces, special characters (such as dashes or underscores), and case changes. When you perform a search, your query is split the same way, and the system looks for name segments that begin with those query fragments.

[th0ger]

@germanosin thanks for explaining. Why does is * treated as whitespace/tokensplitter in topic search but literal character in Consumer search?

[germanosin]

@germanosin thanks for explaining. Why does is * treated as whitespace/tokensplitter in topic search but literal character in Consumer search?

topics use lucene based indexing
consumer groups / acls / ... use ngram filters

[th0ger]

Thanks @germanosin.
Why this difference? This is mildly confusing :-)
Docs should mention how the two search methods work.

Thanks for putting effort into this great product!