[Security Solution] Fix "too many clauses" error on prebuilt rules installation page #223240
+120
−52
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikitaindik
force-pushed
the
fix-prebuilt-rules-installation-clauses-error
branch
from
ae5f001 to
7937d56
Compare
nikitaindik
requested review from
jkelas and
maximpn
and removed request for
jkelas
nikitaindik
added
Team:Detections and Resp
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
nikitaindik
added
bug
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
nikitaindik
added
v9.1.0
v8.19.0
v9.0.3
v8.18.3
backport:version
maximpn
requested changes
Jun 12, 2025
There was a problem hiding this comment.
@nikitaindik and I Zoomed over this PR. We agreed that having better encapsulation of the chunking logic will help to improve maintainability. On top of that the clauses number calculation happens at Lucene codebase make it opaque for Kibana. The following should be done
- filter chunking logic should be moved out to a separate function accepting items, a mapper function and the number of clauses per item
- it's good to have a function to perform an action on chunks like
fetchLatestVersionInfo()or find some results - extra comments are required to provide better understanding of the problem
…installation-clauses-error
|
@maximpn Refactored as per our discussion over Zoom. Please take a look. |
💚 Build Succeeded
Metrics [docs]
History
cc @nikitaindik |
maximpn
approved these changes
Jun 17, 2025
There was a problem hiding this comment.
@nikitaindik Thanks for making improvements to the implementation 🙏
The renewed code looks shorter and cleaner 👍
|
Starting backport for target branches: 8.18, 8.19, 9.0 https://github.com/elastic/kibana/actions/runs/15709754685 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 17, 2025
…stallation page (elastic#223240) **Resolves: elastic#223399 ## Summary This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from `main` branch and going to the "Add Elastic rules" page. <img width="1741" alt="Screenshot 2025-06-10 at 11 28 19" src="https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6" /> ## Changes PR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES into smaller chunks to avoid the error. ## Cause Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error. More specifically, `/prebuilt_rules/installation/_review` route handler calls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger than `maxClauseCount`. `maxClauseCount` value is computed dynamically by ES and its size depends on hardware and available resources ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html), [code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)). The minimum value for `maxClauseCount` is 1024. ## Why it didn't fail before Two reasons: 1. ES changed how `maxClauseCount` is computed. They've recently merged a [PR](elastic/elasticsearch#128293) that made queries against numeric types count three times towards the `maxClauseCount` limit. They plan to revert the change in [this PR](elastic/elasticsearch#129206). 2. Prebuilt rule packages are growing bigger with each version, resulting in a bigger number of clauses. I've tested behaviour with ES change in place on different package versions: - 8.17.1 (contains 1262 rule versions) - no "too many clauses" error - 8.18.1 (contains 1356 rule versions) - causes "too many clauses" error - 9.0.1 (also contains 1356 rule versions) - causes "too many clauses" error The precise number of versions that start to cause errors is 1293 on my laptop. So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes. (cherry picked from commit 482953d)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
nikitaindik
added a commit
to nikitaindik/kibana
that referenced
this pull request
Jun 17, 2025
…stallation page (elastic#223240) **Resolves: elastic#223399 ## Summary This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from `main` branch and going to the "Add Elastic rules" page. <img width="1741" alt="Screenshot 2025-06-10 at 11 28 19" src="https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6" /> ## Changes PR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES into smaller chunks to avoid the error. ## Cause Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error. More specifically, `/prebuilt_rules/installation/_review` route handler calls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger than `maxClauseCount`. `maxClauseCount` value is computed dynamically by ES and its size depends on hardware and available resources ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html), [code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)). The minimum value for `maxClauseCount` is 1024. ## Why it didn't fail before Two reasons: 1. ES changed how `maxClauseCount` is computed. They've recently merged a [PR](elastic/elasticsearch#128293) that made queries against numeric types count three times towards the `maxClauseCount` limit. They plan to revert the change in [this PR](elastic/elasticsearch#129206). 2. Prebuilt rule packages are growing bigger with each version, resulting in a bigger number of clauses. I've tested behaviour with ES change in place on different package versions: - 8.17.1 (contains 1262 rule versions) - no "too many clauses" error - 8.18.1 (contains 1356 rule versions) - causes "too many clauses" error - 9.0.1 (also contains 1356 rule versions) - causes "too many clauses" error The precise number of versions that start to cause errors is 1293 on my laptop. So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes. (cherry picked from commit 482953d) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
nikitaindik
added a commit
to nikitaindik/kibana
that referenced
this pull request
Jun 17, 2025
…stallation page (elastic#223240) **Resolves: elastic#223399 ## Summary This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from `main` branch and going to the "Add Elastic rules" page. <img width="1741" alt="Screenshot 2025-06-10 at 11 28 19" src="https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6" /> ## Changes PR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES into smaller chunks to avoid the error. ## Cause Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error. More specifically, `/prebuilt_rules/installation/_review` route handler calls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger than `maxClauseCount`. `maxClauseCount` value is computed dynamically by ES and its size depends on hardware and available resources ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html), [code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)). The minimum value for `maxClauseCount` is 1024. ## Why it didn't fail before Two reasons: 1. ES changed how `maxClauseCount` is computed. They've recently merged a [PR](elastic/elasticsearch#128293) that made queries against numeric types count three times towards the `maxClauseCount` limit. They plan to revert the change in [this PR](elastic/elasticsearch#129206). 2. Prebuilt rule packages are growing bigger with each version, resulting in a bigger number of clauses. I've tested behaviour with ES change in place on different package versions: - 8.17.1 (contains 1262 rule versions) - no "too many clauses" error - 8.18.1 (contains 1356 rule versions) - causes "too many clauses" error - 9.0.1 (also contains 1356 rule versions) - causes "too many clauses" error The precise number of versions that start to cause errors is 1293 on my laptop. So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes. (cherry picked from commit 482953d) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts
kibanamachine
added a commit
that referenced
this pull request
Jun 17, 2025
…ules installation page (#223240) (#224269) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Fix "too many clauses" error on prebuilt rules installation page (#223240)](#223240) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-17T14:14:56Z","message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page","number":223240,"url":"https://github.com/elastic/kibana/pull/223240","mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.0","8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/223240","number":223240,"mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Nikita Indik <[email protected]>
nikitaindik
added a commit
that referenced
this pull request
Jun 18, 2025
…les installation page (#223240) (#224282) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Fix "too many clauses" error on prebuilt rules installation page (#223240)](#223240) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-17T14:14:56Z","message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page","number":223240,"url":"https://github.com/elastic/kibana/pull/223240","mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/223240","number":223240,"mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/224269","number":224269,"state":"OPEN"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
nikitaindik
added a commit
that referenced
this pull request
Jun 18, 2025
…ules installation page (#223240) (#224283) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Fix "too many clauses" error on prebuilt rules installation page (#223240)](#223240) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-17T14:14:56Z","message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v9.1.0","v8.19.0","v9.0.3","v8.18.3"],"title":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page","number":223240,"url":"https://github.com/elastic/kibana/pull/223240","mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},"sourceBranch":"main","suggestedTargetBranches":["8.18"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/223240","number":223240,"mergeCommit":{"message":"[Security Solution] Fix \"too many clauses\" error on prebuilt rules installation page (#223240)\n\n**Resolves: https://github.com/elastic/kibana/issues/223399**\n\n## Summary\nThis PR fixes an error on the \"Add Elastic rules\" page. The error is\nshown when running a local dev environment from `main` branch and going\nto the \"Add Elastic rules\" page.\n\n<img width=\"1741\" alt=\"Screenshot 2025-06-10 at 11 28 19\"\nsrc=\"https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6\"\n/>\n\n## Changes\nPR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES\ninto smaller chunks to avoid the error.\n\n## Cause\nKibana makes a search request to ES with a filter that has too many\nclauses, so ES rejects with an error.\n\nMore specifically, `/prebuilt_rules/installation/_review` route handler\ncalls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all\ninstallable rules. To do this, we construct a request with thousands of\nclauses in a filter. ES counts the number of clauses in a filter and\nrejects because it's bigger than `maxClauseCount`. `maxClauseCount`\nvalue is computed dynamically by ES and its size depends on hardware and\navailable resources\n([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html),\n[code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)).\nThe minimum value for `maxClauseCount` is 1024.\n\n## Why it didn't fail before\nTwo reasons:\n1. ES changed how `maxClauseCount` is computed. They've recently merged\na [PR](elastic/elasticsearch#128293) that made\nqueries against numeric types count three times towards the\n`maxClauseCount` limit. They plan to revert the change in [this\nPR](https://github.com/elastic/elasticsearch/pull/129206).\n2. Prebuilt rule packages are growing bigger with each version,\nresulting in a bigger number of clauses. I've tested behaviour with ES\nchange in place on different package versions:\n- 8.17.1 (contains 1262 rule versions) - no \"too many clauses\" error\n- 8.18.1 (contains 1356 rule versions) - causes \"too many clauses\" error\n- 9.0.1 (also contains 1356 rule versions) - causes \"too many clauses\"\nerror\nThe precise number of versions that start to cause errors is 1293 on my\nlaptop.\n\nSo even if ES team rolls back their change, we still need to make sure\nwe don't go over the limit with ever-growing prebuilt rule package\nsizes.","sha":"482953ddc5a9e1494a3182c9cedfa4214179a297"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/224269","number":224269,"state":"OPEN"},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/224282","number":224282,"state":"OPEN"},{"branch":"8.18","label":"v8.18.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This was referenced Jun 24, 2025
Closed
banderror
added
release_note:fix
and removed
release_note:skip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: #223399
Summary
This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from
mainbranch and going to the "Add Elastic rules" page.Changes
PR updates methods of
PrebuiltRuleAssetsClientto split requests to ES into smaller chunks to avoid the error.Cause
Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error.
More specifically,
/prebuilt_rules/installation/_reviewroute handler callsPrebuiltRuleAssetsClient.fetchAssetsByVersionto fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger thanmaxClauseCount.maxClauseCountvalue is computed dynamically by ES and its size depends on hardware and available resources (docs, code). The minimum value formaxClauseCountis 1024.Why it didn't fail before
Two reasons:
maxClauseCountis computed. They've recently merged a PR that made queries against numeric types count three times towards themaxClauseCountlimit. They plan to revert the change in this PR.The precise number of versions that start to cause errors is 1293 on my laptop.
So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes.