From bf7f465e7a982c41df40cfa83aa7fa97d2de2b01 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Tue, 28 Jan 2025 10:54:09 -0800 Subject: [PATCH 1/6] Automatic Import: CEL Input --- .../getting-started/automatic-import.asciidoc | 15 ++++++++++++++- docs/serverless/ingest/auto-import.asciidoc | 19 ++++++++++++++----- 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/docs/getting-started/automatic-import.asciidoc b/docs/getting-started/automatic-import.asciidoc index 9ff23fde5c..9a3729524b 100644 --- a/docs/getting-started/automatic-import.asciidoc +++ b/docs/getting-started/automatic-import.asciidoc @@ -22,6 +22,7 @@ TIP: Click https://elastic.navattic.com/automatic-import[here] to access an inte - A working <>. Recommended models: `Claude 3.5 Sonnet`; `GPT-4o`; `Gemini-1.5-pro-002`. - An https://www.elastic.co/pricing[Enterprise] subscription. - A sample of the data you want to import, in a structured or unstructured format (including JSON, NDJSON, and Syslog). +- To import data from a REST API, have its OpenAPI specification (OAS) file ready. -- IMPORTANT: Using Automatic Import allows users to create new third-party data integrations through the use of third-party generative AI models (“GAI models”). Any third-party GAI models that you choose to use are owned and operated by their respective providers. Elastic does not own or control these third-party GAI models, nor does it influence their design, training, or data-handling practices. Using third-party GAI models with Elastic solutions, and using your data with third-party GAI models is at your discretion. Elastic bears no responsibility or liability for the content, operation, or use of these third-party GAI models, nor for any potential loss or damage arising from their use. Users are advised to exercise caution when using GAI models with personal, sensitive, or confidential information, as data submitted may be used to train the models or for other purposes. Elastic recommends familiarizing yourself with the development practices and terms of use of any third-party GAI models before use. You are responsible for ensuring that your use of Automatic Import complies with the terms and conditions of any third-party platform you connect with. @@ -40,6 +41,13 @@ image::images/auto-import-create-new-integration-button.png[The Integrations pag 6. Define your integration's package name, which will prefix the imported event fields. 7. Define your **Data stream title**, **Data stream description**, and **Data stream name**. These fields appear on the integration's configuration page to help identify the data stream it writes to. 8. Select your {filebeat-ref}/configuration-filebeat-options.html[**Data collection method**]. This determines how your new integration will ingest the data (for example, from an S3 bucket, an HTTP endpoint, or a file stream). ++ +.Importing CEL data +[NOTE] +==== +If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume, and your authentication method, before uploading your sample data. +==== ++ 9. Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle. + .Best practices for sample data @@ -57,6 +65,12 @@ image::images/auto-import-review-integration-page.png[The Automatic Import Revie + 12. (Optional) After reviewing the proposed pipeline, you can fine-tune it by clicking **Edit pipeline**. Refer to the <> to learn more about formatting field mappings. When you're satisfied with your changes, click **Save**. + +[NOTE] +.How to edit a CEL program +==== +If your new integration collects CEL data, you must install it before you can edit its CEL program. After installation, you can update the CEL program and API authentication information from the new integration's integration policy. +==== ++ image::images/auto-import-edit-pipeline.gif[A gif showing the user clicking the edit pipeline button and viewing the ingest pipeline flyout] + 13. Click **Add to Elastic**. After the **Success** message appears, your new integration will be available on the Integrations page. @@ -65,6 +79,5 @@ image::images/auto-import-success-message.png[The automatic import success messa + 14. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. -NOTE: Once you've added an integration, you can't edit any details other than the ingest pipeline, which you can edit by going to **Stack Management → Ingest Pipelines**. TIP: You can use the <> to check the health of your data ingest pipelines and field mappings. \ No newline at end of file diff --git a/docs/serverless/ingest/auto-import.asciidoc b/docs/serverless/ingest/auto-import.asciidoc index 2cbc100c9a..b33e5d6900 100644 --- a/docs/serverless/ingest/auto-import.asciidoc +++ b/docs/serverless/ingest/auto-import.asciidoc @@ -26,6 +26,7 @@ Click https://elastic.navattic.com/automatic-import[here] to access an interacti * A working <>. Recommended models: `Claude 3.5 Sonnet`; `GPT-4o`; `Gemini-1.5-pro-002`. * A https://www.elastic.co/pricing/serverless-security[Security Analytics Complete] subscription. * A sample of the data you want to import, in a structured or unstructured format (such as JSON, NDJSON, or Syslog). +* To import data from a REST API, have its OpenAPI specification (OAS) file ready. ==== [IMPORTANT] @@ -50,6 +51,13 @@ image:images/auto-import-create-new-integration-button.png[The Integrations page . Define your integration's package name, which will prefix the imported event fields. . Define your **Data stream title**, **Data stream description**, and **Data stream name**. These fields appear on the integration's configuration page to help identify the data stream it writes to. . Select your https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html[**Data collection method**]. This determines how your new integration will ingest the data (for example, from an S3 bucket, an HTTP endpoint, or a file stream). ++ +.Importing CEL data +[NOTE] +==== +If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume, and your authentication method, before uploading your sample data. +==== ++ . Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle. + .Best practices for sample data @@ -67,6 +75,12 @@ image:images/auto-import-create-new-integration-button.png[The Integrations page image:images/auto-import-review-integration-page.png[The Automatic Import Review page showing proposed field mappings] . (Optional) After reviewing the proposed pipeline, you can fine-tune it by clicking **Edit pipeline**. Refer to the https://www.elastic.co/guide/en/security/current/siem-field-reference.html[{elastic-sec} ECS reference] to learn more about formatting field mappings. When you're satisfied with your changes, click **Save**. + +[NOTE] +.How to edit a CEL program +==== +If your new integration collects CEL data, you must install it before you can edit its CEL program. After installation, you can update the CEL program and API authentication information from the new integration's integration policy. +==== ++ [role="screenshot"] image:images/auto-import-edit-pipeline.gif[A gif showing the user clicking the edit pipeline button and viewing the ingest pipeline flyout] . Click **Add to Elastic**. After the **Success** message appears, your new integration will be available on the Integrations page. @@ -75,11 +89,6 @@ image:images/auto-import-edit-pipeline.gif[A gif showing the user clicking the e image:images/auto-import-success-message.png[The Automatic Import success message] . Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. -[NOTE] -==== -Once you've added an integration, you can't edit any details other than the ingest pipeline, which you can edit by going to **Project Settings → Stack Management → Ingest Pipelines**. -==== - [TIP] ==== You can use the <> to check the health of your data ingest pipelines and field mappings. From 454d9ec2271832a50b91cca9b1bd3e42a7e6aedb Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Tue, 28 Jan 2025 11:48:21 -0800 Subject: [PATCH 2/6] Update docs/getting-started/automatic-import.asciidoc Co-authored-by: Kylie Meli --- docs/getting-started/automatic-import.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/automatic-import.asciidoc b/docs/getting-started/automatic-import.asciidoc index 9a3729524b..45592724b0 100644 --- a/docs/getting-started/automatic-import.asciidoc +++ b/docs/getting-started/automatic-import.asciidoc @@ -68,7 +68,7 @@ image::images/auto-import-review-integration-page.png[The Automatic Import Revie [NOTE] .How to edit a CEL program ==== -If your new integration collects CEL data, you must install it before you can edit its CEL program. After installation, you can update the CEL program and API authentication information from the new integration's integration policy. +If your new integration collects data from an API, you can update the CEL input configuration (program and API authentication information) from the new integration's integration policy. ==== + image::images/auto-import-edit-pipeline.gif[A gif showing the user clicking the edit pipeline button and viewing the ingest pipeline flyout] From 36c4504d6bda28d5435d67bfccc7901f1777b6de Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Tue, 28 Jan 2025 12:00:00 -0800 Subject: [PATCH 3/6] incorporates Kylie's review --- docs/getting-started/automatic-import.asciidoc | 2 ++ docs/serverless/ingest/auto-import.asciidoc | 9 ++++----- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/automatic-import.asciidoc b/docs/getting-started/automatic-import.asciidoc index 45592724b0..35eb9badc5 100644 --- a/docs/getting-started/automatic-import.asciidoc +++ b/docs/getting-started/automatic-import.asciidoc @@ -80,4 +80,6 @@ image::images/auto-import-success-message.png[The automatic import success messa 14. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. +TIP: Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. + TIP: You can use the <> to check the health of your data ingest pipelines and field mappings. \ No newline at end of file diff --git a/docs/serverless/ingest/auto-import.asciidoc b/docs/serverless/ingest/auto-import.asciidoc index b33e5d6900..456098ed3a 100644 --- a/docs/serverless/ingest/auto-import.asciidoc +++ b/docs/serverless/ingest/auto-import.asciidoc @@ -87,9 +87,8 @@ image:images/auto-import-edit-pipeline.gif[A gif showing the user clicking the e + [role="screenshot"] image:images/auto-import-success-message.png[The Automatic Import success message] -. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. +. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. -[TIP] -==== -You can use the <> to check the health of your data ingest pipelines and field mappings. -==== +TIP: Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. + +TIP: You can use the <> to check the health of your data ingest pipelines and field mappings. \ No newline at end of file From 516af61b00784635f45a5685d184910078096824 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Tue, 28 Jan 2025 12:31:24 -0800 Subject: [PATCH 4/6] Update docs/getting-started/automatic-import.asciidoc Co-authored-by: Kylie Meli --- docs/getting-started/automatic-import.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/automatic-import.asciidoc b/docs/getting-started/automatic-import.asciidoc index 35eb9badc5..0c35691297 100644 --- a/docs/getting-started/automatic-import.asciidoc +++ b/docs/getting-started/automatic-import.asciidoc @@ -68,7 +68,7 @@ image::images/auto-import-review-integration-page.png[The Automatic Import Revie [NOTE] .How to edit a CEL program ==== -If your new integration collects data from an API, you can update the CEL input configuration (program and API authentication information) from the new integration's integration policy. +If your new integration collects data from an API, you can update the CEL input configuration (program and API authentication information) from the new integration's integration policy. ==== + image::images/auto-import-edit-pipeline.gif[A gif showing the user clicking the edit pipeline button and viewing the ingest pipeline flyout] From d82c8a6695ae5f0cd7d6b0658a1b0a16c37694a6 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Tue, 28 Jan 2025 12:31:32 -0800 Subject: [PATCH 5/6] Update docs/serverless/ingest/auto-import.asciidoc Co-authored-by: Kylie Meli --- docs/serverless/ingest/auto-import.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/serverless/ingest/auto-import.asciidoc b/docs/serverless/ingest/auto-import.asciidoc index 456098ed3a..03d90bcbc1 100644 --- a/docs/serverless/ingest/auto-import.asciidoc +++ b/docs/serverless/ingest/auto-import.asciidoc @@ -78,7 +78,7 @@ image:images/auto-import-review-integration-page.png[The Automatic Import Review [NOTE] .How to edit a CEL program ==== -If your new integration collects CEL data, you must install it before you can edit its CEL program. After installation, you can update the CEL program and API authentication information from the new integration's integration policy. +If your new integration collects data from an API, you can update the CEL input configuration (program and API authentication information) from the new integration's integration policy. ==== + [role="screenshot"] From 628322c213d2a673046781fd9b0f04b4063eafcd Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Tue, 28 Jan 2025 12:37:58 -0800 Subject: [PATCH 6/6] Incorporates Nastasha's review --- docs/getting-started/automatic-import.asciidoc | 6 ++---- docs/serverless/ingest/auto-import.asciidoc | 5 ++--- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/getting-started/automatic-import.asciidoc b/docs/getting-started/automatic-import.asciidoc index 0c35691297..29ef53bef0 100644 --- a/docs/getting-started/automatic-import.asciidoc +++ b/docs/getting-started/automatic-import.asciidoc @@ -45,7 +45,7 @@ image::images/auto-import-create-new-integration-button.png[The Integrations pag .Importing CEL data [NOTE] ==== -If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume, and your authentication method, before uploading your sample data. +If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume and your authentication method before uploading your sample data. ==== + 9. Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle. @@ -78,8 +78,6 @@ image::images/auto-import-edit-pipeline.gif[A gif showing the user clicking the image::images/auto-import-success-message.png[The automatic import success message] + 14. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. - - -TIP: Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. +15. (Optional) Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. TIP: You can use the <> to check the health of your data ingest pipelines and field mappings. \ No newline at end of file diff --git a/docs/serverless/ingest/auto-import.asciidoc b/docs/serverless/ingest/auto-import.asciidoc index 03d90bcbc1..cfd187a853 100644 --- a/docs/serverless/ingest/auto-import.asciidoc +++ b/docs/serverless/ingest/auto-import.asciidoc @@ -55,7 +55,7 @@ image:images/auto-import-create-new-integration-button.png[The Integrations page .Importing CEL data [NOTE] ==== -If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume, and your authentication method, before uploading your sample data. +If you select *API (CEL input)*, you'll have the additional option to upload the API's OAS file here. After you do, the LLM will use it to determine which API endpoints (GET only), query parameters, and data structures to use in the new custom integration. You will then select which API endpoints to consume and your authentication method before uploading your sample data. ==== + . Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle. @@ -88,7 +88,6 @@ image:images/auto-import-edit-pipeline.gif[A gif showing the user clicking the e [role="screenshot"] image:images/auto-import-success-message.png[The Automatic Import success message] . Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration. - -TIP: Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. +. (Optional) Once you've added an integration, you can edit the ingest pipeline by going to **Project Settings → Stack Management → Ingest Pipelines**. TIP: You can use the <> to check the health of your data ingest pipelines and field mappings. \ No newline at end of file