Added new fields API, reworked section

Author: philippkahr

manage-data/ingest/transform-enrich/readable-maintainable-ingest-pipelines.md

@@ -1,3 +1,4 @@
+
 ---
 mapped_pages:
   - https://www.elastic.co/docs/manage-data/ingest/transform-enrich/common-mistakes.html
@@ -18,75 +19,81 @@ This guide does not provide guidance on optimizing for ingest pipeline performan
 
 When creating ingest pipelines, there are are few options for accessing fields in conditional statements and scripts. All formats can be used to reference fields, so choose the one that makes your pipeline easier to read and maintain.
 
-| Notation | Example | Notes |
-|---|---|---|
-| Dot notation | `ctx.event.action` | Supported in conditionals and painless scripts. |
-| Square bracket notation | `ctx['event']['action']` | Supported in conditionals and painless scripts. |
-| Mixed dot and bracket notation | `ctx.event['action']` | Supported in conditionals and painless scripts. |
+| Notation                       | Example                                               | Notes                                                                           |
+| ------------------------------ | ----------------------------------------------------- | ------------------------------------------------------------------------------- |
+| Dot notation                   | `ctx.event.action`                                    | Supported in conditionals and painless scripts.                                 |
+| Square bracket notation        | `ctx['event']['action']`                              | Supported in conditionals and painless scripts.                                 |
+| Mixed dot and bracket notation | `ctx.event['action']`                                 | Supported in conditionals and painless scripts.                                 |
+| Field API                      | `field('event.action', '')` or `$('event.action','')` | Supported in conditionals and painless scripts. Only available in versions 9.2+ |
+| Field API                      | `field('event.action', '')` or `$('event.action','')` | Supported only in painless scripts.                                             |
 
 Below are some general guidelines for choosing the right option in a situation.
 
+### Field API
+
+Starting with version [9.2](https://github.com/elastic/elasticsearch/pull/131581) we have access to the field API that enables the usage of this API in conditionals (the `if` statement of your processor). Otherwise you can always use the field API in the script processor itself.
+
+:::{note}
+This is the preferred way to access fields.
+:::
+
+**Benefits**
+
+- Clean and easy to read
+- Handles null values automatically
+- Adds support for additional functions like `isEmpty()` to ease comparisions.
+- Handles dots as part of field name
+- Handles dots as dot walking for object notation
+- Handles special characters.
+
+**Limitations**
+
+- Only available starting in 9.2 for conditionals.
+
 ### Dot notation [dot-notation]
 
-**Benefits:**
-* Clean and easy to read.
-* Supports null safety operations `?`. Read more in [Use null safe operators (`?.`)](#null-safe-operators).
+**Benefits**
+
+- Clean and easy to read.
+- Supports null safety operations `?`. Read more in [Use null safe operators (`?.`)](#null-safe-operators).
 
 **Limitations**
-* Does not support field names that contain a `.` or any special characters such as `@`.
+
+- Does not support field names that contain a `.` or any special characters such as `@`.
   Use [Bracket notation](#bracket-notation) instead.
 
 ### Bracket notation [bracket-notation]
 
-**Benefits:**
-* Supports special characters such as `@` in the field name.
+**Benefits**
+
+- Supports special characters such as `@` in the field name.
   For example, if there's a field name called `has@!%&chars`, you would use `ctx['has@!%&chars']`.
-* Supports field names that contain `.`.
+- Supports field names that contain `.`.
   For example, if there's a field named `foo.bar`, if you used `ctx.foo.bar` it will try to access the field `bar` in the object `foo` in the object `ctx`. If you used `ctx['foo.bar']` it can access the field directly.
 
-**Limitations:**
-* Slightly more verbose than dot notation.
-* No support for null safety operations `?`.
+**Limitations**
+
+- Slightly more verbose than dot notation.
+- No support for null safety operations `?`.
   Use [Dot notation](#dot-notation) instead.
 
 ### Mixed dot and bracket notation
 
-**Benefits:**
-* You can also mix dot notation and bracket notation to take advantage of the benefits of both formats.
+**Benefits**
+
+- You can also mix dot notation and bracket notation to take advantage of the benefits of both formats.
   For example, you could use `ctx.my.nested.object['has@!%&chars']`. Then you can use the `?` operator on the fields using dot notation while still accessing a field with a name that contains special characters: `ctx.my?.nested?.object['has@!%&chars']`.
 
-**Limitations:**
-* Slightly more difficult to read.
+**Limitations**
 
+- Slightly more difficult to read.
 
 ## Write concise conditionals (`if` statements) [conditionals]
 
 Use conditionals (`if` statements) to ensure that an ingest pipeline processor is only applied when specific conditions are met.
 
 % In an ingest pipeline, when working with conditionals inside processors. The topic around error processing is a bit more complex, most importantly any errors that are coming from null values, missing keys, missing values, inside the conditional, will lead to an error that is not captured by the `ignore_failure` handler and will exit the pipeline.
 
-### Avoid excessive OR conditions
-
-When using the [boolean OR operator](elasticsearch://reference/scripting-languages/painless/painless-operators-boolean.md#boolean-or-operator) (`||`), `if` conditions can become unnecessarily complex and difficult to maintain, especially when chaining many OR checks. Instead, consider using array-based checks like `.contains()` to simplify your logic and improve readability.
-
-#### ![ ](../../images/icon-cross.svg) **Don't**: Run many ORs
-
-```painless
-"if": "ctx?.kubernetes?.container?.name == 'admin' || ctx?.kubernetes?.container?.name == 'def'
-|| ctx?.kubernetes?.container?.name == 'demo' || ctx?.kubernetes?.container?.name == 'acme'
-|| ctx?.kubernetes?.container?.name == 'wonderful'
-```
-
-#### ![ ](../../images/icon-check.svg) **Do**: Use contains to compare
-
-```painless
-["admin","def","demo","acme","wonderful"].contains(ctx.kubernetes?.container?.name)
-```
-
-:::{tip}
-This example only checks for exact matches. Do not use this approach if you need to check for partial matches.
-:::
-
 ### Use null safe operators (`?.`) [null-safe-operators]
 
 Anticipate potential problems with the data, and use the [null safe operator](elasticsearch://reference/scripting-languages/painless/painless-operators-reference.md#null-safe-operator) (`?.`) to prevent data from being processed incorrectly.
@@ -266,6 +273,29 @@ POST _ingest/pipeline/_simulate
   }
 }
 ```
+
+:::
+
+### Avoid excessive OR conditions
+
+When using the [boolean OR operator](elasticsearch://reference/scripting-languages/painless/painless-operators-boolean.md#boolean-or-operator) (`||`), `if` conditions can become unnecessarily complex and difficult to maintain, especially when chaining many OR checks. Instead, consider using array-based checks like `.contains()` to simplify your logic and improve readability.
+
+#### ![ ](../../images/icon-cross.svg) **Don't**: Run many ORs
+
+```painless
+"if": "ctx?.kubernetes?.container?.name == 'admin' || ctx?.kubernetes?.container?.name == 'def'
+|| ctx?.kubernetes?.container?.name == 'demo' || ctx?.kubernetes?.container?.name == 'acme'
+|| ctx?.kubernetes?.container?.name == 'wonderful'
+```
+
+#### ![ ](../../images/icon-check.svg) **Do**: Use contains to compare
+
+```painless
+["admin","def","demo","acme","wonderful"].contains(ctx.kubernetes?.container?.name)
+```
+
+:::{tip}
+This example only checks for exact matches. Do not use this approach if you need to check for partial matches.
 :::
 
 ## Convert mb/gb values to bytes
@@ -345,10 +375,78 @@ The [rename processor](elasticsearch://reference/enrich-processor/rename-process
 - `ignore_missing`: Useful when you are not sure that the field you want to rename exists.
 - `ignore_failure`: Helps with any failures encountered. For example, the rename processor can only rename to non-existing fields. If you already have the field `abc` and you want to rename `def` to `abc`, the operation will fail.
 
-## Use a script processor
+## Script processor
 
 If no built-in processor can achieve your goal, you may need to use a [script processor](elasticsearch://reference/enrich-processor/script-processor.md) in your ingest pipeline. Be sure to write scripts that are clear, concise, and maintainable.
 
+### Add new fields
+
+All of the above discussed ways to [access fields](#access-fields) and retrieve their values is applicable within the script context. [Null handling](#null-safe-operators) is still an important aspect when accessing the fields.
+
+:::{tip}
+The fields API is the recommended way to add new fields.
+:::
+
+**Fields API**
+We get the following field `cpu.usage` and we want to rename it to `system.cpu.total.norm.pct` which represents a scale from 0-1.0, where 1 is the equivalent of 100%.
+
+```json
+POST _ingest/pipeline/_simulate
+{
+  "docs": [
+    {
+      "_source": {
+        "cpu": {
+          "usage": 90 <1>
+        }
+      }
+    }
+  ],
+  "pipeline": {
+    "processors": [
+      {
+        "script": {
+          "source": """
+            field('system.cpu.total.norm.pct').set($('cpu.usage',0.0)/100.0) <2>
+          """
+        }
+      }
+    ]
+  }
+}
+```
+1. Our field expects 0-1 and not 0-100, we will have to divide by 100 to get the right representation.
+2. The `field` API is exposed as `field(<field name>)`. The `set(<value>)` is responsible for setting the value. Inside we use the `$(<field name>, fallback)` to read the value out of the existing field. Lastly we divide by `100.0`. The `.0` is important, otherwise it will perform an integer only division and return just 0 instead of 0.9.
+
+**No fields API**
+Without the field API this can also be achieved. However there is much more code involved, as we have to ensure that we can walk the full path of `system.cpu.total.norm.pct`.
+
+```json
+{
+  "script": {
+    "source": "
+      if(ctx.system == null){ <1>
+        ctx.system = new HashMap(); <2>
+      }
+      if(ctx.system.cpu == null){
+        ctx.system.cpu = [:]; <3>
+      }
+      if(ctx.system.cpu.total == null){
+        ctx.system.cpu.total = [:];
+      }
+      if(ctx.system.cpu.total.norm == null){
+        ctx.system.cpu.total.norm = [:];
+      }
+      ctx.system.cpu.total.norm.pct = $('cpu.usage', 0.0)/100.0; <4>
+      "
+  }
+}
+```
+1. We need to check whether the objects are null or not and then create them.
+2. We create a new HashMap to store all the objects in it.
+3. Instead of writing `new HashMap()` we can use the shortcut `[:]`.
+4. We perform the same calculation as above and set the value.
+
 ### Calculate `event.duration` in a complex manner
 
 #### ![ ](../../images/icon-cross.svg) **Don't**: Use verbose and error-prone scripting patterns
@@ -364,6 +462,7 @@ If no built-in processor can achieve your goal, you may need to use a [script pr
   }
 }
 ```
+
 1. Avoid accessing fields using square brackets instead of dot notation.
 2. `ctx['event']['duration']`: Do not attempt to access child properties without ensuring the parent property exists.
 3. `timeString.substring(0,2)`: Avoid parsing substrings manually instead of leveraging date/time parsing utilities.
@@ -405,6 +504,7 @@ POST _ingest/pipeline/_simulate
   }
 }
 ```
+
 1. Ensure the `event` object exists before assigning to it.
 2. Use `DateTimeFormatter` and `LocalTime` to parse the duration string.
 3. Store the duration in nanoseconds, as expected by ECS.
@@ -428,14 +528,14 @@ When reconstructing or normalizing IP addresses in ingest pipelines, avoid unnec
   }
 }
 ```
+
 1. Uses square bracket notation for field access instead of dot notation.
 2. Unnecessary casting to `Integer` when parsing string segments.
 3. Allocates an extra variable for the IP string instead of setting the field directly.
 4. Does not check if `destination` is available as an object.
 
 #### ![ ](../../images/icon-check.svg) **Do**: Use concise, readable, and safe scripting
 
-
 ```json
 POST _ingest/pipeline/_simulate
 {
@@ -463,6 +563,7 @@ POST _ingest/pipeline/_simulate
   }
 }
 ```
+
 1. Uses dot notation for field access.
 2. Avoids unnecessary casting and extra variables.
 3. Uses the null safe operator (`?.`) to check for field existence.
@@ -546,3 +647,31 @@ POST _ingest/pipeline/_simulate
 ```
 
 In this example, `{{tags.0}}` retrieves the first element of the `tags` array (`"cool-host"`) and assigns it to the `host.alias` field. This approach is necessary when you want to extract a specific value from an array for use elsewhere in your document. Using the correct index ensures you get the intended value, and this pattern works for any array field in your source data.
+
+### Transform into a JSON string
+
+Whenever you need to store the original `_source` within a field `event.original`, we can use mustache function `{{#toJson}}<field>{{/toJson}}`.
+
+```json
+POST _ingest/pipeline/_simulate
+{
+  "docs": [
+    {
+      "_source": {
+        "foo": "bar",
+        "key": 123
+      }
+    }
+  ],
+  "pipeline": {
+    "processors": [
+      {
+        "set": {
+          "field": "event.original",
+          "value": "{{#toJson}}_source{{/toJson}}"
+        }
+      }
+    ]
+  }
+}
+```