From 740e9dcc3a16b4529cd7b2bef7457f51e35ad586 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 28 Nov 2024 19:58:32 -0300 Subject: [PATCH 01/10] filters: lua: reorganize docs Signed-off-by: lecaros --- pipeline/filters/lua.md | 47 ++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 4f1bab092..189d87122 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -114,13 +114,9 @@ Each callback **must** return three values: | timestamp | double | If code equals 1, the original record timestamp will be replaced with this new value. | | record | table | If code equals 1, the original record information will be replaced with this new value. Note that the _record_ value **must** be a valid Lua table. This value can be an array of tables (i.e., array of objects in JSON format), and in that case the input record is effectively split into multiple records. (see below for more details) | -### Code Examples +## Features -For functional examples of this interface, please refer to the code samples provided in the source code of the project located here: - -[https://github.com/fluent/fluent-bit/tree/master/scripts](https://github.com/fluent/fluent-bit/tree/master/scripts) - -#### Inline configuration +### Inline configuration The [Fluent Bit smoke tests](https://github.com/fluent/fluent-bit/tree/master/packaging/testing/smoke/container) include examples to verify during CI. @@ -180,7 +176,22 @@ pipeline: {% endtab %} {% endtabs %} -#### Environment variable processing +### Number Type + ++Lua treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The `type_int_key` property is available. + +### Protected Mode + +Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html). + + +## Code Examples + +For functional examples of this interface, please refer to the code samples provided in the source code of the project located here: + +[https://github.com/fluent/fluent-bit/tree/master/scripts](https://github.com/fluent/fluent-bit/tree/master/scripts) + +### Processing environment variables As an example that combines a bit of LUA processing with the [Kubernetes filter](./kubernetes.md) that demonstrates using environment variables with LUA regex and substitutions. @@ -197,11 +208,11 @@ We want to extract the `sandboxbsh` name and add it to our record as a special k {% tab title="fluent-bit.conf" %} ``` [FILTER] - Name lua - Alias filter-iots-lua - Match iots_thread.* - Script filters.lua - Call set_landscape_deployment +Name lua +Alias filter-iots-lua +Match iots_thread.* +Script filters.lua +Call set_landscape_deployment ``` {% endtab %} @@ -244,14 +255,6 @@ filters.lua: end ``` -### Number Type - -+Lua treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The `type_int_key` property is available. - -### Protected Mode - -Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html). - ### Record Split The Lua callback function can return an array of tables (i.e., array of records) in its third _record_ return value. With this feature, the Lua filter can split one input record into multiple records according to custom logic. @@ -439,7 +442,6 @@ pipeline: In the output only the messages with response code 0 or greater than 399 are shown. - ### Timeformat Conversion The following example converts a field's specific type of `datetime` format to @@ -568,4 +570,5 @@ The output of this process shows the conversion of the `datetime` of two timezon [0] event_category_a: [[1722452186.727104902, {}], {"event"=>"Restock", "pub_date"=>"2024-07-30T18:01:06Z"}] [0] event_category_b: [[1722452186.730255842, {}], {"event"=>"Soldout", "pub_date"=>"2024-07-29T04:15:00Z"}] ... -``` \ No newline at end of file +``` + From b66c64e6ec9b081e397024fb8a8fc6c486bff885 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 28 Nov 2024 20:04:46 -0300 Subject: [PATCH 02/10] filters: lua: address typos Signed-off-by: lecaros --- pipeline/filters/lua.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 189d87122..7ef5a72ee 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -334,7 +334,7 @@ See also [Fluent Bit: PR 811](https://github.com/fluent/fluent-bit/pull/811). ### Response code filtering -In this example, we want to filter istio logs to exclude lines with response codes between 1 and 399. +In this example, we want to filter Istio logs to exclude lines with response codes between 1 and 399. Istio is configured to write the logs in json format. #### Lua script @@ -356,7 +356,7 @@ end #### Configuration -Configuration to get istio logs and apply response code filter to them. +Configuration to get Istio logs and apply response code filter to them. {% tabs %} {% tab title="fluent-bit.conf" %} @@ -442,7 +442,7 @@ pipeline: In the output only the messages with response code 0 or greater than 399 are shown. -### Timeformat Conversion +### Time format Conversion The following example converts a field's specific type of `datetime` format to `utc ISO 8601` format. From 07db988a6995fa021c886f832c6e97242d6f6de0 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 28 Nov 2024 20:11:15 -0300 Subject: [PATCH 03/10] filters: lua: add example to access configuration variables. Signed-off-by: lecaros --- pipeline/filters/lua.md | 50 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 7ef5a72ee..9519b2eb9 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -572,3 +572,53 @@ The output of this process shows the conversion of the `datetime` of two timezon ... ``` +### Using configuration variables + +Fluent Bit supports definition of configuration variables, which can be done in the following way: + +```yaml +env: + myvar1: myvalue1 +``` + +These variables can be accessed from the Lua code by referring to the FLB_ENV Lua table. +Being this a Lua table, the subrecords can be accessed following the same syntax, i.e. `FLB_ENV['A']`. + +#### Configuration + +```yaml +env: + A: aaa + B: bbb + C: ccc + +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: random + tag: test + samples: 10 + + filters: + - name: lua + match: "*" + call: append_tag + code: | + function append_tag(tag, timestamp, record) + new_record = record + new_record["my_env"] = FLB_ENV + return 1, timestamp, new_record + end + + outputs: + - name: stdout + match: "*" +``` + +#### Output + +test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}] + From 59ab88f9db12785dd640f9bea40c25c410234de9 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 28 Nov 2024 20:12:49 -0300 Subject: [PATCH 04/10] filters: lua: add format to output sample Signed-off-by: lecaros --- pipeline/filters/lua.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 9519b2eb9..0c3bee2b9 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -620,5 +620,5 @@ pipeline: #### Output -test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}] +`test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}]` From 665fedd644137a0d5f47b43c48758140f610e990 Mon Sep 17 00:00:00 2001 From: lecaros Date: Thu, 28 Nov 2024 20:15:24 -0300 Subject: [PATCH 05/10] filters: lua: add format to output sample Signed-off-by: lecaros --- pipeline/filters/lua.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 0c3bee2b9..95f6cd060 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -620,5 +620,6 @@ pipeline: #### Output -`test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}]` - +``` +test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}] +``` From 7a8cafed41f0031e8b33991ea4a66d37b671205a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Fri, 29 Nov 2024 18:26:15 -0300 Subject: [PATCH 06/10] filters: lua: add suggestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Pat Signed-off-by: José Lecaros --- pipeline/filters/lua.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 95f6cd060..b0eff83f0 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -178,7 +178,9 @@ pipeline: ### Number Type -+Lua treats number as double. It means an integer field (e.g. IDs, log levels) will be converted double. To avoid type conversion, The `type_int_key` property is available. +Lua treats numbers as a double type. +It means an integer field (e.g. IDs, log levels) will be converted to a double. +To avoid type conversion, the `type_int_key` property is available. ### Protected Mode From 24966db93230dd4474ab3921b23e98a0810f910b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Fri, 29 Nov 2024 18:26:26 -0300 Subject: [PATCH 07/10] filters: lua: add suggestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Pat Signed-off-by: José Lecaros --- pipeline/filters/lua.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index b0eff83f0..638c1dadc 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -184,7 +184,8 @@ To avoid type conversion, the `type_int_key` property is available. ### Protected Mode -Fluent Bit supports protected mode to prevent crash when executes invalid Lua script. See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html). +Fluent Bit supports protected mode to prevent crashes if it executes an invalid Lua script. +See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html). ## Code Examples From aa1bc828d0f2d6b6cb37b1aaa0644788b47f25a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Mon, 9 Dec 2024 11:02:32 -0500 Subject: [PATCH 08/10] filters: lua: commit suggestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adam Locke Signed-off-by: José Lecaros --- pipeline/filters/lua.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 638c1dadc..d863106bb 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -178,9 +178,9 @@ pipeline: ### Number Type -Lua treats numbers as a double type. -It means an integer field (e.g. IDs, log levels) will be converted to a double. -To avoid type conversion, the `type_int_key` property is available. +Lua treats numbers as a `double` type, which means an `integer` type +containing data like user IDs and log levels will be converted to a `double`. +To avoid type conversion, use the `type_int_key` property. ### Protected Mode From 2881a2eb7c2818d19222197518152120b85cece6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Mon, 9 Dec 2024 11:03:25 -0500 Subject: [PATCH 09/10] filters: lua: commit suggestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adam Locke Signed-off-by: José Lecaros --- pipeline/filters/lua.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index d863106bb..0c1690be7 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -185,7 +185,8 @@ To avoid type conversion, use the `type_int_key` property. ### Protected Mode Fluent Bit supports protected mode to prevent crashes if it executes an invalid Lua script. -See also [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html). +See [Error Handling in Application Code](https://www.lua.org/pil/24.3.1.html) in +the Lua documentation for more information. ## Code Examples From f68c1edc2703a88d307e2d9bc14cddbf507ecaf0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Mon, 9 Dec 2024 11:08:10 -0500 Subject: [PATCH 10/10] filters: lua: commit suggestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Adam Locke Signed-off-by: José Lecaros --- pipeline/filters/lua.md | 1 - 1 file changed, 1 deletion(-) diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md index 0c1690be7..f7d2745eb 100644 --- a/pipeline/filters/lua.md +++ b/pipeline/filters/lua.md @@ -193,7 +193,6 @@ the Lua documentation for more information. For functional examples of this interface, please refer to the code samples provided in the source code of the project located here: -[https://github.com/fluent/fluent-bit/tree/master/scripts](https://github.com/fluent/fluent-bit/tree/master/scripts) ### Processing environment variables