Proposal: Inject Fluent Bit Configuration Options via Env Vars #521
Description
I would love to hear anyone's feedback on the following configuration options
Issue: Configuring aws-for-fluent-bit can be difficult
Currently for ECS, advanced configuration of aws-for-fluent-bit, customers need to either build a new aws-for-fluent-bit image with an updated fluent-bit configuration file, or use the init image to pass in a configuration file from s3 or an embedded file.
Customers may not want to worry about maintaining a custom logging image or the hassle of uploading an s3 configuration file and referencing it in their task definitions.
It would be nice to have the entire logging configuration for ECS in the task definition via environment variables, so logging configuration could be thought of almost as a part of the task definition rather than a separate image needing configuration.
A Solution: plugin configuration environment variables
To make configuration easier, we could add some new plugin configuration options as environment variables which would be used to configure Fluent Bit. These variables allow us to use and configure plugins easily without the need to pass in an external file reference or rebuild the image.
The following environment variables could be used:
FLUENT_BIT_INPUT_PLUGINS_<PLUGIN_NAME>_<PLUGIN_NUMBER>: This variable would contain the configuration details for the specific plugin in a string format such as "key1=value1,key2=value2". If multiple instances of the same plugin are used, <PLUGIN_NUMBER> would be used to separate each variable. For example, FLUENT_BIT_INPUT_PLUGINS_TAIL_0 could be set to "path=/var/log"
FLUENT_BIT_INPUT_PLUGINS_<PLUGIN_NAME>_<PLUGIN_NUMBER>: This environment variable would allow customers to specify the configuration for a specific output plugin.
FLUENT_BIT_FILTER_PLUGINS_<PLUGIN_NAME>_<PLUGIN_NUMBER>: This environment variable would allow customers to specify the configuration for a specific filter plugin.
Implementation
Each of the above environment variables would be translated to a Fluent Bit configuration block such and appended to the Fluent Bit configuration file. The environment variable FLUENT_BIT_INPUT_PLUGINS_TAIL_0 if set to path=/var/log/*.log,parser=json would result in the following Fluent Bit configuration block being appended to the configuration file
[INPUT]
name tail
path var/log/*.log
parser json
Plugin configuration environment variables will allow customers to include their entire logging configuration in their task definition, rather than relying on writing to s3, rebuilding the container, or mounting a volume.