in_elasticsearch with AWS for Fluent Bit, plugin name that don't exists

[rust0258]

Describe the question/issue

I am running AWS for Fluent Bit on EKS to forward logs to cloudwatch, I am trying to ship logs of Elasticsearch running on eks to cloudwatch. but I am getting an error

Configuration

  open-search.conf: |
    [INPUT]
        Name             elasticsearch
        Tag                 opensearch.firstlog
        listen              0.0.0.0
        port                9200
        buffer_max_size     20M
        buffer_chunk_size   10M
        
    [OUTPUT]
        Name                cloudwatch_logs
        Match               opensearch.*
        region              us-east-1
        log_group_name      /aws/containerinsights/eks/opensearch
        log_stream_prefix   opensearch-
        auto_create_group   true
        extra_user_agent    container-insights

Fluent Bit Log Output

Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/21 16:41:16] [error] [config] section 'elasticsearch' tried to instance a plugin name that don't exists
[2023/07/21 16:41:16] [error] configuration file contains errors, aborting.
AWS for Fluent Bit Container Image Version 2.31.11

Fluent Bit Version Info

public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
-->

Cluster Details

Application Details

I am running Opensearch on EKS and having logs written in the OS log

Steps to reproduce the issue

use the Configuration section above in Fluentbit on eks,

[PettitWesley]

[INPUT]
    Name             elasticsearch

Elasticsearch is not an input, its an output:

[OUTPUT]
     Name elasticsearch

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

[rust0258]

Hi @PettitWesley,
I am referring to this page https://docs.fluentbit.io/manual/pipeline/inputs/elasticsearch.
nevertheless, I was looking at the C code of the in_elasticsearch, I think this is related to the logging of the _Bulk API while I am looking to forward opensearch logs found at /usr/share/opensearch/logs inside the pod.
is there a way to have fluentbit tail a json file that is inside a pod?

[PettitWesley]

@ibrahimjelliti Sorry, I didn't realize there was an input for elasticsearch too. It looks like that was added in 2.x, which is not supported in this distro yet. Please post a note here to request 2.x support in AWS distro: #494

forward opensearch logs found at /usr/share/opensearch/logs inside the pod.
is there a way to have fluentbit tail a json file that is inside a pod?

Yes! Configure a tail input to read log files: https://docs.fluentbit.io/manual/pipeline/inputs/tail

And make sure that Fluent Bit has access to the location on the filesystem. If the logs are created in another container, make sure to set up a volume mount to Fluent Bit.