populate event.dataset to allow for dedicated partitions in "Log Rate" ML job

[mbarretta]

The Log Rate component of the Logs UI sets up an ML job to look for anomalies in log rate counts by event.dataset. For logs formatted via ecs-logging-java, that field is not set, so the logs show as "unknown" in the Log Rate UI and are grouped together with all other log sources using ecs-logging-java, which removes the out-of-the-box ability to see if one source has an unusual amount of logs.

[felixbarny]

Do you have a suggestion what the value of event.dataset may be? Would it be a static value like application or something dynamic, based on the log event, or the same value as service.name?

[webmat]

Yeah perhaps default to service.name, and call out to this being customizable as well, in the logger config?

[mbarretta]

@felixbarny ha, yeah, the more I looked at the ML job and the ECS field descriptions, the more confused I got. I'm not really sure what the distinction is between event.dataset and service.name or even how events are different than services...aren't service logs events?

So yes, agree with @webmat that defaulting to service.name makes sense for my ask, but we could run into bigger problems if other event.* fields aren't populated and other Kibana ML/UI components start depending on them.

[webmat]

event.dataset is strongly inspired by Beats. It's the literal source of the event. E.g. in the case of Filebeat modules, module "apache" deals with two log types: access logs and error logs. So the two values you would see in event.dataset are "apache.access" and "apache.error".

[mbarretta]

Ok, so for a Java app, maybe the recommendation would be to set `event.dataset` (assuming that attribute becomes configurable) to `<service name>.<service tier>`, or something along those lines. And if you only have one "dataset" generating logs/metrics, then it'd be equal to `service.name`

…

On Wed, Feb 5, 2020 at 1:24 PM Mathieu Martin ***@***.***> wrote: event.dataset is strongly inspired by Beats. It's the literal source of the event. E.g. in the case of Filebeat modules, module "apache" deals with two log types: access logs and error logs. So the two values you would see in event.dataset are "apache.access" and "apache.error". — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#58?email_source=notifications&email_token=AAHX43B4CH2FNBIESD6LDRLRBL75PA5CNFSM4KQMWIAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4O2QI#issuecomment-582544705>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHX43HF4DK7WTHBPH22D6LRBL75PANCNFSM4KQMWIAA> .

-- Mike Barretta // Manager of Solution Architecture, IC // Elastic Certified Engineer mike.barretta@elastic.co +1 703.665.0151 VA, USA

[felixbarny]

Not sure how we would know the service tier. Initially, I mapped the event.dataset to what's now log.logger but the cardinality would be too high for that. By default, log.logger is the fully qualified class name that initiated the log. Devs can override that though and set any string value.

Instead of duplicating event.dataset, could the ML job look at service.name as a fallback?

[mbarretta]

re service tier, I was suggesting that users could set event.dataset to the name.tier pattern using their knowledge of their system, thereby somewhat keeping aligned with how we're doing it in beats.

I don't think it makes sense to tie together event.dataset and log.logger both because of the cardinality, as you mentioned, but also because most apps would have their logs split across multiple "datasets", and some apps that use the same dependencies would have their logs conflated together.

Regardless, I think the spirit of this enhancement request is to just add a configurable event.dataset field so that logs can be natively handled by the Log Rate ML jobs.

As for changing the ML job, that's a whole different issue!

[felixbarny]

The event.dataset can already be configured by adding custom key/value pairs. See https://github.com/elastic/ecs-logging-java/tree/master/log4j2-ecs-layout#layout-parameters

[mbarretta]

I'm using Logback so was following those docs which didn't mention the custom key/value. Is that something you're looking to add?

[felixbarny]

Good point, that feature is indeed missing with logback.

I have created a PR for that: #59

[felixbarny]

@mbarretta could you try this out?

[mbarretta]

@felixbarny LGTM.

For doc purposes, the groovy config for this (and the encoder) looks like:

    encoder(EcsEncoder) {
        serviceName = "my-service"
        additionalField(EcsEncoder.Pair) {
            key="event.dataset"
            value="my-service"
        }
    }

[felixbarny]

Having seen a demo of the log categorization, it seems like ${service.name}.log (for example my-application.log) would be a good default. There should also be a first class citizen configuration for the dataset like this:

    encoder(EcsEncoder) {
        serviceName = "my-service"
        eventDataset = "my-service.logins"
    }