syslog-ng-4.5.0

4.5.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending log messages to OpenObserve

The openobserve-log() destination feeds OpenObserve via the JSON API.

Example config:

openobserve-log(
    url("http://openobserve-endpoint")
    port(5080)
    stream("default")
    user("[email protected]")
    password("V2tsn88GhdNTKxaS")
);

(#4698)

Sending messages to Google Pub/Sub

The google-pubsub() destination feeds Google Pub/Sub via the HTTP REST API.

Example config:

google-pubsub(
  project("syslog-ng-project")
  topic("syslog-ng-topic")
  auth(
    service-account(
      key("/path/to/service-account-key.json")
    )
  )
);

See the Google Pub/Sub documentation to learn more about configuring a service account.
(#4651)

Parsing PostgreSQL logs

The postgresql-csvlog-parser(): add a new parser to process CSV log formatted by
PostgreSQL (https://www.postgresql.org/docs/current/runtime-config-logging.html).
The CSV format is extracted into a set of name-value pairs.
(#4586)

Features

• http(): Added support for using templates in the url() option.

  In syslog-ng a template can only be resolved on a single message, as the same
  template might have different resolutions on different messages. A http batch
  consists of multiple messages, so it is not trivial to decide which message should
  be used for the resolution.

  When batching is enabled and multiple workers are configured it is important to
  only batch messages which generate identical URLs. In this scenario one must set
  the worker-partition-key() option with a template that contains all the templates
  used in the url() option, otherwise messages will be mixed.

  For security reasons, all the templated contents in the url() option are getting
  URL encoded automatically. Also the following parts of the url cannot be templated:

  • scheme
  • host
  • port
  • user
  • password
    (#4663)

• $TRANSPORT: this is a new name-value pair that syslog-ng populates
  automatically. It indicates the "transport" mechanism used to
  retrieve/receive the message. It is up to the source driver to determine
  the value. Currently the following values were implemented:

  BSD syslog drivers: tcp(), udp() & network()

  • rfc3164+tls
  • rfc3164+tcp
  • rfc3164+udp
  • rfc3164+proxied-tls
  • rfc3164+<custom logproto like altp>

  UNIX domain drivers: unix-dgram(), unix-stream()

  • unix-stream
  • unix-dgram

  RFC5424 style syslog: syslog():

  • rfc5426: syslog over udp
  • rfc5425: syslog over tls
  • rfc6587: syslog over tcp
  • rfc5424+<custom logproto like altp>: syslog over a logproto plugin

  Other drivers:

  • otlp: otel() driver
  • mqtt: mqtt() driver
  • hypr-api: hypr-audit-source() driver

  $IP_PROTO: indicate the IP protocol version used to retrieve/receive the
  message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6.
  (#4673)

• network() and syslog() drivers: Added ignore-validity-period as a new flag to ssl-options().

  By specifying ignore-validity-period, you can ignore the validity periods
  of certificates during the certificate validation process.
  (#4642)

• tls() in udp()/tcp()/network() and syslog() drivers: add support
  for a new http() compatible ssl-version() option. This makes the TLS
  related options for http() and other syslog-like drivers more similar. This
  requires OpenSSL 1.1.0.
  (#4682)

• cloud-auth(): Added a new plugin for drivers, which implements different cloud related authentications.

  Currently the only supported authentication is GCP's Service Account for the http() destination.

  Example config:

  http(
    cloud-auth(
      gcp(
        service-account(
          key("/path/to/service-account-key.json")
          audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher")
        )
      )
    )
  );
  

  (#4651)

• csv-parser(): allow parsing the extracted values into matches ($1, $2, $3 ...)
  by omitting the columns() parameter, which normally specifies the column
  names.
  (#4678)

• --check-startup: a new command line option for syslog-ng along with the
  existing --syntax-only. This new option will do a complete configuration
  initialization and then exit with exit code indicating the result. Since
  this also initializes things like network listeners, it will probably not
  work when there is another syslog-ng instance running in the background. The
  recommended use of this option is a dedicated config check container, as
  explained in #4592.
  (#4646)

Bugfixes

• s3: Fixed an ImportError.

  ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object'
  (#4700)

• loki(): fixed mixing non-related label values
  (#4713)

• type hinting: Parsing and casting fractions are now done locale independently.
  (#4702)

• metrics-probe(): Fixed a crash.

  This crash occurred when a metrics-probe() instance was used in multiple source threads,
  like a network() source with multiple connections.
  (#4685)

• flags() argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used.
  No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file.
  (#4689)

• Fix $PROTO value for transport(tls) connections, previously it was set
  to "0" while in reality these are tcp connections (e.g. "6").

  Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source
  drivers (e.g. udp6()/tcp6() or when using ip-protocol(6) for tcp()/udp()).
  Previously V4-mapped addresses would be represented as
  "::ffff:<ipv4 address>". This is not wrong per-se, but would potentially
  cause the same host to be represented in multiple ways. With the fix,
  syslog-ng would just use "<ipv4 address>" in these cases.
  (#4673)

• db-parser(): support nested match characters in @QSTRING@ pattern parser
  (#4717)

Other changes

• LogSource and LogFetcher: additional documentation was added to these
  Python classes to cover explicit source-side batching functionalities (e.g.
  the auto_close_batch attribute and the close_batch() method).
  (#4673)

• rate-limit(): Renamed the template() option to key(), which better communicates the intention.
  (#4679)

• templates: The template-escape() option now only escapes the top-level template function.

  Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting
  escaped the output of each template function, so the parent template function received an
  already escaped string. This was never the intention of the template-escape() option.

  Although this is a breaking change, we do not except anyone having a config that is affected.
  If you have such a config, make sure to follow-up this change. If you need help with it, feel
  free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server.
  (#4666)

• loki(): The timestamp() option now supports quoted strings.

  The valid values are the following, with or without quotes, case insensitive:

  • "current"
  • "received"
  • "msg"
    (#4688)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Cedric Ar...

syslog-ng-4.4.0

4.4.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending messages between syslog-ng instances via OTLP/gRPC

The syslog-ng-otlp() source and destination helps to transfer the internal representation
of a log message between syslog-ng instances. In contrary to the syslog-ng() (ewmm())
drivers, syslog-ng-otlp() does not transfer the messages on simple TCP connections, but uses
the OpenTelemetry protocol to do so.

It is easily scalable (workers() option), uses built-in application layer acknowledgement,
out of the box supports google service authentication (ADC or ALTS), and gives the possibility
of better load balancing.

The performance is currently similar to ewmm() (OTLP is ~30% quicker) but there is a source
side limitation, which will be optimized. We measured 200-300% performance improvement with a
PoC optimized code using multiple threads, so stay tuned.

Note: The syslog-ng-otlp() source is only an alias to the opentelemetry() source.
This is useful for not needing to open different ports for the syslog-ng messages and other
OpenTelemetry messages. The syslog-ng messages are marked with a @syslog-ng scope name and
the current syslog-ng version as the scope version. Both sources will handle the incoming
syslog-ng messages as syslog-ng messages, and all other messages as simple OpenTelemetry
messages.
(#4564)

Grafana Loki destination

The loki() destination sends messages to Grafana Loki using gRPC.
The message format conforms to the documented HTTP endpoint:
https://grafana.com/docs/loki/latest/reference/api/#push-log-entries-to-loki

Example config:

loki(
    url("localhost:9096")
    labels(
        "app" => "$PROGRAM",
        "host" => "$HOST",
    )

    workers(16)
    batch-timeout(10000)
    batch-lines(1000)
);

Loki requires monotonic timestamps within the same label-set, which makes
it difficult to use the original message timestamp without the possibility
of message loss. In case the monotonic property is violated, Loki discards
the problematic messages with an error. The source of the timestamps can be
configured with the timestamp() option (current, received, msg).

(#4631)

S3 destination

The s3() destination stores log messages in S3 objects.

Minimal config:

s3(
    url("http://localhost:9000")
    bucket("syslog-ng")
    access-key("my-access-key")
    secret-key("my-secret-key")
    object-key("${HOST}/my-logs")
    template("${MESSAGE}\n")
);

Compression

Setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the
created object's key. Use the compresslevel() options to set the level of compression (0-9).

Rotation based on object size

The max-object-size() option configures syslog-ng to finish an object if it reaches a certain
size. syslog-ng will append an index ("-1", "-2", ...) to the end of the object key when
starting a new object after rotation.

Rotation based on timestamp

The object-key-timestamp() option can be used to set a datetime related template, which gets
appended to the end of the object (e.g. "${R_MONTH_ABBREV}${R_DAY}" => "-Sep25"). When a log
message arrives with a newer timestamp template resolution, the previous timestamped object gets
finised and a new one is started with the new timestamp. Backfill messages do not reopen and append
the old object, but starts a new object with the key having an index appended to the old object.

Rotation based on timeout

The flush-grace-period() option sets the number of minutes to wait for new messages to arrive to
objects, if the timeout expires the object is finished, and a new message will start a new with
an index appended.

Upload options

The objects are uploaded with the multipart upload API. Chunks are composed locally. When a chunk
reaches a certain size (by default 5 MiB), the chunk is uploaded. When an object is finished, the
multipart upload gets completed and the chunks are merged by S3.

Upload parameters can be configured with the chunk-size(), upload-threads() and
max-pending-uploads() options.

Additional options

Additional options include region(), storage-class() and canned-acl().

(#4624)

Features

• http(): Added compression ability for use with metered egress/ingress

  The new features can be accessed with the following options:

  • accept-encoding() for requesting the compression of HTTP responses form the server.
    (These are currently not used by syslog-ng, but they still contribute to network traffic.)
    The available options are identity (for no compression), gzip or deflate.
    If you want the driver to accept multiple compression types, you can list them separated by
    commas inside the quotation mark, or write all, if you want to enable all available compression types.
  • content-compression() for compressing messages sent by syslog-ng. The available options are
    identity for no compression, gzip, or deflate.

  Below you can see a configuration example:

  destination d_http_compressed{
    http(url("127.0.0.1:80"), content-compression("deflate"), accept-encoding("all"));
  };
  

  (#4137)

• opensearch: Added a new destination.

  It is similar to elasticsearch-http(), with the difference that it does not have the type()
  option, which is deprecated and advised not to use.
  (#4560)

• Added metrics for message delays: a new metric is introduced that measures the
  delay the messages accumulate while waiting to be delivered by syslog-ng.
  The measurement is sampled, e.g. syslog-ng would take the very first message
  in every second and expose its delay as a value of the new metric.

  There are two new metrics:

  • syslogng_output_event_delay_sample_seconds -- contains the latency of
    outgoing messages
  • syslogng_output_event_delay_sample_age_seconds -- contains the age of the last
    measurement, relative to the current time.
    (#4565)

• metrics-probe: Added dynamic labelling support via name-value pairs

  You can use all value-pairs options, like key(), rekey(), pair() or scope(), etc...

  Example:

  metrics-probe(
    key("foo")
    labels(
      "static-label" => "bar"
      key(".my_prefix.*" rekey(shift-levels(1)))
    )
  );
  

  syslogng_foo{static_label="bar",my_prefix_baz="almafa",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4
  

  (#4610)

• systemd-journal(): Added support for enabling multiple systemd-journal() sources

  Using multiple systemd-journal() sources are now possible as long as each source uses a unique
  systemd namespace. The namespace can be configured with the namespace() option, which has a
  default value of "*".
  (#4553)

• stdout(): added a new destination that allows you to write messages easily
  to syslog-ng's stdout.
  (#4620)

• network(): Added ignore-hostname-mismatch as a new flag to ssl-options().

  By specifying ignore-hostname-mismatch, you can ignore the subject name of a
  certificate during the validation process. This means that syslog-ng will
  only check if the certificate itself is trusted by the current set of trust
  anchors (e.g. trusted CAs) ignoring the mismatch between the targeted
  hostname and the certificate subject.
  (#4628)

Bugfixes

• syslog-ng: fix runtime undefined symbol: random_choice_generator_parser' when executing syslog-ng -V or
  using an example plugin
  (#4615)

• Fix threaded destination crash during a configuration revert

  Threaded destinations that do not support the workers() option crashed while
  syslog-ng was trying to revert to an old configuration.
  (#4588)

• redis(): fix incrementing seq_num
  (#4588)

• python(): fix crash when using Persist or LogTemplate without global python{} code block in configuration
  (#4572)

• mqtt() destination: fix template option initialization
  (#4605)

• opentelemetry: Fixed error handling in case of insert failure.
  (#4583)

• pdbtool: add validation for types of <value> tags

  In patterndb, you can add extra name-value pairs following a match with the tags.
  But the actual value of these name-value pairs were never validated against their types,
  meaning that an incorrect value could be set using this construct.
  (#4621)

• grouping-by(), group-lines(): Fixed a persist name generating error.
  (#4478)

Packaging

• debian: Added tzdata-legacy to Buil...

syslog-ng-4.3.1

4.3.1

This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed
a python-parser() related crash and a metrics related memory leak. It also
added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()
destination.

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    **N...

syslog-ng-4.3.0

4.3.0

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    NOTE: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick)
  • dbi_driver_dir to define an optional DBI driver location for DBD initializati...

syslog-ng-4.2.0

4.2.0

Read Axoflow's blog post for more details.

Highlights

Sending messages to Splunk HEC

The splunk-hec-event() destination feeds Splunk via the HEC events API.

Minimal config:

destination d_splunk_hec_event {
  splunk-hec-event(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
  );
};

Additional options include:

• event()
• index()
• source()
• sourcetype()
• host()
• time()
• default-index()
• default-source()
• default-sourcetype()
• fields()
• extra-headers()
• extra-queries()
• content-type()

The splunk-hec-raw() destination feeds Splunk via the HEC raw API.

Minimal config:

destination d_splunk_hec_raw {
  splunk-hec-raw(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
    channel("05ed4617-f186-4ccd-b4e7-08847094c8fd")
  );
};

(#4462)

Smart multi-line for recognizing backtraces

multi-line-mode(smart):
With this multi-line mode, the inherently multi-line data backtrace format is
recognized even if they span multiple lines in the input and are converted
to a single log message for easier analysis. Backtraces for the following
programming languages are recognized : Python, Java, JavaScript, PHP, Go,
Ruby and Dart.

The regular expressions to recognize these programming languages are
specified by an external file called
/usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on
configure arguments), in a format that is described in that file.

group-lines() parser: this new parser correlates multi-line messages
received as separate, but subsequent lines into a single log message.
Received messages are first collected into streams related messages (using
key()), then collected into correlation contexts up to timeout() seconds.
The identification of multi-line messages are then performed on these
message contexts within the time period.

  group-lines(key("$FILE_NAME")
              multi-line-mode("smart")
        template("$MESSAGE")
        timeout(10)
        line-separator("\n")
  );

(#4225)

HYPR Audit Trail source

hypr-audit-trail() & hypr-app-audit-trail() source drivers are now
available to monitor the audit trails for HYPR applications.

See the README.md file in the driver's directory to see usage information.

(#4175)

ebpf() plugin and reuseport packet randomizer

A new ebpf() plugin was added as a framework to leverage the kernel's eBPF
infrastructure to improve performance and scalability of syslog-ng.

Example:

source s_udp {
        udp(so-reuseport(yes) port(2000) persist-name("udp1")
                ebpf(reuseport(sockets(4)))
        );
        udp(so-reuseport(yes) port(2000) persist-name("udp2"));
        udp(so-reuseport(yes) port(2000) persist-name("udp3"));
        udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};

NOTE: The ebpf() plugin is considered advanced usage so its compilation is
disabled by default. Please don't use it unless all other avenues of
configuration solutions are already tried. You will need a special
toolchain and a recent kernel version to compile and run eBPF programs.

(#4365)

Features

• network source: During a TLS handshake, syslog-ng now automatically sets the
  certificate_authorities field of the certificate request based on the ca-file()
  and ca-dir() options. The pkcs12-file() option already had this feature.
  (#4412)

• metrics-probe(): Added level() option to set the stats level of the generated metrics.
  (#4453)

• metrics-probe(): Added increment() option.

  Users can now set a template, which resolves to a number that modifies
  the increment of the counter. If not set, the increment is 1.
  (#4447)

• python: Added support for typed custom options.

  This applies for python source, python-fetcher source, python destination,
  python parser and python-http-header inner destination.

  Example config:

  python(
    class("TestClass")
    options(
      "string_option" => "example_string"
      "bool_option" => True  # supported values are: True, False, yes, no
      "integer_option" => 123456789
      "double_option" => 123.456789
      "string_list_option" => ["string1", "string2", "string3"]
      "template_option" => LogTemplate("${example_template}")
    )
  );
  

  Breaking change! Previously values were converted to strings if possible, now they are passed
  to the python class with their real type. Make sure to follow up these changes
  in your python code!
  (#4354)

• mongodb destination: Added support for list, JSON and null types.
  (#4437)

• add-contextual-data(): significantly reduce memory usage for large CSV
  files.
  (#4444)

• python(): new LogMessage methods for querying as string and with default values

  • get(key[, default])
    Return the value for key if key exists, else default. If default is
    not given, it defaults to None, so that this method never raises a
    KeyError.

  • get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal'):
    Return the string value for key if key exists, else default.
    If default is not given, it defaults to None, so that this method never
    raises a KeyError.

    The string value is decoded using the codec registered for encoding.
    errors may be given to set the desired error handling scheme.

    Note that currently repr='internal' is the only available representation.
    We may implement another more Pythonic representation in the future, so please
    specify the repr argument explicitly if you want to avoid future
    representation changes in your code.
    (#4410)

• kubernetes() source: Added support for json-file logging driver format.
  (#4419)

• The new $RAWMSG_SIZE hard macro can be used to query the original size of the
  incoming message in bytes.

  This information may not be available for all source drivers.
  (#4440)

• syslog-ng configuration identifier

  A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example:

  @config-id: cfg-20230404-13-g02b0850fc
  

  This keyword can be used for config identification in managed environments, where syslog-ng instances and their
  configuration are deployed/generated automatically.

  syslog-ng-ctl config --id can be used to query the active configuration ID and the SHA256 hash of the full
  "preprocessed" syslog-ng configuration. For example:

  $ syslog-ng-ctl config --id
  cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83)
  

  (#4420)

• syslog-ng: add --config-id command line option

  Similarly to --syntax-only, this command line option parses the configuration
  and then prints its ID before exiting.

  It can be used to query the ID of the current configuration persisted on
  disk.
  (#4435)

• Health metrics and syslog-ng-ctl healthcheck

  A new syslog-ng-ctl command has been introduced, which can be used to query a healthcheck status from syslog-ng.
  Currently, only 2 basic health values are reported.

  syslog-ng-ctl healthcheck --timeout <seconds> can be specified to use it as a boolean healthy/unhealthy check.

  Health checks are also published as periodically updated metrics.
  The frequency of these checks can be configured with the stats(healthcheck-freq()) option.
  The default is 5 minutes.
  (#4362)

• $(format-json) and template functions which support value-pairs
  expressions: new key transformations upper() and lower() have been added to
  translate the caps of keys while formatting the output template. For
  example:

  template("$(format-json test.* --upper)\n")
  

  Would convert all keys to uppercase. Only supports US ASCII.
  (#4452)

• python(), python-fetcher() sources: Added a mapping for the flags() option.

  The state of the flags() option is mapped to the self.flags variable, which is
  a Dict[str, bool], for example:

  {
      'parse': True,
      'check-hostname': False,
      'syslog-protocol': True,
      'assume-utf8': False,
      'validate-utf8': False,
      'sanitize-utf8': False,
      'multi-line': True,
      'store-legacy-msghdr': True,
      'store...

syslog-ng-4.1.1

4.1.1

This is the combination of the news entries of 4.1.0 and 4.1.1.
4.1.1 hotfixed a grouping-by() and db-parser() related crash.

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol
used by network load balancers, such as Amazon Elastic Load Balancer and
HAProxy, to carry original source/destination address information, as described
in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages
passing through based on the metadata of each message. The parser creates
labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not
been filtered out from the related log path, it does care about whether there
are any destinations in it or that any destination delivers or drops the
message.

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and
  formats it according to an strftime() format string. The DATETIME
  representation in syslog-ng is a UNIX timestamp formatted as a decimal
  number, with an optional fractional part, where the seconds and the
  fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is
  used.

  Options:
  --time-zone <TZstring> -- override timezone of the original timestamp
  (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424
  SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style
  structured data string. It can be used to parse this relatively complex
  format separately.
  (#4281)

• system() source: the system() source was changed on systemd platforms to
  fetch journal messages that relate to the current boot only (e.g. similar
  to journalctl -fb) and to ignore messages generated in previous boots,
  even if those messages were succesfully stored in the journal and were not
  picked up by syslog-ng. This change was implemented as the journald access
  APIs work incorrectly if time goes backwards across reboots, which is an
  increasingly frequent event in virtualized environments and on systems that
  lack an RTC. If you want to retain the old behaviour, please bypass the
  system() source and use systemd-journal() directly, where this option
  can be customized. The change is not tied to @version as we deemed the new
  behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to
  allow you to constrain the collection of journal records to a subset of what
  is in the journal. match-boot() is a yes/no value that allows you to fetch
  messages that only relate to the current boot. matches() allows you to
  specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store
  the resulting timestamp in a name-value pair, instead of changing the
  timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as
  strings when stored as a part of a log message. syslog-ng simply remembers
  the type it was stored as. Whenever the value is used as a specific type in
  a type-aware context where we need the value of the specific type, an
  automatic string parsing takes place. This parsing happens for instance
  whenever syslog-ng stores a datetime value in MongoDB or when
  $(format-date) template function takes a name-value pair as parameter.
  The datetime() type has stored its value as the number of milliseconds since
  the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
  it possible to store timestamps up to nanosecond resolutions along with an
  optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type,
  use the timezone offset if one is available.
  (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields,
  like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior)
    (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work
  with syslog-ng-s macros, which by default use . as a delimiter. The added
  key-delimiter() option changes this behavior by storing the parsed
  metadata fields with a custom delimiter. In order to reach the fields, the
  accessor side has to use the new delimiter format, e.g. --key-delimiter
  option in $(format-json).
  (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
  to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
  PR description.
  (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
  (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size().
  Note that changing the disk-buf-size() of an existing disk-queue was never supported,
  but could cause errors, which are fixed now.
  (#4308)

• dqtool: fix dqtool assign
  ([#4355](https://github.com/sys...

syslog-ng-4.1.0

4.1.0

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol
used by network load balancers, such as Amazon Elastic Load Balancer and
HAProxy, to carry original source/destination address information, as described
in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages
passing through based on the metadata of each message. The parser creates
labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not
been filtered out from the related log path, it does care about whether there
are any destinations in it or that any destination delivers or drops the
message.

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and
  formats it according to an strftime() format string. The DATETIME
  representation in syslog-ng is a UNIX timestamp formatted as a decimal
  number, with an optional fractional part, where the seconds and the
  fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is
  used.

  Options:
  --time-zone <TZstring> -- override timezone of the original timestamp
  (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424
  SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style
  structured data string. It can be used to parse this relatively complex
  format separately.
  (#4281)

• system() source: the system() source was changed on systemd platforms to
  fetch journal messages that relate to the current boot only (e.g. similar
  to journalctl -fb) and to ignore messages generated in previous boots,
  even if those messages were succesfully stored in the journal and were not
  picked up by syslog-ng. This change was implemented as the journald access
  APIs work incorrectly if time goes backwards across reboots, which is an
  increasingly frequent event in virtualized environments and on systems that
  lack an RTC. If you want to retain the old behaviour, please bypass the
  system() source and use systemd-journal() directly, where this option
  can be customized. The change is not tied to @version as we deemed the new
  behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to
  allow you to constrain the collection of journal records to a subset of what
  is in the journal. match-boot() is a yes/no value that allows you to fetch
  messages that only relate to the current boot. matches() allows you to
  specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store
  the resulting timestamp in a name-value pair, instead of changing the
  timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as
  strings when stored as a part of a log message. syslog-ng simply remembers
  the type it was stored as. Whenever the value is used as a specific type in
  a type-aware context where we need the value of the specific type, an
  automatic string parsing takes place. This parsing happens for instance
  whenever syslog-ng stores a datetime value in MongoDB or when
  $(format-date) template function takes a name-value pair as parameter.
  The datetime() type has stored its value as the number of milliseconds since
  the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
  it possible to store timestamps up to nanosecond resolutions along with an
  optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type,
  use the timezone offset if one is available.
  (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields,
  like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior)
    (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work
  with syslog-ng-s macros, which by default use . as a delimiter. The added
  key-delimiter() option changes this behavior by storing the parsed
  metadata fields with a custom delimiter. In order to reach the fields, the
  accessor side has to use the new delimiter format, e.g. --key-delimiter
  option in $(format-json).
  (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
  to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
  PR description.
  (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
  (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size().
  Note that changing the disk-buf-size() of an existing disk-queue was never supported,
  but could cause errors, which are fixed now.
  (#4308)

• dqtool: fix dqtool assign
  (#4355)

• example-diskq-source: Fixed failing to read the disk-queue content in some cases.
  ([#4308](ht...

syslog-ng-4.0.1

4.0.1

This is the combination of the news entries of 4.0.0 and 4.0.1.

This is a new major version of syslog-ng, ending the 3.x series which
started roughly 13 years ago, on 17th February 2009.

Like all releases in the 3.x series, 4.0.0 is not a breaking change either.
Long-term compatibility has been and continues to be an essential objective
of syslog-ng; thus, you can still run unchanged configurations that were
originally created for syslog-ng 3.0.0.

You can safely upgrade to 4.0.0 if you followed along 3.x, and you should
probably also consider upgrading if you are stuck with an older 3.x release.

The new version number primarily indicates that this version of syslog-ng is
much more than the software we released 13 years ago. While it does have
certain "big-bang" items in its feature list, new features were continuously
introduced throughout our 3.x series as well. Our engineering practices
have not changed simply because we were working on a new major release: this
is the continuation of our previous releases in every respect, produced in
the same manner, just with a more catchy version number.

For this reason, there is no separate deprecation or support period for 3.x
releases, similarly with our existing practice. We support earlier syslog-ng
releases by providing maintenance and fixes in the new release track.
Fixes to problems are not backported to earlier releases by the syslog-ng
project.

Highlights

Introduce runtime type information to name-value pairs

syslog-ng uses a data model where a log message contains an unordered set
of name-value pairs. The values stored in these name-value pairs are
usually textual, so syslog-ng has traditionally stored these values in
text format.

With the increase of JSON-based message sources and destinations, types
became more important. If we encounter a message where a name-value pair
originates from a JSON document, and this document contains a member that
is numeric, we may want to reproduce that as we send this data to a
consumer.

For example, sometimes we extract a numerical metric from a log message,
and we need to send this to a consumer, again with the correct type.

To be able to do this, we added runtime type information to the syslog-ng
message model: each name-value pair becomes a (name, type, value) triplet.

We introduced the following types:

• string: simple textual data, mostly utf8 (but not always)
• int: an integer representable by a 64 bit signed value
• double: a double precision floating point number
• boolean: true or false
• datetime: Date and Time represented by the milliseconds since epoch
• list: list of strings
• json: JSON snippet
• null: an unset value

Apart from the syslog-ng core supporting the notion of types, its use is
up to the sources, filters, rewrite rules, parsers and destinations that
set or make use of them in any way it makes the most sense for the component
in question.

Type-aware comparisons

syslog-ng uses filter expressions to make routing decisions and during the
transformation of messages. These filter expressions are used in filter
{} or if {} statements, for example.

In these expressions, you can use comparison operators. This example, for
instance, uses the '>' operator to check for HTTP response codes
greater-or-equal than 500:

     if ("${apache.response}" >= 500) {
     };

Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the
other for string-based comparisons (eq, ne, gt, lt).

The separate operators were cumbersome to use. Users often forgot which
operator was the right one for a specific case.

Typing allows us to do the right thing in most cases automatically, and a
syntax that allows the user to override the automatic decisions in the
rare case.

With that, starting with 4.0, the old-numeric operators have been
converted to be type-aware operators. It would compare as strings if both
sides of the comparisons are strings. It would compare numerically if at
least one side is numeric. A great deal of inspiration was taken from
JavaScript, which was considered to be a good model, since the problem
space is similar.

See this blog post for more details:
https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/

Capture type information from JSON

When using json-parser(), syslog-ng converts all members of a JSON object
to syslog-ng name-value pairs. Prior to the introduction of type support,
these name-value pairs were all stored as strings. Any type information
originally present in the incoming JSON object was lost.

This meant that if you regenerated the JSON from the name-value pairs using
the $(format-json) template function, all numbers, booleans and other
types became strings in the output.

There has been a feature in syslog-ng that alleviated the loss of types.
This feature was called "type-hints". Type-hints tell $(format-json) to
use a specific type on output, independently of a name-value pair's
original type, but this type conversion needed to be explicit in the
configuration.

An example configuration that parses JSON on input and produces a JSON on
output:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.*)\n")); };
};

To augment the above with type hinting, you could use:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};

NOTE the presence of the int64() type hint in the 2nd example.

The new feature introduced with typing is that syslog-ng would
automatically store the JSON type information as a syslog-ng type, thus it
will transparently carry over types from inputs to output, without having
to be explicit about them.

Typing support for various components in syslog-ng

Typing is a feature throughout syslog-ng, and although the gust of it has
been explained in the highlights section, some further details are
documented in the list down below:

• type-aware comparisons in filter expressions: as detailed above, the
  previously numeric operators become type-aware, and the exact comparison
  performed will be based on types associated with the values we compare.

• json-parser() and $(format-json): JSON support is massively improved
  with the introduction of types. For one: type information is retained
  across input parsing->transformation->output formatting. JSON lists
  (arrays) are now supported and are converted to syslog-ng lists so they
  can be manipulated using the $(list-*) template functions. There are
  other important improvements in how we support JSON.

• set(), groupset(): in any case where we allow the use of templates,
  support for type-casting was added, and the type information is properly
  promoted.

• db-parser() type support: db-parser() gets support for type casts,
  assignments within db-parser() rules can associate types with
  values using the "type" attribute, e.g. <value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
  associates $foobar with an integer type. db-parser()’s internal parsers
  (e.g. @NUMBER@) will also associate type information with a name-value
  pair automatically.

• add-contextual-data() type support: any new name-value pair that is
  populated using add-contextual-data() will propagate type information,
  similarly to db-parser().

• map-value-pairs() type support: propagate type information

• SQL type support: the sql() driver gained support for types, so that
  columns with specific types will be stored as those types.

• template type support: templates can now be casted explicitly to a
  specific type, but they also propagate type information from
  macros/template functions and values in the template string

• value-pairs type support: value-pairs form the backbone of specifying a
  set of name-value pairs and associated transformations to generate JSON
  or a key-value pair format. It also gained support for types, the
  existing type-hinting feature that was already part of value-pairs was
  adapted and expanded to other parts of syslog-ng.

• python() typing: support for typing was added to all Python components
  (sources, destinations, parsers and template functions), along with more
  documentation & examples on how the Python bindings work. All types except
  json() are supported as they are queried- or changed by Python code.

• on-disk serialized formats (e.g. disk buffer/logstore): we remain
  compatible with messages serialized with an earlier version of
  syslog-ng, and the format we choose remains compatible for “downgrades”
  as well. E.g. even if a new version of syslog-ng serialized a message,
  the old syslog-ng and associated tools will be able to read it (sans
  type information of course)

Improved support for lists (arrays)

For syslog-ng, everything is traditionally a string. A convention was
started with syslog-ng in v3.10, where a comma-separated format
could be used as a kind of array using the $(list-*) family of template
functions.

For example, $(list-head) takes off the first element in a list, while
$(list-tail) takes the last. You can index and slice list elements using
the $(list-slice) and $(list-nth) functions and so on.

syslog-ng has started to return such lists in various cases, so they can
be manipulated using these list-specific template functions. These
include the xml-parser(), or the $(explode) template function, but there
are others.

Here is an example that has worked since syslog-ng 3.10:

`...

syslog-ng-4.0.0

4.0.0

This is a new major version of syslog-ng, ending the 3.x series which
started roughly 13 years ago, on 17th February 2009.

Like all releases in the 3.x series, 4.0.0 is not a breaking change either.
Long-term compatibility has been and continues to be an essential objective
of syslog-ng; thus, you can still run unchanged configurations that were
originally created for syslog-ng 3.0.0.

You can safely upgrade to 4.0.0 if you followed along 3.x, and you should
probably also consider upgrading if you are stuck with an older 3.x release.

The new version number primarily indicates that this version of syslog-ng is
much more than the software we released 13 years ago. While it does have
certain "big-bang" items in its feature list, new features were continuously
introduced throughout our 3.x series as well. Our engineering practices
have not changed simply because we were working on a new major release: this
is the continuation of our previous releases in every respect, produced in
the same manner, just with a more catchy version number.

For this reason, there is no separate deprecation or support period for 3.x
releases, similarly with our existing practice. We support earlier syslog-ng
releases by providing maintenance and fixes in the new release track.
Fixes to problems are not backported to earlier releases by the syslog-ng
project.

Highlights

Introduce runtime type information to name-value pairs

syslog-ng uses a data model where a log message contains an unordered set
of name-value pairs. The values stored in these name-value pairs are
usually textual, so syslog-ng has traditionally stored these values in
text format.

With the increase of JSON-based message sources and destinations, types
became more important. If we encounter a message where a name-value pair
originates from a JSON document, and this document contains a member that
is numeric, we may want to reproduce that as we send this data to a
consumer.

For example, sometimes we extract a numerical metric from a log message,
and we need to send this to a consumer, again with the correct type.

To be able to do this, we added runtime type information to the syslog-ng
message model: each name-value pair becomes a (name, type, value) triplet.

We introduced the following types:

• string: simple textual data, mostly utf8 (but not always)
• int: an integer representable by a 64 bit signed value
• double: a double precision floating point number
• boolean: true or false
• datetime: Date and Time represented by the milliseconds since epoch
• list: list of strings
• json: JSON snippet
• null: an unset value

Apart from the syslog-ng core supporting the notion of types, its use is
up to the sources, filters, rewrite rules, parsers and destinations that
set or make use of them in any way it makes the most sense for the component
in question.

Type-aware comparisons

syslog-ng uses filter expressions to make routing decisions and during the
transformation of messages. These filter expressions are used in filter
{} or if {} statements, for example.

In these expressions, you can use comparison operators. This example, for
instance, uses the '>' operator to check for HTTP response codes
greater-or-equal than 500:

     if ("${apache.response}" >= 500) {
     };

Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the
other for string-based comparisons (eq, ne, gt, lt).

The separate operators were cumbersome to use. Users often forgot which
operator was the right one for a specific case.

Typing allows us to do the right thing in most cases automatically, and a
syntax that allows the user to override the automatic decisions in the
rare case.

With that, starting with 4.0, the old-numeric operators have been
converted to be type-aware operators. It would compare as strings if both
sides of the comparisons are strings. It would compare numerically if at
least one side is numeric. A great deal of inspiration was taken from
JavaScript, which was considered to be a good model, since the problem
space is similar.

See this blog post for more details:
https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/

Capture type information from JSON

When using json-parser(), syslog-ng converts all members of a JSON object
to syslog-ng name-value pairs. Prior to the introduction of type support,
these name-value pairs were all stored as strings. Any type information
originally present in the incoming JSON object was lost.

This meant that if you regenerated the JSON from the name-value pairs using
the $(format-json) template function, all numbers, booleans and other
types became strings in the output.

There has been a feature in syslog-ng that alleviated the loss of types.
This feature was called "type-hints". Type-hints tell $(format-json) to
use a specific type on output, independently of a name-value pair's
original type, but this type conversion needed to be explicit in the
configuration.

An example configuration that parses JSON on input and produces a JSON on
output:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.*)\n")); };
};

To augment the above with type hinting, you could use:

log {
    source { ... };
    parser { json-parser(prefix('.json.')); };
    destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};

NOTE the presence of the int64() type hint in the 2nd example.

The new feature introduced with typing is that syslog-ng would
automatically store the JSON type information as a syslog-ng type, thus it
will transparently carry over types from inputs to output, without having
to be explicit about them.

Typing support for various components in syslog-ng

Typing is a feature throughout syslog-ng, and although the gust of it has
been explained in the highlights section, some further details are
documented in the list down below:

• type-aware comparisons in filter expressions: as detailed above, the
  previously numeric operators become type-aware, and the exact comparison
  performed will be based on types associated with the values we compare.

• json-parser() and $(format-json): JSON support is massively improved
  with the introduction of types. For one: type information is retained
  across input parsing->transformation->output formatting. JSON lists
  (arrays) are now supported and are converted to syslog-ng lists so they
  can be manipulated using the $(list-*) template functions. There are
  other important improvements in how we support JSON.

• set(), groupset(): in any case where we allow the use of templates,
  support for type-casting was added, and the type information is properly
  promoted.

• db-parser() type support: db-parser() gets support for type casts,
  assignments within db-parser() rules can associate types with
  values using the "type" attribute, e.g. <value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
  associates $foobar with an integer type. db-parser()’s internal parsers
  (e.g. @NUMBER@) will also associate type information with a name-value
  pair automatically.

• add-contextual-data() type support: any new name-value pair that is
  populated using add-contextual-data() will propagate type information,
  similarly to db-parser().

• map-value-pairs() type support: propagate type information

• SQL type support: the sql() driver gained support for types, so that
  columns with specific types will be stored as those types.

• template type support: templates can now be casted explicitly to a
  specific type, but they also propagate type information from
  macros/template functions and values in the template string

• value-pairs type support: value-pairs form the backbone of specifying a
  set of name-value pairs and associated transformations to generate JSON
  or a key-value pair format. It also gained support for types, the
  existing type-hinting feature that was already part of value-pairs was
  adapted and expanded to other parts of syslog-ng.

• python() typing: support for typing was added to all Python components
  (sources, destinations, parsers and template functions), along with more
  documentation & examples on how the Python bindings work. All types except
  json() are supported as they are queried- or changed by Python code.

• on-disk serialized formats (e.g. disk buffer/logstore): we remain
  compatible with messages serialized with an earlier version of
  syslog-ng, and the format we choose remains compatible for “downgrades”
  as well. E.g. even if a new version of syslog-ng serialized a message,
  the old syslog-ng and associated tools will be able to read it (sans
  type information of course)

Improved support for lists (arrays)

For syslog-ng, everything is traditionally a string. A convention was
started with syslog-ng in v3.10, where a comma-separated format
could be used as a kind of array using the $(list-*) family of template
functions.

For example, $(list-head) takes off the first element in a list, while
$(list-tail) takes the last. You can index and slice list elements using
the $(list-slice) and $(list-nth) functions and so on.

syslog-ng has started to return such lists in various cases, so they can
be manipulated using these list-specific template functions. These
include the xml-parser(), or the $(explode) template function, but there
are others.

Here is an example that has worked since syslog-ng 3.10:

  # MSG contains foo:bar:baz
  # - the $(list-head) take...

syslog-ng-3.38.1

3.38.1

Highlights

Sneak peek into syslog-ng v4.0

syslog-ng v4.0 is right around the corner.

This release (v3.38.1) contains all major changes, however, they are
currently all hidden behind a feature flag.
To enable and try those features, you need to specify @version: 4.0 at the
top of the configuration file.

You can find out more about the 4.0 changes and features here.

Read our practical introduction to typing at
syslog-ng-future.blog.

Features

• grouping-by(): added inject-mode(aggregate-only)

  This inject mode will drop individual messages that make up the correlation
  context (key() groups) and would only yield the aggregate messages
  (e.g. the results of the correlation).
  (#3998)

• add-contextual-data(): add support for type propagation, e.g. set the
  type of name-value pairs as they are created/updated to the value returned
  by the template expression that we use to set the value.

  The 3rd column in the CSV file (e.g. the template expression) now supports
  specifying a type-hint, in the format of "type-hint(template-expr)".

  Example line in the CSV database:

  selector-value,name-value-pair-to-be-created,list(foo,bar,baz)
  (#4051)

• $(format-json): add --key-delimiter option to reconstruct JSON objects
  using an alternative structure separator, that was created using the
  key-delimiter() option of json-parser().
  (#4093)

• json-parser(): add key-delimiter() option to extract JSON structure
  members into name-value pairs, so that the names are flattened using the
  character specified, instead of dot.

  Example:
  Input: {"foo":{"key":"value"}}

  Using json-parser() without key-delimiter() this is extracted to:

    foo.key="value"
  

  Using json-parser(key-delimiter("~")) this is extracted to:

    foo~key="value"
  

  This feature is useful in case the JSON keys contain dots themselves, in
  those cases the syslog-ng representation is ambigious.
  (#4093)

Bugfixes

• Fixed buffer handling of syslog and timestamp parsers (CVE-2022-38725)

  Multiple buffer out-of-bounds issues have been fixed, which could cause
  hangs, high CPU usage, or other undefined behavior.
  (#4110)

• Fixed building with LibreSSL
  (#4081)

• network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character
  in case of a character conversion error.
  (#4084)

• redis(): Fixed bug where using redis driver without the batch-lines option caused program crash.
  (#4114)

• pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool
  exits. Apart from being an ugly crash that produces a core file,
  functionally the tool behaved correctly and this case does not affect
  syslog-ng itself.
  (#4037)

• regexp-parser(): due to a change introduced in 3.37, named capture groups
  are stored indirectly in the LogMessage to avoid copying of the value. In
  this case the name-value pair created with the regexp is only stored as a
  reference (name + length of the original value), which improves performance
  and makes such name-value pairs use less memory. One omission in the
  original change in 3.37 is that syslog-ng does not allow builtin values to
  be stored indirectly (e.g. $MESSAGE and a few of others) and this case
  causes an assertion to fail and syslog-ng to crash with a SIGABRT. This
  abort is now fixed. Here's a sample config that reproduces the issue:

  regexp-parser(patterns('(?<MESSAGE>.*)'));
  

  (#4043)

• set-tag: fix cloning issue when string literal were used (see #4062)
  (#4065)

• add-contextual-data(): fix high memory usage when using large CSV files
  (#4067)

Other changes

• The json-c library is no longer bundled in the syslog-ng source tarball

  Since all known OS package managers provide json-c packages nowadays, the json-c
  submodule has been removed from the source tarball.

  The --with-jsonc=internal option of the configure script has been removed
  accordingly, system libraries will be used instead. For special cases, the JSON
  support can be disabled by specifying --with-jsonc=no.
  (#4078)

• platforms: Dropped support for ubuntu-impish as it became EOL
  (#4088)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler,
Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy,
HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha,
Szilárd Parrag, Vivin Peris