Cannot rename fields unless they're present in every log record

[agronholm]

I tried the following configuration to get field names matching the OpenTelemetry semantic conventions:

json:
  (): pythonjsonlogger.jsonlogger.JsonFormatter
  rename_fields:
    otelTraceID: trace_id
    otelSpanID: span_id
    otelServiceName: service.name
     levelname: severity
     message: body
     threadName: thread.name
     exc_type: exception.type
     exc_val: exception.message
     traceback: exception.stacktrace
     pathname: code.filepath
     lineno: code.lineno
     funcName: code.function
  format: '%(levelname)s %(name)s %(threadName)s %(message)s %(pathname)s %(lineno)s %(funcName)s'

It would not work because some of these fields were not present in every log record, but the field rename function requires cannot handle missing fields. In particular, renaming the exception related fields is impossible.

[agronholm]

Addendum: explicitly adding these fields to format removes the error, but then they have null values where a log record does not contain an exception. Would it be possible to get fields renamed without making them mandatory?

[fiarabbit]

I have this problem too. I really hope this problem to be fixed 👍

[lippertto]

I have the problem too. I need to rename fields for GCP's Error Reporting to pick up exceptions.

@madzak - how should we proceed here? The code change looks safe. Is there anyone who can review+merge it? (And publish a new version)

[lippertto]

For what it's worth: A custom formatter implementation will avoid the bug. However, it will would be better if we could use the standard mechanisms. Here is what I use:

import logging
from pythonjsonlogger import jsonlogger

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    """As long as https://github.com/madzak/python-json-logger/issues/171 is still open, we need to have our
    own JsonFormatter implementation that fixes the bug."""

    def _perform_rename_log_fields(self, log_record):
        for old_field_name, new_field_name in self.rename_fields.items():
            if old_field_name not in log_record:
                continue
            log_record[new_field_name] = log_record[old_field_name]
            del log_record[old_field_name]

def _setup_logging():
    handler = logging.StreamHandler()
    handler.setFormatter(
        CustomJsonFormatter(
            "%(levelname)s %(message)s",
            rename_fields={"levelname": "severity", "exc_info": "stack_trace"},
        )
    )
    logging.getLogger().setLevel(logging.INFO)
    logging.getLogger().addHandler(handler)

[nhairs]

As it looks like python-json-logger is currently unmaintained, I am working on a maintained fork.

I've fixed this in the v3.1.0 release.