feat: ✨ support DictConfigurator prefixes for rename_fields and static_fields

[rubensa]

Python logging config supports access to external and internal objects via DictConfiguator prefixes.

This works internally by warpping the dict with a ConvertingDict that requires an explicit call to it's __getitem__ dunder method for the conversion to take place (using the specific converter for the specified prefix).

This allows, for example, having a log_config.yaml file like:

version: 1
disable_existing_loggers: False
formatters:
  default:
    "()": pythonjsonlogger.json.JsonFormatter
    format: "%(asctime)s %(levelname)s %(name)s %(module)s %(funcName)s %(lineno)s %(message)s"
    rename_fields:
      "asctime": "timestamp"
      "levelname": "status"
    static_fields:
      "service": ext://logging_config.PROJECT_NAME
      "env": ext://logging_config.ENVIRONMENT
      "version": ext://logging_config.PROJECT_VERSION
      "app_log": "true"
handlers:
  default:
    formatter: default
    class: logging.StreamHandler
    stream: ext://sys.stderr
  access:
    formatter: default
    class: logging.StreamHandler
    stream: ext://sys.stdout
loggers:
  uvicorn.error:
    level: INFO
    handlers:
      - default
    propagate: no
  uvicorn.access:
    level: INFO
    handlers:
      - access
    propagate: no

where service, env and version values are taken from the external resource logging_config.py.

The content for logging_config.py could be something like (for getting some values from project metadata or environment variables):

import importlib.metadata
import os


def get_version_metadata():
    # https://stackoverflow.com/a/78082532
    version = importlib.metadata.version(PROJECT_NAME)
    return version


PROJECT_NAME = 'test-api'
PROJECT_VERSION = get_version_metadata()
ENVIRONMENT = os.environ.get('ENVIRONMENT', 'dev')

[nhairs]

First of all - this is very well put together PR 👌

Overall I'm onboard with the change.

I've left a few comments on the files. In addition I think we need to:

• Add something to the docs here - I'd be happy to mostly replace the ini example with your yaml + dictConfig example.
• Add tests to show that the conversion is working as intended.
• Update the change log (if you add an UNRELEASED version header I'll handle links and versions etc later).

[rubensa]

@nhairs First of all, thank you for reviewing this PR.

All suggested changes have been applied but test addition (as I really have no idea on how to do that). I simply tested they are working in my own project.

It would be great if someone with more knowledge could add some.

[rubensa]

@nhairs Finally I managed to add a test to check dictConfig external reference support.

Please check it as I'm not confident it is the best way to do it.

[nhairs]

@nhairs Finally I managed to add a test to check dictConfig external reference support.

Please check it as I'm not confident it is the best way to do it.

Yep I'll take a look 🙂

Also I'll note (in case we need more changes), I pretty much only do squash merges so don't fret too much about update history

[rubensa]

Hi @nhairs

Did you have time to check out the new test for dictConfig external reference support?

[nhairs]

I've taken a quick look and overall everything looks okay - there's a few minor changes I'd like to push up but haven't had the chance

[nhairs]

Alright I've made some minor wording changes in a couple of places.

Let me know if you're okay with my changes and I'll get this merged and look at doing a release soon.

[rubensa]

Your changes are perfect. Thanks @nhairs

[rubensa]

Hi @nhairs

Do you have any ETA for the new release?

[nhairs]

Hi @rubensa,
I was waiting on potentially other changes but I don't think they are happening so I'll probably look at doing it this weekend or next.

That said I'm currently debating on whether this change is a breaking (major) or non-breaking (minor) and thus what type of version.

One could argue that this is a bug that is being fixed and should be classed as a minor or even patch version.

But, this has never been supported by the library and so using it might have unintended consequences for people who never expected it to handle the special prefixes, thus should be classified as a major version.

[rubensa]

Thanks for the update @nhairs

As you can see (updated version from "3.3.0" to "3.3.1.dev0") I classed the change as a patch version (cause I thought it was a "mistake" in python-json-logger implementation details, preventing the already available feature in standard logging library to work as expected).

I don't think that anyone might be using the prefixes expecting them to be "literal" (If anyone tried to use them I suppose that they were doing so, like me, expecting them to work like in standard logging library).

That said, any version change you decide is ok to me.

PS: I suppose that if you include this then it will be a major version change (cause it would change the way things are already defined to work).

[rubensa]

Hi @nhairs

I'm sorry to bother you again but it's been over 2 weeks. I'm waiting for this feature to be available so I can include the library in my project. Do you have a new ETA for the release?

[nhairs]

Yeah understandable.

I'm thinking it's probably a breaking release so there's some other stuff I want to look at included.

Would a development release (on PyPI) be acceptable in the mean time?

[rubensa]

Ok. Let's do this with a development release for now.

[nhairs]

Hi @rubensa, sorry this took a few more days than I said.

This is now uploaded: https://pypi.org/project/python-json-logger/4.0.0.dev0/

[rubensa]

No worries. Thanks @nhairs