Skip to content

[v3] revisit runtime config #1772

@jhamman

Description

@jhamman

This issue tracks a evaluation of the v3 runtime config

Context

The v3 branch runtime config currently looks like this:

@dataclass(frozen=True)
class RuntimeConfiguration:
order: Literal["C", "F"] = "C"
concurrency: Optional[int] = None
asyncio_loop: Optional[AbstractEventLoop] = None

This is then attached to Array/Group classes

@dataclass(frozen=True)
class AsyncArray:
metadata: ArrayMetadata
store_path: StorePath
runtime_configuration: RuntimeConfiguration

A few things are missing here:

  1. User experience
    • as a user, I may want to set config settings and forget about them (e.g. order, concurrency)
  2. Portability
    • I don't know for sure but I really doubt that putting the AsyncIO loop on the Array class is going to work when it comes to serialization

Improvements

So looking for some ideas for how to manage this better. Two ideas:

  1. Xarray style set-options: https://docs.xarray.dev/en/stable/generated/xarray.set_options.html
    • Pros: allows for validation and is typed
    • Cons: a bit bespoke, doesn't support environment variables or a config file option
  2. Dask style config - https://donfig.readthedocs.io/en/latest/
    • Pros: very flexible framework, support for environment variables and config files, nested namespaces, etc.
    • Cons: extra dependency (though we could vendor it), no typing or validation

what do we expect to go in the runtime config?

  • Order
  • Concurrency
  • logging settings
  • what else?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions