Replace versions.py with a TOML file

[NickLarsenNZ]

We currently track supported product versions and dependencies in a versions.py file (which is imported by the conf.py in the root of the repo, which is in turn imported by bake.

Ideally, configurations should be static which would also simplify how this is imported.

Ultimately, bake could then be upgraded to not import a conf.py but rather load in TOML configurations.

This could also then become the source of truth for tracking versions for upcoming releases. It could also help with making versions concrete leading up to a release (eg: a PR opened for the release, and kept open until version freeze a few weeks before a release).

Note

bake is still required for builds as it constructs a dependency graph and builds them before

Configuration Example

Option 1

The top-level releases table keeps track of the Stackable releases. Each product can then declare which versions are available for that Stackable version. The example below, lists two Stackable releases: 24.3 and 24.7. The 24.7 release is marked as active. bake would need to validate that only one Stackable release can be marked as active. The example then declares available product versions per release, in this case 2.6.3 and 2.8.1 for 24.3. The table then lists dependencies and their version.

[releases."24.3"]
active = false

[releases."24.3"."2.6.3"]
python = "3.9"
vector = "0.39.0"

[releases."24.3"."2.8.1"]
python = "3.10"
vector = "0.39.0"

[releases."24.7"]
active = true

JSON structure

{
  "releases": {
    "24.3": {
      "active": false,
      "2.6.3": {
        "python": "3.9",
        "vector": "0.39.0"
      },
      "2.8.1": {
        "python": "3.10",
        "vector": "0.39.0"
      }
    },
    "24.7": {
      "active": true
    }
  }
}

---

Option 2

This option changes the declaration of product versions and it's dependencies slightly compared to above. It is closer to the current versions.py format.

[releases."24.3"]
active = false

[[releases."24.3".versions]]
product = "2.6.3"
python = "3.9"
vector = "0.39.0"

[[releases."24.3".versions]]
product = "2.8.1"
python = "3.10"
vector = "0.39.0"

[releases."24.7"]
active = true

JSON structure

{
  "releases": {
    "24.3": {
      "active": false,
      "versions": [
        {
          "product": "2.6.3",
          "python": "3.9",
          "vector": "0.39.0"
        },
        {
          "product": "2.8.1",
          "python": "3.10",
          "vector": "0.39.0"
        }
      ]
    },
    "24.7": {
      "active": true
    }
  }
}

[NickLarsenNZ]

Nice, I like the options. I'm also torn. I like both the .versions bit but also like products (a faux key) being hoisted up.

I think I lean toward option 1. It seems more "correct".

[razvan]

+1 for switching from .py to .toml
-1 for managing multiple releases in main. We already have release-* branches that may contain much more than just versions. These would clash and make maintenance harder.

[Techassi]

-1 for managing multiple releases in main. We already have release-* branches that may contain much more than just versions. These would clash and make maintenance harder.

To be honest, I'm not a huge fan of release branches as they can often be hard to work with. But I can understand, why we want to keep the current workflow and how tracking Stackable releases in multiple places will cause pain points later down the road.

An alternative syntax could look like this:

[versions."2.8.3"]
dependencies = { python = "3.9", vector = "0.39.0" }

# Identical to above, just an alternate way of writing
[versions."3.0.0".dependencies]
python = "3.11"
vector = "0.39.0"

JSON structure

{
  "versions": {
    "2.8.3": {
      "dependencies": {
        "python": "3.9",
        "vector": "0.39.0"
      }
    },
    "3.0.0": {
      "dependencies": {
        "python": "3.11",
        "vector": "0.39.0"
      }
    }
  }
}

---

Switching off release branches is completely out of scope for this issue, so let's discuss that at a later point.

[NickLarsenNZ]

As part of the work to test container images on PRs, we would like a flag to mark which versions should be included in tests. Extending @Techassi's example from above, this could look like:

[versions."2.8.3"]
build_on_pr = true
dependencies = { python = "3.9", vector = "0.39.0" }

build_on_pr would be optional, and default to false.