TCK Upgrade Story

[ktoso]

This issue came up during discussions on the "bring back the TCK" pull request: #91

In general we need some story about when we change the spec, and the tests, that implementors will be notified about some changed (perhaps optional!) behaviour. In the TCK tests which are optional are skipped if they fail to verify currently, this could cause adding optional behaviour in 0.x+1 simply be skipped to be verified, and the implementor may assume "oh good, my impl covers everything". This is not really limited to adding optional tests, but in general behaviour which we can't test for example too.

This issue means to start a discussion about if we want to provide more info for implementors in the TCK itself or simply assume implementors will read changelogs and apply all changes.

One idea on the more extreme side would be that each spec would contain a value conformsTo() { return Spec_0_4; }), then if the dependency is updated to 0.5, we fail the tests listing what has changed... This might be overkill, and simply "please read the change logs" could be our solution here.

I'm interested if you guys see this as a problem that needs solving or we simply document that "if you upgrade, please do read the changelog in detail".

[viktorklang]

Perhaps we need to make sure that the tests based on the TCK doesn't compile if it is a breaking change in the TCK that needs to be addressed?

[ktoso]

I wouldn't go as far as "not compile", we should simply fail the entire suite then with an error explaining the changes etc. See my original proposal of a conformsTo method, story would be:

• X implemented 1.0
• 1.1 contains "important change"
• In TCK: we add a method SpecVersion conformsTo() { return null; }
• X updates dependency to 1.1, doesn't change any sources - we see SpecVersion being null - we can notify "you're not on 1.1, and it has this and that important change"
• X fixes impl, makes it return conformsTo() { return SpecVersion.v1_1; }

For further upgrades it would be like so:

• 1.2 includes such changes again,
• we make TCK check that it requires a v1_2 to be returned,
• X upgrades dependency, does not change sources, gets notified with "between 1.1 and 1.2 we made important changes" - an error which fails the entire test suite.
• X reads / updates / passes 1.2 test suite.

How about such story? The idea would be to make the update errors really useful. Whereas your proposal would, well, make it not compile, but wouldn't include much more hints "where to look for the changelog etc".

[viktorklang]

@ktoso I think it sounds interesting, the question is how much time it'd take to get something in place.

[ktoso]

So the hardest part is actually maintaining these while prepping a release.
The how-hard it is, depends on how much info we want to include. If we make sure to properly tag releases and put issues into github milestones, it could be just a link to a github milestone. We have not been doing this so far (pre 1.0), so it's more of a question how we want to deal with releases in general in the future (yes, even though we assume there won't be much need to release often).

I mean, I can easily code up what I've outlined above, the question is on broader feedback on the idea and how we want to tackle releases in the post 1.0 era of the project. To me it sounds these version bumps should be all together in the "bumping version" commit on each release - which is someone from contributors (so not me), so we need feedback from others on this one..

[rkuhn]

I’m not convinced that technical measure can fix the problem: patching a string in the source to say “1.2” and then running the verification successfully will always be possible. We need to trust implementors to watch the spec evolve and adapt accordingly. If an unimplemented spec update leads to runtime failures later then we should by definition be in a position to write a test that provokes those, but I estimate that the transition from “everybody does their best” to “the TCK covers everything” will take quite a while.

So, all in all I’m proposing to close this ticket.

[viktorklang]

@rkuhn I would even go as far as say that I hope we never need to ship anything beyond 1.0.0.final :)

[ktoso]

Just a bump here, the TCK readme contains a section on this which is a TODO (been looking and cleaning it up recently). If we want to write something in that section explicitly it should be before 1.0 I think.

Now we're using methods with required_ prefixes etc, so just deprecating / removing removing methods should be reasonable and good enough I feel, opinions?

[viktorklang]

Open a PR :)