Support different Java versions in CI

[sentrivana]

pyspark 4.0.0 needs a newer Java version (e.g. 21), while <4.0.0 can run on 11.

Find a way to support using different Java versions for different test targets. Right now this is an issue since runtox will run all pinned targets at the same time (so in this case, pyspark 3.x and 4.x) and there is no way to switch the Java version in between.

Originally surfaced in #4435