Scaladoc tasks in CI use wrong version of stdlib

[odersky]

See #13423. Here the compilation rules have changed so that class MatchError of the original StdLib213 no longer compiles. I have updated MatchError and even merged the updates into master in #13425. But the ScalaDoc tasks still fail with an error
that indicates they see the previous version of the library. Here is the error:

9 |final class MatchError(@transient obj: Any) extends RuntimeException {
Error:     |                                  ^
Error:     |obj is marked `@transient` but it is retained as a field in class MatchError

To compare, the new version of MatchError in the StdLib included in CB reads as follows

9 |final class MatchError(@transient private val obj: Any) extends RuntimeException {

Expected

Use the latest version of the standard library in the CB for ScalaDoc tasks

[pikinier20]

I don't know if it's really a problem of Scaladoc. In order to generate documentation of standard library Scaladoc needs TASTY files of stdlib. The failure does not come from Scaladoc but from compilation of stdlib sources with scala3-compiler. Sources of stdlib are copied from Maven and there we still have old version of MatchError. Should we change the way we fetch stdlib sources to dotty repo?

[julienrf]

In order to generate documentation of standard library Scaladoc needs TASTY files of stdlib.

Would it be possible to read those sources from the current repository? What would be the drawbacks of doing so?

[pikinier20]

I only see two projects that depend on stdlib sources: Scaladoc and Stdlib-tasty-tests. I can only see a drawback that documentation published on website would not describe stdlib used by users but the version from repo.

[julienrf]

I agree that end-users of scaladoc should not use the stdlib sources of the repo. IMHO, they should not even fetch the sources from Maven. The TASTy files should be part of the distributed scaladoc tool. But I am digressing.

Here we are in a specific situation where we want to produce the scaladoc of the stdlib itself. Is it necessary to use the TASTy files of the stdlib to do so?

[pikinier20]

Scaladoc can only produce documentation from TASTy files

[odersky]

Lets' suppose we find an unsoundness issue in the type system. This is not so unusual. We want to fix it, but in order to do that we need a small, binary compatible change in the standard library. Previously there were several situations like this, maybe one every couple of months.

But now it means that there is no way we can fix the bug since there is no way to fix the standard library for the tests. So what should we do in a situation like this?

I think we need to rethink our approach to scaladoc generation and also Stdlib-tasty tests.

[julienrf]

@pikinier20 I might have misunderstood how scaladoc works, but it seems to me that the stdlib sources are downloaded even when users want to produce the documentation of their projects? What I wanted to say is that if the TASTy files of the stdlib are necessary to generate the documentation of user projects, we should ship those TASTy files in the scaladoc binaries. Now, when we want to produce the documentation of the stdlib itself, I guess we can just read its TASTy files in the compilation artifacts directory (just like user projects TASTy files are read), and we don’t need to download them from anywhere, right?

Sources of stdlib are copied from Maven and there we still have old version of MatchError. Should we change the way we fetch stdlib sources to dotty repo?

So, what I don’t understand is why we “download” the sources of the stdlib instead of reading them from the file system.

[pikinier20]

Yes, I think we might have misunderstood each other. The problem is that in order to generate documentation of stdlib we need TASTy files but stdlib 2.13 jar doesn't contain those so we need to compile stdlib before generating docs. Now, when published version of stdlib doesn't compile with current compiler, we cannot generate documentation. The solution might be to not fetch sources from maven and get the most recent version that compiles. However, end-users are using stdlib version published to Maven and Scaladoc would then document unreleased version of stdlib.

Also, to clarify: Scaladoc doesn't need stdlib TASTy files to generate docs of end-user libraries.

[odersky]

However, end-users are using stdlib version published to Maven and Scaladoc would then document unreleased version of stdlib.

I think that's unavoidable. And it should be OK, since the two versions are still binary compatible. We generally just do minor changes in order to be able to compile the stdlib under changed rules. The example I showed is a good point in case. Nothing has changed here, we just made explicit that the parameter of MatchError is retained as a field. It was already retained as a field before (that's why the two versions are binary compatible), but we expressed this in a somewhat lazy manner which is no longer supported.

[julienrf]

it should be OK, since the two versions are still binary compatible

Do we check that somewhere?

[odersky]

I think we check it with MiMa, but I am not sure.