-Wunused:all is slow to compile
#19671
Comments
oyvindberg
added
itype:bug
stat:needs triage
|
Just want to say that we've been experiencing a similar issue at $WORK. One of our services uses Scala 3, and the compiling time is roughly ~50-100% longer when I should also point out that since IntelliJ enables |
|
In the linked gist this is what I get with a cold compiler for $WORK.
https://gist.github.com/hamnis/acca674f19cd8cc253ae916f48709fa1 I included the number of source files to indicate size of the module. |
Gedochao
added
itype:performance
area:linting
|
Thanks @hamnis for data! It seems that the problem started with Scala 3.4.0, hopefully it is a regression caused by some change that is easy to revert in linting cases. |
|
I'm unable to share a repro / something more useful right now, but we have anecdotal reports at work of this being experienced in 3.3.3 as well. |
|
Hi, in case this might help someone with optimizing this issue, it seems that one good OS repo to replicate this is regression in compiling time is kyo. Overall, more than half of the compilation time of some modules is spent on checking unused imports: 
|
It doesn't mean that it's *fast* yet, but it is already a significant step in that direction. In particular, this goes in the direction of addressing #19671. The most important commit is "Simplify the logic for checking unused imports.", whose commit message follows: Instead of dealing with entire `tpd.Import`s at the end of the scope, we eagerly flatten them into individual `ImportSelector`s. We store them along with some data, including a mutable flag for whether a selector has been used. This allows to dramatically simplify `isInImport`, as well as more aggressively cache the resolution of selectors. We also get rid of the `IdentityHashMap`. The algorithm is still `O(n*m)` where n is the number of imports in a scope, and m the number of references found in that scope. It is not entirely clear to me whether the previous logic was already `O(n*m)` or worse (it may have included an additional `p` factor for the number of possible selections from a given qualifier). Regardless, it is already quite a bit faster than before, thanks to smaller constant factors.
|
@kyri-petrou How did you get that profiling information? Is this something we can print out from the compiler or something internal to Intellij? |
|
This seems to be improved considerably by #20321. |
Compiler version
3.4.0-RC4
Problem
First I want to celebrate the work that has been done for Scala 3.4 to provide Intellij with timing details so that you can see which phases are slow, which files cause much compilation time and so on. Great work!
So at $WORK we just migrated to Scala 3, and are facing slow compiles. Now I finally had the tools to dig a bit deeper. The first thing I found was that the two unused phases consume more time than typer for many of our sbt modules! I'll show an illustrative example here:
The data shown there seems to be correct, and can be reproduced with sbt.
Clean/compile for the same module with
-Wunused:allenabled (slightly shorter times because of warmer compiler)And without:
I'm also very happy that the unused functionality is back, and hope it can be optimized a bit. For now it's very easy to add all the flags from
sbt-tpolecat, end up with a slow build and conclude that "Scala is slow to compile".Reproducing
So this is a private project, but I suspect it will reproduce elsewhere. I tried for instance lichess lila (a prominent public scala 3 repo).
with unused:
without unused:
I can also be of assistance with for instance profiling data if that helps.
Concurrent linting question
I assume compiler trees are immutable and linting phases don't affect the output trees. Did you look into running them in the background as the compiler progresses?
The text was updated successfully, but these errors were encountered: