error with Julia 1.10 when loading packages that trigger precompilation

[matthias314]

Suppose I want to load a package, say TestPkg.jl with content

module TestPkg end

If the package triggers precompilation, then in IJulia with kernel Julia 1.10.0-beta1 I get

In [1]: using TestPkg
MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint})
The applicable method may be too new: running in world age 31360, while current world is 31378.

The same happens for import TestPkg.

With command-line Julia 1.10 or IJulia with kernel 1.9 I get the expected result,

In [1]: using TestPkg
[ Info: Precompiling TestPkg [top-level]

I'm using Jupyter Notebook 6.3.0 with IJulia 1.24.2.

EDIT: The problem persists with Julia 1.10.0-beta2. Also, for me it appears even for trivial packages like TestPkg above, but only with IJulia: If I create a script test.jl with the line using TestPkg, then both julia test.jl and julia -L test.jl work fine if TestPkg needs precompilation.

[svilupp]

+1 on 1.10 beta2! I presume it's the same root cause.
It broke a lot of our reports, but I could swear it used to work on 1.10 beta1 - I need to try that.

An error occurred while executing the following cell:

using MyPkg

MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint})
The applicable method may be too new: running in world age 31373, while current world is 31593.

Closest candidates are:
pipe_writer(::IJulia.IJuliaStdio) (method too new to be called from this world context.)
@ IJulia ~/.julia/packages/IJulia/Vo51o/src/stdio.jl:15
pipe_writer(::Base.ProcessChain)
@ Base process.jl:36
pipe_writer(::Base.Process)
@ Base process.jl:22
...

Stacktrace:
[1] setup_stdio(stdio::IJulia.IJuliaStdio{Base.PipeEndpoint}, readable::Bool)
@ Base ./process.jl:260
[2] setup_stdios(f::Base.var"#784#785"{Cmd}, stdios::Vector{Union{RawFD, IO}})
@ Base ./process.jl:221
[3] _spawn
@ Base ./process.jl:138 [inlined]
[4] open(cmds::Cmd, stdio::Base.DevNull; write::Bool, read::Bool)
@ Base ./process.jl:393
[5] open (repeats 2 times)
@ Base ./process.jl:383 [inlined]
[6] open(::Pkg.Registry.var"#19#22"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd; kwargs::@kwargs{})
@ Base ./process.jl:415
[7] open
@ Base ./process.jl:414 [inlined]
[8] uncompress_registry(tar_gz::String)
@ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:261
[9] Pkg.Registry.RegistryInstance(path::String)
@ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:328
[10] reachable_registries(; depots::Vector{String})
@ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:434
[11] reachable_registries
@ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:404 [inlined]
[12] Context
@ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Types.jl:395 [inlined]
[13] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::IJulia.IJuliaStdio{Base.PipeEndpoint}, kwargs::@kwargs{_from_loading::Bool})
@ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:150
[14] precompile
@ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
[15] #precompile#114
@ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:146 [inlined]
[16] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:1957
[17] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1806
[18] #invoke_in_world#3
@ Base ./essentials.jl:921 [inlined]
[19] invoke_in_world
@ Base ./essentials.jl:918 [inlined]
[20] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:1797
[21] macro expansion
@ Base ./loading.jl:1784 [inlined]
[22] macro expansion
@ Base ./lock.jl:267 [inlined]
[23] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:1747
[24] #invoke_in_world#3
@ Base ./essentials.jl:921 [inlined]
[25] invoke_in_world
@ Base ./essentials.jl:918 [inlined]
[26] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1740
LoadError: MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint})
The applicable method may be too new: running in world age 31373, while current world is 31593.

Closest candidates are:
pipe_writer(::IJulia.IJuliaStdio) (method too new to be called from this world context.)
@ IJulia ~/.julia/packages/IJulia/Vo51o/src/stdio.jl:15
pipe_writer(::Base.ProcessChain)

IJulia: 1.24.2

Versioninfo:

Julia Version 1.10.0-beta2
Commit a468aa198d0 (2023-08-17 06:27 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 1 on 6 virtual cores

[svilupp]

It seems to be similar to: JuliaLang/julia#28042

I can confirm that the issue is 1.10 beta2.

I can run it fine on 1.9 and 1.10 beta1.

I haven't had much luck with a reproducible example yet; it's easy to repro with our internal packages, but not with mock ones.

[svilupp]

This seems to be replicable on my machine:

generate a random module, eg, ABC/ABC.jl

module ABC

greet() = print("Hello World!")

struct A1 end
struct B1 end

greet(::A1) = print("Hello World!")

x(::A1) = "x"

end # module ABC

In your script simply have:

using StatsPlots # not sure why but it helps to have some complex package there
using ABC
ABC.greet(ABC.A1)

If you first open REPL and precompile it, IJulia will work fine.
If you make some changes, eg, change x(::A1) = "x" to x(::B1) = "x" (which requires precompilation), it will throw the error.
If you precompile "manually" by opening a fresh session and running the code, IJulia will work fine again.

Notes:

• If you have any daemon running, you need to kill it first (eg, if you use Quarto)

EDIT: sometimes the change has to be bigger (eg, introduce a new struct and a new function with a method for it) to trigger the issue

EDIT2: For whatever reason, putting this at the top solves the issue???
using Pkg; Pkg.precompile(["ABC"]; strict=true); But it works only in this toy example, not in real-world scenarios.

[svilupp]

Quick update:

• 1.9.3 works perfectly fine
• 1.10 beta1 consistently fails
• 1.10 beta2 consistently fails
• 1.11 main consistently fails

The only workaround so far is to add explicit precompile (using Pkg; Pkg.precompile()), but it works only for the MWE. It doesn't work in my real-world use cases, so I had to revert to 1.9

[svilupp]

It should be fixed by JuliaLang/julia#51397

I believe this issue can be closed