Use artifacts on windows to get the compiler #91

KristofferC · 2020-02-05T16:20:43Z

No description provided.

rapus95 · 2020-02-05T16:52:18Z

I've been hit by that:

julia> create_sysimage([:Makie, :MakieLayout], replace_default=true)
[ Info: PackageCompilerX: creating system image object file, this might take a while...
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: <unknown-file>:0: syntax error
Warning: .drectve `-export:ccalllib_C:\Users\Matthis\.julia\packages\FreeType\2dE5w\deps\usr\bin\libfreetype-6.dll,data ' unrecognized
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export FreeType: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export Users: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export ccalllib_C:: symbol not found
C:/Users/Matthis/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export packages: symbol not found
collect2.exe: error: ld returned 1 exit status
ERROR: failed process: Process(`'C:\Users\Matthis\.julia\artifacts\572b61b5075459e3ed62317e674398166ca98dd4\mingw64\bin\gcc.exe' -m64 -shared '-LC:\Users\Matthis\AppData\Local\Programs\Julia\Julia-1.4.0-rc1\bin' -o 'C:\Users\Matthis\AppData\Local\Temp\jl_dnMjRC\sys.dll' -Wl,--whole-archive 'C:\Users\Matthis\AppData\Local\Temp\jl_qhN0xvaymk.o' -Wl,--no-whole-archive -ljulia -Wl,--export-all-symbols`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] create_sysimg_from_object_file(::String, ::String) at C:\Users\Matthis\.julia\packages\PackageCompilerX\nfbuY\src\PackageCompilerX.jl:366
 [5] create_sysimage(::Array{Symbol,1}; sysimage_path::Nothing, project::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, base_sysimage::Nothing, isapp::Bool) at C:\Users\Matthis\.julia\packages\PackageCompilerX\nfbuY\src\PackageCompilerX.jl:338
 [6] top-level scope at REPL[207]:1
caused by [exception 1]
IOError: symlink: operation not permitted (EPERM)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] symlink(::String, ::String) at .\file.jl:955
 [3] probe_symlink_creation(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\PlatformEngines.jl:132
 [4] unpack(::String, ::String; verbose::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\PlatformEngines.jl:992
 [5] download_verify_unpack(::String, ::String, ::String; tarball_path::Nothing, ignore_existence::Bool, force::Bool, verbose::Bool, quiet_download::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\PlatformEngines.jl:1168
 [6] download_artifact(::Base.SHA1, ::String, ::String; verbose::Bool, quiet_download::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Artifacts.jl:750
 [7] (::Pkg.Artifacts.var"#44#46"{Bool,Bool,Base.SHA1,String})() at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Artifacts.jl:890
 [8] with_show_download_info(::Pkg.Artifacts.var"#44#46"{Bool,Bool,Base.SHA1,String}, ::String, ::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Artifacts.jl:907
 [9] ensure_artifact_installed(::String, ::Dict{String,Any}, ::String; platform::Pkg.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Artifacts.jl:889
 [10] do_artifact_str(::String, ::Dict{String,Any}, ::String, ::Module) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\Artifacts.jl:1023
 [11] #invokelatest#1 at .\essentials.jl:712 [inlined]
 [12] invokelatest at .\essentials.jl:711 [inlined]
 [13] get_compiler() at C:\Users\Matthis\.julia\packages\PackageCompilerX\nfbuY\src\PackageCompilerX.jl:32
 [14] create_sysimg_from_object_file(::String, ::String) at C:\Users\Matthis\.julia\packages\PackageCompilerX\nfbuY\src\PackageCompilerX.jl:364
 [15] create_sysimage(::Array{Symbol,1}; sysimage_path::Nothing, project::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, base_sysimage::Nothing, isapp::Bool) at C:\Users\Matthis\.julia\packages\PackageCompilerX\nfbuY\src\PackageCompilerX.jl:338
 [16] top-level scope at REPL[207]:1

StefanKarpinski · 2020-02-05T16:54:58Z

Looks like symlinks in the artifact on Windows.

KristofferC · 2020-02-05T16:57:46Z

That is just Pkg checking if the machine supports symlinks:

 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] symlink(::String, ::String) at .\file.jl:955
 [3] probe_symlink_creation(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Pkg\src\PlatformEngines.jl:132

https://github.com/JuliaLang/Pkg.jl/blob/831c11addd93f234df2350623b57f92138df6a14/src/PlatformEngines.jl#L112-L143

rapus95 · 2020-02-05T16:59:43Z

Well, I guess, if it's probing it should be caught. Crashing the whole program doesn't seem like its intended behaviour 😁

KristofferC · 2020-02-05T17:02:29Z

It isn't crashing from that, it is crashing from

ERROR: failed process: Process(`'C:\Users\Matthis\.julia\artifacts\572b61b5075459e3ed62317e674398166ca98dd4\mingw64\bin\gcc.exe' -m64 -shared '-LC:\Users\Matthis\AppData\Local\Programs\Julia\Julia-1.4.0-rc1\bin' -o 'C:\Users\Matthis\AppData\Local\Temp\jl_dnMjRC\sys.dll' -Wl,--whole-archive 'C:\Users\Matthis\AppData\Local\Temp\jl_qhN0xvaymk.o' -Wl,--no-whole-archive -ljulia -Wl,--export-all-symbols`, ProcessExited(1)) [1]

It seems that the exception stack is messed up...

rapus95 · 2020-02-05T17:09:28Z

Is it possible that the IOError somehow hasn't been cleared up and thus was taken as cause for the next native error (the pipeline one)?

KristofferC · 2020-02-05T17:09:59Z

Yeah, I think it is a red herring here.

rapus95 · 2020-02-05T17:21:52Z

Just seeing it now, it looks like the -L is misplaced.

KristofferC · 2020-02-05T17:27:41Z

This repros with:

sysimage.jl:

Base.reinit_stdio()
Base.init_load_path()
Base.init_depot_path()

using FreeTypeAbstraction

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

then run:

julia --startup-file=no --output-o sys.o -J"C:\\Users\\Kristoffer\\AppData\\Local\\Julia-1.3.1\\lib\\julia\\sys.dll" sysimage.jl

gcc -shared -o sys.so -Wl,--whole-archive sys.o -Wl,--no-whole-archive -L"C:\\Users\\Kristoffer\\AppData\\Local\\Julia-1.3.1\\lib" -ljulia

I don't really understand how the FreeType symbols get inside the object file...

KristofferC · 2020-02-05T17:38:29Z

Commenting out the __init__ seems to fix it:

https://github.com/JuliaGraphics/FreeTypeAbstraction.jl/blob/055ee900f4786ab18e5c167165805cbae2794f4e/src/FreeTypeAbstraction.jl#L18-L30

davidanthoff · 2020-02-05T17:56:26Z

I'm getting the same error:

[ Info: PackageCompilerX: creating system image object file, this might take a while...
ERROR: LoadError: failed process: Process(`'C:\Users\david\.julia\artifacts\572b61b5075459e3ed62317e674398166ca98dd4\mingw64\bin\gcc.exe' -m64 -shared '-LC:\Users\david\AppData\Local\Julia-1.3.1\bin' -o 'C:\Users\david\.julia\environments\cctest\JuliaSysimage.dll' -Wl,--whole-archive 'C:\Users\david\AppData\Local\Temp\jl_7490.tmp.o' -Wl,--no-whole-archive -ljulia -Wl,--export-all-symbols`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] create_sysimg_from_object_file(::String, ::String) at c:\Users\david\source\julia-vscode\scripts\tasks\sysimageenv\PackageCompilerX\src\PackageCompilerX.jl:366
 [5] #create_sysimage#5(::String, ::String, ::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::String, ::Nothing, ::Bool, ::typeof(PackageCompilerX.create_sysimage), ::Array{Symbol,1}) at c:\Users\david\source\julia-vscode\scripts\tasks\sysimageenv\PackageCompilerX\src\PackageCompilerX.jl:338
 [6] (::PackageCompilerX.var"#kw##create_sysimage")(::NamedTuple{(:sysimage_path, :project),Tuple{String,String}}, ::typeof(PackageCompilerX.create_sysimage), ::Array{Symbol,1}) at .\none:0
 [7] top-level scope at c:\Users\david\source\julia-vscode\scripts\tasks\task_compileenv.jl:17
 [8] include at .\boot.jl:328 [inlined]
 [9] include_relative(::Module, ::String) at .\loading.jl:1105
 [10] include(::Module, ::String) at .\Base.jl:31
 [11] exec_options(::Base.JLOptions) at .\client.jl:287       
 [12] _start() at .\client.jl:460
in expression starting at c:\Users\david\source\julia-vscode\scripts\tasks\task_compileenv.jl:17
The terminal process terminated with exit code: 1

The env I'm trying to build has only one package in it (DataTables.jl), which has no dep on FreeTypeAbstraction.jl.

KristofferC · 2020-02-05T22:06:48Z

@davidanthoff, You're issue is different but I think I know what it is and how to fix it. Will mention when you can try again.

Regarding the original issue, the problem is a weird symbol in the object file:

nm sys.o
...
0000000000019d88 B ccalllib_C:\Users\Kristoffer Carlsson\.julia\dev\FreeType\deps\usr\bin\libfreetype-6.dll
00000000000122a0 B ccalllib_credui.dll
0000000000000900 B ccalllib_kernel32
000000000001f188 B ccalllib_libcholmod

Comes from https://github.com/JuliaLang/julia/blob/f2d68ad6eb6b735f5beed779b9570f0d9fbbab28/src/ccall.cpp#L71. That absolute path is weird...

KristofferC · 2020-02-06T07:52:08Z

@rapus95 Can you try with JuliaGraphics/FreeType.jl#34.

With that I can make a sysimage with Makie and MakieLayout.

KristofferC · 2020-02-06T08:25:58Z

@davidanthoff, could you test again with this branch please.

rapus95 · 2020-02-06T08:26:36Z

@rapus95 Can you try with JuliaGraphics/FreeType.jl#34.

With that I can make a sysimage with Makie and MakieLayout.

I'd like to but I'm a bit upset: FreeType doesn't have a Project.toml thus I can't dev it. Thus I need to do a manual clone, switch branches to PR and then somehow explain Julia to use that one instead of the one currently in use. Remember, there's no Project.toml, thus no UUID...

KristofferC · 2020-02-06T08:29:29Z

In 1.3 it should be enough to do

add https://github.com/KristofferC/FreeType.jl#kc/use_soname

rapus95 · 2020-02-06T08:30:23Z

I'm on 1.4-rc1:

(stuff) pkg> add https://github.com/KristofferC/FreeType.jl#kc/use_soname
    Cloning git-repo `https://github.com/KristofferC/FreeType.jl`
   Updating git-repo `https://github.com/KristofferC/FreeType.jl`
ERROR: could not find project file in package at https://github.com/KristofferC/FreeType.jl

KristofferC · 2020-02-06T08:32:44Z

Yes, so try with 1.3? :P

rapus95 · 2020-02-06T08:33:28Z

Yes, so try with 1.3? :P

Damn, got to install that first 😞 Why was it made an error on 1.4?

KristofferC · 2020-02-06T08:35:43Z

Because packages need to have a Project file now. There was a lot of awkwardness in Pkg of trying to handle REQUIRE and Project files at the same time so it was ripped out.

rapus95 · 2020-02-06T08:55:08Z

It's definitively missing from NEWS: https://github.com/JuliaLang/julia/blob/release-1.4/NEWS.md

KristofferC · 2020-02-06T09:02:30Z

This isn't really the right place to talk about this. If you don't feel like trying it out, it is ok. I tried it out locally myself and it seems to work.

rapus95 · 2020-02-06T09:22:42Z

Now on 1.3.1:

julia> create_sysimage([:Makie, :MakieLayout], replace_default=true)
[ Info: PackageCompilerX: creating system image object file, this might take a while...
[ Info: PackageCompilerX: default sysimg replaced, restart Julia for the new sysimg to be in effect
ERROR: IOError: unlink: permission denied (EACCES)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] unlink(::String) at .\file.jl:781
 [3] #rm#9(::Bool, ::Bool, ::typeof(rm), ::String) at .\file.jl:261
 [4] #rm at .\none:0 [inlined]
 [5] #checkfor_mv_cp_cptree#10(::Bool, ::typeof(Base.Filesystem.checkfor_mv_cp_cptree), ::String, ::String, ::String) at .\file.jl:299
 [6] #checkfor_mv_cp_cptree at .\none:0 [inlined]
 [7] #mv#13 at .\file.jl:385 [inlined]
 [8] #mv at .\none:0 [inlined]
 [9] #create_sysimage#7(::Nothing, ::String, ::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::String, ::Nothing, ::Bool, ::typeof(create_sysimage), ::Array{Symbol,1}) at C:\Users\Matthis\.julia\packages\PackageCompilerX\3cLet\src\PackageCompilerX.jl:356
 [10] (::PackageCompilerX.var"#kw##create_sysimage")(::NamedTuple{(:replace_default,),Tuple{Bool}}, ::typeof(create_sysimage), ::Array{Symbol,1}) at .\none:0
 [11] top-level scope at REPL[7]:1

Restarting Julia won't have the new image. So I guess the Info about "default replaced" is placed too early 🙈

KristofferC · 2020-02-06T09:38:38Z

Great that means that it worked.

The error you get there is because Windows is sometimes very grumpy about replacing files that are already opened (here the sysimage.dll which is open in the running Julia session). I don't really have a good solution to that other than to just manually replace the default sysimage when Julia is exited.

KristofferC · 2020-02-06T09:40:49Z

I'll tweak the docs to mention that.

aminya · 2020-02-06T09:43:10Z

Yay 🚀

rapus95 · 2020-02-06T10:02:00Z

The error you get there is because Windows is sometimes very grumpy about replacing files that are already opened (here the sysimage.dll which is open in the running Julia session). I don't really have a good solution to that other than to just manually replace the default sysimage when Julia is exited.

https://serverfault.com/questions/503721/replacing-dll-files-while-the-application-is-running
There is an intended way to replace a dll in use! 🎉
Both cmd and Base.Filesystem.rename seem to work 🎉
For further progress see #95

davidanthoff · 2020-02-06T18:14:35Z

This works if I try it with a simple env, yay! With more complicated ones I get errors, but presumably that is just a package specific thing.

This is excellent, because I can ship this support out of the box with the VS Code extension now.

KristofferC · 2020-02-06T18:39:11Z

Could you post an example of the error? I want to se what it looks like.

KristofferC mentioned this pull request Feb 5, 2020

hoist the get_compiler call #93

Closed

staticfloat mentioned this pull request Feb 5, 2020

ccall() with an absolute library path while outputting an .o file causes linker issues on Windows JuliaLang/julia#34680

Closed

KristofferC added 4 commits February 6, 2020 08:20

artifact for win wip

2974a27

Update index.md

fee1c1a

actually test

7b592d8

fixes

6d30a50

KristofferC mentioned this pull request Feb 6, 2020

use the name of the library instead of the full path to work around julia bug JuliaGraphics/FreeType.jl#34

Closed

KristofferC force-pushed the kc/artifact branch 2 times, most recently from 4d493e1 to 6d30a50 Compare February 6, 2020 08:21

oops

58277ee

KristofferC merged commit 1e5e834 into master Feb 6, 2020

delete-merged-branch bot deleted the kc/artifact branch February 6, 2020 09:41

KristofferC mentioned this pull request Feb 6, 2020

win: compiler chain as artifact dependency #81

Closed

Uh oh!

Use artifacts on windows to get the compiler #91

Use artifacts on windows to get the compiler #91

Uh oh!

Conversation

KristofferC commented Feb 5, 2020

Uh oh!

rapus95 commented Feb 5, 2020

Uh oh!

StefanKarpinski commented Feb 5, 2020

Uh oh!

KristofferC commented Feb 5, 2020

Uh oh!

rapus95 commented Feb 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Feb 5, 2020

Uh oh!

rapus95 commented Feb 5, 2020

Uh oh!

KristofferC commented Feb 5, 2020

Uh oh!

rapus95 commented Feb 5, 2020

Uh oh!

KristofferC commented Feb 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Feb 5, 2020

Uh oh!

davidanthoff commented Feb 5, 2020

Uh oh!

KristofferC commented Feb 5, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

aminya commented Feb 6, 2020

Uh oh!

rapus95 commented Feb 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidanthoff commented Feb 6, 2020

Uh oh!

KristofferC commented Feb 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rapus95 commented Feb 5, 2020 •

edited

Loading

KristofferC commented Feb 5, 2020 •

edited

Loading

rapus95 commented Feb 6, 2020 •

edited

Loading

rapus95 commented Feb 6, 2020 •

edited

Loading

rapus95 commented Feb 6, 2020 •

edited

Loading

rapus95 commented Feb 6, 2020 •

edited

Loading