Skip to content

GH-113464: A copy-and-patch JIT compiler #113465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 459 commits into from
Closed

GH-113464: A copy-and-patch JIT compiler #113465

Changes from all commits
Commits
Show all changes
459 commits
Select commit Hold shift + click to select a range
8f6c5fe
Remove the ip_offset hack and executor local
brandtbucher Sep 1, 2023
e0888de
Merge branch 'main' into justin
brandtbucher Sep 1, 2023
365f7a2
Get rid of ip_offset
brandtbucher Sep 1, 2023
7fb79b8
Catch up with main
brandtbucher Sep 5, 2023
66a5659
fixup
brandtbucher Sep 5, 2023
f90bd1e
Catch up with main
brandtbucher Sep 13, 2023
a98d4fe
Crank up the trace length for nbody
brandtbucher Sep 13, 2023
dc308ba
Catch up with main
brandtbucher Sep 14, 2023
f2b18d7
Add rough assembly dumps to jit_stencils.h
brandtbucher Sep 14, 2023
e2ae7bc
Make machine code disassemblies more robust
brandtbucher Sep 15, 2023
be88965
Go back to llvm-objdump like the good 'ol days
brandtbucher Sep 15, 2023
56976b3
Remove old debugging struct
brandtbucher Sep 15, 2023
e33b4ab
-fno-pic
brandtbucher Sep 17, 2023
3ac1946
Take a triple (and debug flag) instead of guessing
brandtbucher Sep 18, 2023
50731b5
Use more regular configs and fix debug builds
brandtbucher Sep 21, 2023
e559898
Try using cross-compiles for aarch64
brandtbucher Sep 22, 2023
317ba79
Fix quotes
brandtbucher Sep 22, 2023
07e03a1
apt update
brandtbucher Sep 22, 2023
e649059
No pkg-config
brandtbucher Sep 22, 2023
11c7f44
Fix PGO and disable multiprocessing
brandtbucher Sep 22, 2023
4981573
Add missing excludes
brandtbucher Sep 22, 2023
1688830
Switch to enums
brandtbucher Sep 22, 2023
29936e6
Catch up with main
brandtbucher Sep 22, 2023
ed75bd6
More cleanup
brandtbucher Sep 22, 2023
8488def
Rename some things, and don't constantly regen
brandtbucher Sep 22, 2023
28351a0
Clean up a bunch of stuff
brandtbucher Sep 23, 2023
cc5db79
Patch opargs and operands in the continuation
brandtbucher Sep 24, 2023
f2817bb
Fix GHCCC hack
brandtbucher Sep 24, 2023
fb96638
Blacken
brandtbucher Sep 24, 2023
77e1487
Fix incorrect signature
brandtbucher Sep 24, 2023
5e9259d
-fpic -mcmodel=small
brandtbucher Sep 24, 2023
ff1d6a7
-fno-pic -mcmodel=medium
brandtbucher Sep 24, 2023
af9f114
Catch up with main
brandtbucher Sep 25, 2023
c8b5073
Catch up with justin
brandtbucher Sep 29, 2023
57a8019
Switch to -mcmodel=small -fpic on two platforms
brandtbucher Sep 29, 2023
68a87a1
Ditto for Intel Macs
brandtbucher Sep 29, 2023
f44767b
ELF cleanup
brandtbucher Sep 29, 2023
60d6295
More cleanup
brandtbucher Sep 29, 2023
d634cd3
Catch up with main
brandtbucher Sep 29, 2023
ca4956f
Add R_X86_64_PC32
brandtbucher Sep 29, 2023
092c795
Fixes for aarch64 and i686
brandtbucher Sep 29, 2023
15ba80d
ELF everywhere!
brandtbucher Oct 2, 2023
f2927ae
Fix some leftover bugs
brandtbucher Oct 2, 2023
26abedf
More cleanup
brandtbucher Oct 2, 2023
1cb4a1f
Catch up with main
brandtbucher Oct 2, 2023
b2e47f9
Catch up with justin
brandtbucher Oct 2, 2023
1799b79
Get rid of an old workaround
brandtbucher Oct 2, 2023
aef1e42
Not my fault
brandtbucher Oct 3, 2023
7b33e21
More cleanup
brandtbucher Oct 3, 2023
32b2708
Fix relocations
brandtbucher Oct 3, 2023
94448ed
Get rid of dynamic lookup
brandtbucher Oct 3, 2023
d08d53c
Clean things up for merge
brandtbucher Oct 3, 2023
502ab9c
Get rid of some extra padding
brandtbucher Oct 4, 2023
f87aab2
Clean up the diff
brandtbucher Oct 4, 2023
83492dd
Fix 64-bit Windows
brandtbucher Oct 4, 2023
c4b166f
Clean up formatting
brandtbucher Oct 4, 2023
0c4e147
fixup
brandtbucher Oct 4, 2023
0f23be3
More cleanup
brandtbucher Oct 9, 2023
8c62663
Merge branch 'main' into justin
brandtbucher Oct 9, 2023
3497f8c
Add a short README
brandtbucher Oct 9, 2023
0be489a
More cleanup
brandtbucher Oct 10, 2023
73eb3ce
Catch up with main
brandtbucher Oct 10, 2023
e529809
Reenable more tests
brandtbucher Oct 10, 2023
1a71c1a
Add ip_offset back
brandtbucher Oct 10, 2023
b13056f
Remove some comments
brandtbucher Oct 10, 2023
898dcc2
Comments
brandtbucher Oct 10, 2023
720878f
Load opargs and operands only when needed
brandtbucher Oct 11, 2023
07fb485
Be smarter about opargs and operands
brandtbucher Oct 11, 2023
aa9d3b3
Rerun CI
brandtbucher Oct 12, 2023
8db6838
Add error stub and clean things up more
brandtbucher Oct 13, 2023
e0a69d1
Split stuff into instructions and data
brandtbucher Oct 14, 2023
d33cc95
Fix Windows?
brandtbucher Oct 14, 2023
2c78f70
...fix Windows?
brandtbucher Oct 14, 2023
a62e82d
More fixes...
brandtbucher Oct 14, 2023
878dc80
Last try tonight
brandtbucher Oct 14, 2023
c440e55
Okay, for real
brandtbucher Oct 14, 2023
169acb3
Fix missing ghccc
brandtbucher Oct 15, 2023
d6fa4ad
Clean up opargs and operands
brandtbucher Oct 15, 2023
4ffe087
Reuse static const data
brandtbucher Oct 15, 2023
ff195e2
Clean up relocations
brandtbucher Oct 15, 2023
7de5664
Get the addend from the correct segment on i686
brandtbucher Oct 16, 2023
1963c66
Don't mprotect 0 bytes of memory
brandtbucher Oct 16, 2023
3b54bb6
Catch up with main
brandtbucher Oct 16, 2023
fb3afb5
Fix tier two
brandtbucher Oct 17, 2023
d934ddf
Clean up the diff
brandtbucher Oct 17, 2023
21a9efd
Don't overallocate data
brandtbucher Oct 17, 2023
7eaec09
Fix alignment stuff
brandtbucher Oct 17, 2023
cdf8b1d
Catch up with main
brandtbucher Oct 18, 2023
5f4e950
Catch up with main
brandtbucher Oct 30, 2023
c4bf874
Catch up with main
brandtbucher Nov 2, 2023
eda424d
Catch up with main
brandtbucher Nov 6, 2023
3c816f8
Make BEFORE_WITH a uop
brandtbucher Nov 6, 2023
824cb77
Catch up with main
brandtbucher Nov 7, 2023
7aa253c
Fix bad merge
brandtbucher Nov 7, 2023
4f60977
Clean up the diff a tiny bit
brandtbucher Nov 7, 2023
d65308a
Disable BEFORE_ASYNC_WITH/BEFORE_WITH/INSERT
brandtbucher Nov 8, 2023
fb676a6
Disable the GHCCC hack
brandtbucher Nov 8, 2023
0e8a50a
Don't use ELF everywhere (WIP)
brandtbucher Nov 8, 2023
c997b66
Fix burn-in on LLVM 16
brandtbucher Nov 9, 2023
c26012b
Increase data sharing
brandtbucher Nov 9, 2023
82579d2
Get rid of ghccc hack (for now)
brandtbucher Nov 9, 2023
3930170
Add COFF support back
brandtbucher Nov 9, 2023
71dd8be
Catch up with justin
brandtbucher Nov 9, 2023
d2687e4
Only compile once
brandtbucher Nov 9, 2023
5774f6d
Mach-O!
brandtbucher Nov 11, 2023
cd05c17
Re-enable macOS job
brandtbucher Nov 11, 2023
241ce0f
Catch up with main
brandtbucher Nov 11, 2023
5137145
Catch up with justin
brandtbucher Nov 11, 2023
ffac31b
Fix types and require LLVM 16
brandtbucher Nov 14, 2023
888d9ca
Un-skip some tests
brandtbucher Nov 14, 2023
4611265
Catch up with main
brandtbucher Nov 14, 2023
4d3a081
Catch up with justin
brandtbucher Nov 14, 2023
96c3499
Catch up with main
brandtbucher Nov 14, 2023
cc201a9
Catch up with justin
brandtbucher Nov 14, 2023
0446e98
Catch up with main
brandtbucher Nov 15, 2023
6f86a0f
Remove unnecessary relocation types
brandtbucher Nov 15, 2023
7a7e995
Start cleaning up the allocator
brandtbucher Nov 15, 2023
c4904e4
Clean up and rename stencil stuff
brandtbucher Nov 16, 2023
8021c98
Remove groups and excludes, just for fun
brandtbucher Nov 17, 2023
f17495b
More tweaks
brandtbucher Nov 17, 2023
c2b5b57
fixup
brandtbucher Nov 17, 2023
d39d915
fixup
brandtbucher Nov 17, 2023
249e12a
fixup
brandtbucher Nov 18, 2023
f142c2f
Rework build.py to match jit.c a bit more closely
brandtbucher Nov 18, 2023
6490a71
Fix M1 builds
brandtbucher Nov 19, 2023
27a50cf
Rename stuff and protect readable memory
brandtbucher Nov 19, 2023
89996d9
Catch up with main
brandtbucher Nov 19, 2023
58bd20c
Make the jit an optional option
brandtbucher Nov 20, 2023
7bb54b5
Catch up with main
brandtbucher Nov 20, 2023
7a592f0
Catch up with main
brandtbucher Nov 20, 2023
1a8a63b
Catch up with justin
brandtbucher Nov 20, 2023
40614ee
Skip some more aarch64 tests
brandtbucher Nov 20, 2023
5bbda58
More cleanup
brandtbucher Nov 21, 2023
8342f07
black + mypy
brandtbucher Nov 21, 2023
925f140
Refactor JIT code: Remove deoptimize and error
brandtbucher Nov 21, 2023
4ae9371
Rename trampoline to wrapper, and only warn once
brandtbucher Nov 21, 2023
e295407
Add schema for JSON, and clean things up
brandtbucher Nov 21, 2023
3b64124
Remove deoptimize and error stubs
brandtbucher Nov 21, 2023
4cb7a87
Catch up with main
brandtbucher Nov 21, 2023
5bf5187
Catch up with main
brandtbucher Nov 28, 2023
dfface9
Enable CI temporarily
brandtbucher Nov 28, 2023
a9637a8
Catch up with man
brandtbucher Nov 29, 2023
0466d7a
Stop using the static data hack
brandtbucher Nov 30, 2023
00abf88
Comment out memory debugging code
brandtbucher Nov 30, 2023
a9c613e
Clean up JIT initialization and memory allocation
brandtbucher Nov 30, 2023
094581a
Fix Windows madness
brandtbucher Dec 1, 2023
bc5ba12
Clean up some types
brandtbucher Dec 2, 2023
42d96de
Get rid of runtime trampolines
brandtbucher Dec 6, 2023
1fa6d7b
Fix byteorder issue
brandtbucher Dec 6, 2023
01ffefc
Remove trampoline.c and update STUBS list
brandtbucher Dec 6, 2023
ed95f11
Catch up with main
brandtbucher Dec 6, 2023
772d868
Catch up with justin
brandtbucher Dec 6, 2023
7809cce
Get rid of trampolines
brandtbucher Dec 6, 2023
dcb290a
Use dataclasses.replace
brandtbucher Dec 6, 2023
86ed6a1
Catch up with main
brandtbucher Dec 7, 2023
8349218
Lots of cleanup
brandtbucher Dec 7, 2023
e8176b8
Don't leak memory, and don't emit wrappers!
brandtbucher Dec 7, 2023
c3e2aab
Clean up relocations and fix Windows nojit builds
brandtbucher Dec 7, 2023
3af4a4b
More Windows fixes
brandtbucher Dec 7, 2023
a4a2862
More cleanup
brandtbucher Dec 8, 2023
780b678
Make it possible to enable the tier 2 interpreter
brandtbucher Dec 8, 2023
9901843
A bunch of cleanup (and proper debug compiles)
brandtbucher Dec 11, 2023
ab556e7
Don't emit unwind info (and fix __bzero)
brandtbucher Dec 11, 2023
c458df4
Catch up with main
brandtbucher Dec 11, 2023
9c1867b
asm not needed for large -mcmodel=large
brandtbucher Dec 11, 2023
882263c
Catch up with main
brandtbucher Dec 13, 2023
522adc5
Catch up with main
brandtbucher Dec 13, 2023
86d4fd7
fixup
brandtbucher Dec 13, 2023
94f0877
Tons of cleanup... prepare to revert!
brandtbucher Dec 14, 2023
bfa6533
fixup
brandtbucher Dec 14, 2023
c5c44c8
fixup
brandtbucher Dec 14, 2023
56fbb07
fixup
brandtbucher Dec 14, 2023
a8931fe
...fixup?
brandtbucher Dec 14, 2023
1fcdc94
...fixup.
brandtbucher Dec 14, 2023
64f903c
Put pyconfig.h where it needs to be
brandtbucher Dec 14, 2023
4e67db4
Fix Windows/out-of-tree/hashing/verbosity
brandtbucher Dec 14, 2023
78f01ae
Fix bit manipulation bugs in jit.c
brandtbucher Dec 14, 2023
73ed82b
Catch up with main
brandtbucher Dec 14, 2023
53c35f3
Clean up configure
brandtbucher Dec 15, 2023
dc55796
Catch up with main
brandtbucher Dec 15, 2023
770a480
Clean up jit.vcxproj
brandtbucher Dec 16, 2023
8998952
fixup
brandtbucher Dec 16, 2023
3d04346
Put jit_stencils.h in the build root
brandtbucher Dec 16, 2023
46063fb
fixup
brandtbucher Dec 16, 2023
8addbbe
Catch up with main
brandtbucher Dec 16, 2023
6b19d7c
Clean up the AArch64 stuff
brandtbucher Dec 18, 2023
b63610e
Break stuff up
brandtbucher Dec 19, 2023
44a024a
More cleanup
brandtbucher Dec 20, 2023
af7606d
Merge Parser and Target
brandtbucher Dec 20, 2023
f238057
Turn relocations into holes quickly
brandtbucher Dec 20, 2023
50b0df8
Fix AArch64 Macs.
brandtbucher Dec 24, 2023
0d358ef
Final-ish cleanup
brandtbucher Dec 24, 2023
323a9d6
blurb add
brandtbucher Dec 24, 2023
d36c9de
Enable JIT tests on pull_request
brandtbucher Dec 24, 2023
7026d0c
Nevermind
brandtbucher Dec 24, 2023
412c68d
Catch up with main
brandtbucher Jan 2, 2024
b1f1c9f
Add Include/internal/mimalloc to include path
brandtbucher Jan 2, 2024
e9faa27
Catch up with main
brandtbucher Jan 2, 2024
4229327
Re-enable JIT CI
brandtbucher Jan 2, 2024
2643439
Lots of little cleanups from code review
brandtbucher Jan 3, 2024
3342775
Catch up with main
brandtbucher Jan 3, 2024
c1b7007
Move some logic out of try blocks
brandtbucher Jan 3, 2024
20ad5f5
Clean up some type annotations
brandtbucher Jan 3, 2024
c518c09
Catch up with main
brandtbucher Jan 9, 2024
5e229ed
Consider target when caching the JIT stencils
brandtbucher Jan 11, 2024
d01a2c7
Clarify why LLVM is required
brandtbucher Jan 11, 2024
4266f5a
Catch up with main (and never interpret tier two on JIT builds)
brandtbucher Jan 11, 2024
6ebd085
fixup
brandtbucher Jan 11, 2024
2d17e22
JIT everything (even the counter thing)
brandtbucher Jan 11, 2024
afbc1f5
fixup
brandtbucher Jan 11, 2024
5315b1d
Wrap machine code bytes.
brandtbucher Jan 12, 2024
6b84187
Use better hex formatting
brandtbucher Jan 12, 2024
675dd0c
Catch up with main
brandtbucher Jan 12, 2024
bc6763e
Clean up the LLVM utilities
brandtbucher Jan 12, 2024
fb62bc8
Remove duplicated code (and unnecessary sorting)
brandtbucher Jan 12, 2024
344683f
Fix machine code formatting
brandtbucher Jan 12, 2024
2636ece
Clean up the build script (mostly comments and renaming)
brandtbucher Jan 12, 2024
efbe31a
Split things up a bit more
brandtbucher Jan 15, 2024
23fad98
Add missing braces to Hole's C dump
brandtbucher Jan 15, 2024
44b194f
Move relocation processing to StencilGroup
brandtbucher Jan 15, 2024
eb5f21d
Move symbols to StencilGroup
brandtbucher Jan 15, 2024
0dca530
Make some nested schema items private
brandtbucher Jan 15, 2024
a249170
Move more logic to from Target to StencilGroup
brandtbucher Jan 15, 2024
1a956bd
Move the version check to where it's needed
brandtbucher Jan 15, 2024
e78341d
Catch up with main
brandtbucher Jan 15, 2024
0bba20f
Remove bad asserts
brandtbucher Jan 15, 2024
b4474d0
Catch up with main
brandtbucher Jan 16, 2024
41162ca
Rework PCbuild stuff based on PR review
brandtbucher Jan 17, 2024
debb24d
Make JIT workflow conditional on files changed, add a dispatch button…
brandtbucher Jan 17, 2024
4f656e0
Make the README more future-proof
brandtbucher Jan 17, 2024
0c9c806
Set up incremental builds and fix PGO
brandtbucher Jan 17, 2024
79b1bbf
Raise instead of warning, rename bit-shifting arguments, and move pat…
brandtbucher Jan 18, 2024
6643773
Fix error message
brandtbucher Jan 18, 2024
663ba31
Catch up with main
brandtbucher Jan 18, 2024
21ac0f2
Add const qualifier
brandtbucher Jan 18, 2024
b352c6b
Catch up with main
brandtbucher Jan 20, 2024
763f527
Actions runners are beefier now
brandtbucher Jan 20, 2024
b15529c
Fix some typos
brandtbucher Jan 20, 2024
1cee739
Reorder ELF and COFF
brandtbucher Jan 20, 2024
81d0fb6
Don't include Python/, add a --force option, and unify section/symbol…
brandtbucher Jan 25, 2024
437bc53
Make _PyJIT_Compile take a trace and length
brandtbucher Jan 25, 2024
26e248a
Move header comment to build.py
brandtbucher Jan 25, 2024
f3984e6
Clean up the schema
brandtbucher Jan 25, 2024
113a1f1
Reorder some keys
brandtbucher Jan 25, 2024
6648d84
Rearrange (and add) comments for patch function
brandtbucher Jan 25, 2024
46bee49
Use generated pyconfig.h directory
brandtbucher Jan 25, 2024
c33c78f
Catch up with main
brandtbucher Jan 25, 2024
4d3e207
Add back Python/ include
brandtbucher Jan 25, 2024
581036e
Catch up with main
brandtbucher Jan 29, 2024
b00e021
Merge branch 'main' into justin
brandtbucher Jan 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

These merge commits were added into this branch cleanly.

There are no new changes to show.