Skip to content

bpo-27129: Use instruction offsets, not byte offsets, in bytecode and internally. #25069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 1, 2021
Merged

bpo-27129: Use instruction offsets, not byte offsets, in bytecode and internally. #25069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
05a6c62
Use instruction offset, rather than bytecode offset. Streamlines inte…
markshannon Mar 26, 2021
5d3c577
Fix up dis and its test.
markshannon Mar 26, 2021
2a93cf1
Fix up libpython.py
markshannon Mar 29, 2021
a6f3a79
Add NEWS item
markshannon Mar 29, 2021
db2f86b
Restore use of INSTR_OFFSET macro.
markshannon Mar 29, 2021
325d135
Use more appropriate API for getting lineno.
markshannon Mar 30, 2021
3d3b995
Use correct version.
markshannon Mar 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions Lib/dis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,8 +338,11 @@ def _get_instructions_bytes(code, varnames=None, names=None, constants=None,
argval, argrepr = _get_const_info(arg, constants)
elif op in hasname:
argval, argrepr = _get_name_info(arg, names)
elif op in hasjabs:
argval = arg*2
argrepr = "to " + repr(argval)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not go a step further and divide the labels in dis output by 2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of reasons.

  • I wanted to keep changes to a minimum.
  • There is no guarantee that instructions will remain the same size, so using byte offsets keeps things more consistent across versions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. However now we have a confusing discrepancy between what's displayed and how we're encouraged to think about the bytecode in the C code. I guess there's no perfect answer, so status quo wins.

elif op in hasjrel:
argval = offset + 2 + arg
argval = offset + 2 + arg*2
argrepr = "to " + repr(argval)
elif op in haslocal:
argval, argrepr = _get_name_info(arg, varnames)
Expand Down Expand Up @@ -437,9 +440,9 @@ def findlabels(code):
for offset, op, arg in _unpack_opargs(code):
if arg is not None:
if op in hasjrel:
label = offset + 2 + arg
label = offset + 2 + arg*2
elif op in hasjabs:
label = arg
label = arg*2
else:
continue
if label not in labels:
Expand Down
1 change: 1 addition & 0 deletions Lib/importlib/_bootstrap_external.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,7 @@ def _write_atomic(path, data, mode=0o666):
# Python 3.10a2 3432 (Function annotation for MAKE_FUNCTION is changed from dict to tuple bpo-42202)
# Python 3.10a2 3433 (RERAISE restores f_lasti if oparg != 0)
# Python 3.10a6 3434 (PEP 634: Structural Pattern Matching)
# Python 3.10a7 3435 Use instruction offsets (as opposed to byte offsets).

#
# MAGIC must change whenever the bytecode emitted by the compiler may no
Expand Down
60 changes: 30 additions & 30 deletions Lib/test/test_dis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,10 @@ def bug708901():
%3d 6 CALL_FUNCTION 2
8 GET_ITER
>> 10 FOR_ITER 4 (to 16)
>> 10 FOR_ITER 2 (to 16)
12 STORE_FAST 0 (res)
%3d 14 JUMP_ABSOLUTE 10
%3d 14 JUMP_ABSOLUTE 5 (to 10)
%3d >> 16 LOAD_CONST 0 (None)
18 RETURN_VALUE
Expand Down Expand Up @@ -276,11 +276,11 @@ def bug42562():
10 INPLACE_ADD
12 STORE_NAME 0 (x)
2 14 JUMP_ABSOLUTE 6
2 14 JUMP_ABSOLUTE 3 (to 6)
"""

dis_traceback = """\
%3d 0 SETUP_FINALLY 14 (to 16)
%3d 0 SETUP_FINALLY 7 (to 16)
%3d 2 LOAD_CONST 1 (1)
4 LOAD_CONST 2 (0)
Expand All @@ -293,11 +293,11 @@ def bug42562():
%3d >> 16 DUP_TOP
18 LOAD_GLOBAL 0 (Exception)
20 JUMP_IF_NOT_EXC_MATCH 58
20 JUMP_IF_NOT_EXC_MATCH 29 (to 58)
22 POP_TOP
24 STORE_FAST 0 (e)
26 POP_TOP
28 SETUP_FINALLY 20 (to 50)
28 SETUP_FINALLY 10 (to 50)
%3d 30 LOAD_FAST 0 (e)
32 LOAD_ATTR 1 (__traceback__)
Expand Down Expand Up @@ -358,7 +358,7 @@ def _tryfinallyconst(b):
b()

dis_tryfinally = """\
%3d 0 SETUP_FINALLY 12 (to 14)
%3d 0 SETUP_FINALLY 6 (to 14)
%3d 2 LOAD_FAST 0 (a)
4 POP_BLOCK
Expand All @@ -377,7 +377,7 @@ def _tryfinallyconst(b):
)

dis_tryfinallyconst = """\
%3d 0 SETUP_FINALLY 12 (to 14)
%3d 0 SETUP_FINALLY 6 (to 14)
%3d 2 POP_BLOCK
Expand Down Expand Up @@ -450,13 +450,13 @@ def foo(x):
Disassembly of <code object <listcomp> at 0x..., file "%s", line %d>:
%3d 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 12 (to 18)
>> 4 FOR_ITER 6 (to 18)
6 STORE_FAST 1 (z)
8 LOAD_DEREF 0 (x)
10 LOAD_FAST 1 (z)
12 BINARY_ADD
14 LIST_APPEND 2
16 JUMP_ABSOLUTE 4
16 JUMP_ABSOLUTE 2 (to 4)
>> 18 RETURN_VALUE
""" % (dis_nested_1,
__file__,
Expand Down Expand Up @@ -1009,7 +1009,7 @@ def jumpy():
Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=10, argrepr='10', offset=2, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=4, starts_line=None, is_jump_target=False),
Instruction(opname='GET_ITER', opcode=68, arg=None, argval=None, argrepr='', offset=6, starts_line=None, is_jump_target=False),
Instruction(opname='FOR_ITER', opcode=93, arg=34, argval=44, argrepr='to 44', offset=8, starts_line=None, is_jump_target=True),
Instruction(opname='FOR_ITER', opcode=93, arg=17, argval=44, argrepr='to 44', offset=8, starts_line=None, is_jump_target=True),
Instruction(opname='STORE_FAST', opcode=125, arg=0, argval='i', argrepr='i', offset=10, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=12, starts_line=4, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=14, starts_line=None, is_jump_target=False),
Expand All @@ -1018,21 +1018,21 @@ def jumpy():
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=20, starts_line=5, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=22, starts_line=None, is_jump_target=False),
Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', argrepr='<', offset=24, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=30, argval=30, argrepr='', offset=26, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=8, argval=8, argrepr='', offset=28, starts_line=6, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=15, argval=30, argrepr='to 30', offset=26, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=4, argval=8, argrepr='to 8', offset=28, starts_line=6, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=30, starts_line=7, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, argrepr='6', offset=32, starts_line=None, is_jump_target=False),
Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', argrepr='>', offset=34, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=42, argval=42, argrepr='', offset=36, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=21, argval=42, argrepr='to 42', offset=36, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=38, starts_line=8, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=52, argval=52, argrepr='', offset=40, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=8, argval=8, argrepr='', offset=42, starts_line=7, is_jump_target=True),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=26, argval=52, argrepr='to 52', offset=40, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=4, argval=8, argrepr='to 8', offset=42, starts_line=7, is_jump_target=True),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=44, starts_line=10, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=46, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=48, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=50, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=52, starts_line=11, is_jump_target=True),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=96, argval=96, argrepr='', offset=54, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=48, argval=96, argrepr='to 96', offset=54, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=56, starts_line=12, is_jump_target=True),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=58, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=60, starts_line=None, is_jump_target=False),
Expand All @@ -1044,30 +1044,30 @@ def jumpy():
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=72, starts_line=14, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=6, argrepr='6', offset=74, starts_line=None, is_jump_target=False),
Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', argrepr='>', offset=76, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=82, argval=82, argrepr='', offset=78, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=52, argval=52, argrepr='', offset=80, starts_line=15, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=41, argval=82, argrepr='to 82', offset=78, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=26, argval=52, argrepr='to 52', offset=80, starts_line=15, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=82, starts_line=16, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=84, starts_line=None, is_jump_target=False),
Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', argrepr='<', offset=86, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=92, argval=92, argrepr='', offset=88, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=104, argval=104, argrepr='', offset=90, starts_line=17, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=46, argval=92, argrepr='to 92', offset=88, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=52, argval=104, argrepr='to 104', offset=90, starts_line=17, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=92, starts_line=11, is_jump_target=True),
Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=56, argval=56, argrepr='', offset=94, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=28, argval=56, argrepr='to 56', offset=94, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=96, starts_line=19, is_jump_target=True),
Instruction(opname='LOAD_CONST', opcode=100, arg=6, argval='Who let lolcatz into this test suite?', argrepr="'Who let lolcatz into this test suite?'", offset=98, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=100, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=102, starts_line=None, is_jump_target=False),
Instruction(opname='SETUP_FINALLY', opcode=122, arg=96, argval=202, argrepr='to 202', offset=104, starts_line=20, is_jump_target=True),
Instruction(opname='SETUP_FINALLY', opcode=122, arg=12, argval=120, argrepr='to 120', offset=106, starts_line=None, is_jump_target=False),
Instruction(opname='SETUP_FINALLY', opcode=122, arg=48, argval=202, argrepr='to 202', offset=104, starts_line=20, is_jump_target=True),
Instruction(opname='SETUP_FINALLY', opcode=122, arg=6, argval=120, argrepr='to 120', offset=106, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=5, argval=1, argrepr='1', offset=108, starts_line=21, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=7, argval=0, argrepr='0', offset=110, starts_line=None, is_jump_target=False),
Instruction(opname='BINARY_TRUE_DIVIDE', opcode=27, arg=None, argval=None, argrepr='', offset=112, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=114, starts_line=None, is_jump_target=False),
Instruction(opname='POP_BLOCK', opcode=87, arg=None, argval=None, argrepr='', offset=116, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=24, argval=144, argrepr='to 144', offset=118, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=12, argval=144, argrepr='to 144', offset=118, starts_line=None, is_jump_target=False),
Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=120, starts_line=22, is_jump_target=True),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=2, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=122, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_IF_NOT_EXC_MATCH', opcode=121, arg=212, argval=212, argrepr='', offset=124, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_IF_NOT_EXC_MATCH', opcode=121, arg=106, argval=212, argrepr='to 212', offset=124, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=126, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=128, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=130, starts_line=None, is_jump_target=False),
Expand All @@ -1076,9 +1076,9 @@ def jumpy():
Instruction(opname='CALL_FUNCTION', opcode=131, arg=1, argval=1, argrepr='', offset=136, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=138, starts_line=None, is_jump_target=False),
Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=140, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=44, argval=188, argrepr='to 188', offset=142, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=22, argval=188, argrepr='to 188', offset=142, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=144, starts_line=25, is_jump_target=True),
Instruction(opname='SETUP_WITH', opcode=143, arg=24, argval=172, argrepr='to 172', offset=146, starts_line=None, is_jump_target=False),
Instruction(opname='SETUP_WITH', opcode=143, arg=12, argval=172, argrepr='to 172', offset=146, starts_line=None, is_jump_target=False),
Instruction(opname='STORE_FAST', opcode=125, arg=1, argval='dodgy', argrepr='dodgy', offset=148, starts_line=None, is_jump_target=False),
Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=150, starts_line=26, is_jump_target=False),
Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Never reach this', argrepr="'Never reach this'", offset=152, starts_line=None, is_jump_target=False),
Expand All @@ -1090,9 +1090,9 @@ def jumpy():
Instruction(opname='DUP_TOP', opcode=4, arg=None, argval=None, argrepr='', offset=164, starts_line=None, is_jump_target=False),
Instruction(opname='CALL_FUNCTION', opcode=131, arg=3, argval=3, argrepr='', offset=166, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=168, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=16, argval=188, argrepr='to 188', offset=170, starts_line=None, is_jump_target=False),
Instruction(opname='JUMP_FORWARD', opcode=110, arg=8, argval=188, argrepr='to 188', offset=170, starts_line=None, is_jump_target=False),
Instruction(opname='WITH_EXCEPT_START', opcode=49, arg=None, argval=None, argrepr='', offset=172, starts_line=None, is_jump_target=True),
Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=178, argval=178, argrepr='', offset=174, starts_line=None, is_jump_target=False),
Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=89, argval=178, argrepr='to 178', offset=174, starts_line=None, is_jump_target=False),
Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=176, starts_line=None, is_jump_target=False),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=178, starts_line=None, is_jump_target=True),
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=180, starts_line=None, is_jump_target=False),
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2021-03-29-16-20-29.bpo-27129.BF03A5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
The bytecode interpreter uses instruction, rather byte, offsets internally.
This reduces the number of EXTENDED_ARG instructions needed and streamlines
instruction dispatch a bit.
2 changes: 1 addition & 1 deletion Objects/codeobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1246,7 +1246,7 @@ PyTypeObject PyCode_Type = {
int
PyCode_Addr2Line(PyCodeObject *co, int addrq)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to treat the second argument as an instruction offset, rather than a byte offset. Rather than changing PyCode_Addr2Line() calls. It sounds unfortunate that the API allows to ask the address in the middle of a word (2 bytes).

It sounds common to pass f_lasti to PyCode_Addr2Line(): PyCode_Addr2Line(code, f_lasti) would still work, no?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depends on how you think the API is defined.
If you want the the line number from an offset derived from code.co_linetable, then the API should be unchanged.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyCode_Addr2Line(code, offset) must take a byte offset.
To getf_lasti from the frame, the only API to do so is something like PyObject_GetAttrString(frame, "f_lasti"), which would return a byte offset.

Also see:
https://www.python.org/dev/peps/pep-0626/#c-api
https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers

{
if (addrq == -1) {
if (addrq < 0) {
return co->co_firstlineno;
}
assert(addrq >= 0 && addrq < PyBytes_GET_SIZE(co->co_code));
Expand Down
Loading