Instruction-Offsets vs Byte-Offsets in dis.dis output NOT A BUG

[mrolle45]

Please ignore this issue. It was caused by my running the 3.7 library dis in the 3.10 environment.

Bug report

This is new to version 3.10
When disassembling code code block with jumps, bytecode for the jump shows the wrong target instruction number, and/or the bytecode with the wrong address flagged with >>.

To reproduce, here is an example, I have interspersed the source code with the disassembled output:

try: x = 1
  3           4 SETUP_FINALLY           15 (to 21)  --- (to 2 * 15 = 30)
              6 SETUP_FINALLY            4 (to 12)  --- (to 8 + 2 * 4 = 16)
              8 LOAD_CONST               1 (1)
             10 STORE_NAME               1 (x)
        >>   12 POP_BLOCK                           --- no >>
             14 JUMP_FORWARD             6 (to 22)  --- (to 16 + 2 * 6 = 28)

 except: x = 2
  4          16 POP_TOP                             --- >> (target of 6)
             18 POP_TOP
             20 POP_TOP
        >>   22 LOAD_CONST               2 (2)      --- no >>
             24 STORE_NAME               1 (x)
             26 POP_EXCEPT
        >>   28 POP_BLOCK                           --- >> (target of 14), not >> (target of 48).

finally: x = 3
  5          30 LOAD_CONST               3 (3)
             32 STORE_NAME               1 (x)
             34 JUMP_FORWARD             3 (to 39)  --- (to 36 + 2 * 3 = 42)
             36 LOAD_CONST               3 (3)      --- >> (target of 4)
             38 STORE_NAME               1 (x)
             40 CONTINUE_LOOP            0


if x == 3:
  7          42 LOAD_NAME                1 (x)      --- >> (target of 34)
             44 LOAD_CONST               3 (3)
             46 COMPARE_OP               2 (==)
             48 POP_JUMP_IF_FALSE       28          --- (to 28 * 2 = 56)

	x = 4
  8          50 LOAD_CONST               4 (4)
             52 STORE_NAME               1 (x)
             54 JUMP_FORWARD             2 (to 58)  --- (to 56 + 2 * 2 = 60)

else:
 	x = 5
10          56 LOAD_CONST               5 (5)
        >>   58 STORE_NAME               1 (x)      --- no >>

import sys
 14          60 LOAD_CONST               6 (0)      --- >> (target of 54)
             62 LOAD_CONST               7 (None)
             64 IMPORT_NAME              2 (sys)
             66 STORE_NAME               2 (sys)

Compare with output from 3.9. The compiled bytecodes are slightly different, but the jump addresses and labelled indictors are correct.
In my opinion, though, the true oparg values (17, etc.), which are shown in 3.10, should be shown instead of the doubled values (34, etc.), which are shown in 3.9. I would consider this a bug in 3.9.

  3           4 SETUP_FINALLY           34 (to 40)
              6 SETUP_FINALLY            8 (to 16)
              8 LOAD_CONST               1 (1)
             10 STORE_NAME               1 (x)
             12 POP_BLOCK
             14 JUMP_FORWARD            16 (to 32)

  4     >>   16 POP_TOP
             18 POP_TOP
             20 POP_TOP
             22 LOAD_CONST               2 (2)
             24 STORE_NAME               1 (x)
             26 POP_EXCEPT
             28 JUMP_FORWARD             2 (to 32)
             30 RERAISE
        >>   32 POP_BLOCK

  5          34 LOAD_CONST               3 (3)
             36 STORE_NAME               1 (x)
             38 JUMP_FORWARD             6 (to 46)
        >>   40 LOAD_CONST               3 (3)
             42 STORE_NAME               1 (x)
             44 RERAISE

  7     >>   46 LOAD_NAME                1 (x)
             48 LOAD_CONST               3 (3)
             50 COMPARE_OP               2 (==)
             52 POP_JUMP_IF_FALSE       60

  8          54 LOAD_CONST               4 (4)
             56 STORE_NAME               1 (x)
             58 JUMP_FORWARD             4 (to 64)

 10     >>   60 LOAD_CONST               5 (5)
             62 STORE_NAME               1 (x)

 14     >>   64 LOAD_CONST               6 (0)
             66 LOAD_CONST               7 (None)
             68 IMPORT_NAME              2 (sys)
             70 STORE_NAME               2 (sys)

Your environment

• CPython versions tested on: Python 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
• Operating system and architecture: Windows

[sweeneyde]

Python 3.9 is now only accepting security-related fixes.

[mrolle45]

Python 3.9 is now only accepting security-related fixes.

Fine.
I wonder if Python releases come with a list of known bugs, and whether they will be fixed in a future release.
Do older versions of Python accept fixes to documentation?

[sweeneyde]

Do older versions of Python accept fixes to documentation?

No, not typically for security-only branches.

This isn't a bug so much as a deliberate change in behavior. In Python 3.9, the opcode arguments of jumps were byte offsets. In Python 3.10+, opcode arguments of jumps are stored as instruction offsets. In both cases, the stored number is displayed as the opcode argument. Besides, as noted in the dis module docs:

Bytecode is an implementation detail of the CPython interpreter. No guarantees are made that bytecode will not be added, removed, or changed between versions of Python. Use of this module should not be considered to work across Python VMs or Python releases.

In the changelog for 3.10, it was noted that

bpo-27129: The bytecode interpreter uses instruction, rather byte, offsets internally. This reduces the number of EXTENDED_ARG instructions needed and streamlines instruction dispatch a bit.

[sweeneyde]

I believe it has been suggested that the even-numbered byte-offset labels down the left-hand side could be replaced by instruction-offsets (half their current value) (and (to xx) labels likewise halved), which might help clear things up.

[sweeneyde]

There was an exchange about this on the original PR for the change.

cc @markshannon @gvanrossum

[gvanrossum]

Um, looking at the 3.10 example it does look wrong — how can the “to” address be odd?

[brandtbucher]

I'm not able to reproduce the 3.10.0 output. It appears that some of its dis output (like the odd "to" addresses) may have been modified in the original issue report when the annotations were added? It appears that the annotations don't account for the change in oparg values to use instruction offsets instead of bytes in 3.10.

If not, can you give us the source and (unannotated) dis output that you believe is incorrect?

[mrolle45]

Sorry, I didn't think to also provide the python file and dis output separately. Now that you asked for it, I realize that it would help you folks in debugging it.

example.py.txt
dis.txt

Interesting discussion over whether to display instruction offsets or byte offsets. Personally I don't care, as long as you are consistent and accurate. My suggestion of showing an instruction offset in the oparg and then showing the byte offset in the (to ...) was, I now see, a bad idea. You can show (1) the bytecode address, (2) the oparg, and (3) the target, as either all instruction offsets or byte offsets.

I wonder why 3.10 doesn't just do what 3.9 does. Does it have something to do with changing the oparg to byte offsets? That sounds like a good idea for reducing the occurrences of opargs >= 256. Just be sure you cover all the cases correctly.

looking at the 3.10 example it does look wrong — how can the “to” address be odd?

The answer is that dis() was adding an offset to a byte address.

[sweeneyde]

What is your procedure for generating dis.txt? Is it possible that you are using an earlier version of Python to disassemble bytecode from Python 3.10?

Python 3.8+ does not have a CONTINUE_LOOP opcode (119 is now RERAISE), so it should not appear in any output from 3.10.

[mrolle45]

Nice detective work, @sweeneyde. You've saved your team a good bit of work. Thank you!

Python 3.8+ does not have a CONTINUE_LOOP opcode (119 is now RERAISE), so it should not appear in any output from 3.10.

Whoops. In my python 3.10, I seem to be pointing to the lib in 3.7.
Here's how I generated the dis file:

D:\Solutions\trunk\Python\Example>py10
Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis(open('example.py.txt').read(), file=open('dis.txt', 'w'))
>>> dis
<module 'dis' from 'D:\\Python37\\lib\\dis.py'>

Running with my usual Python (3.7):

D:\Solutions\trunk\Python\Example>python
Python 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis(open('example.py.txt').read(), file=open('dis.txt', 'w'))
>>> dis
<module 'dis' from 'D:\\Python37\\lib\\dis.py'>

This time, it produced a correct (for 3.7) output with the right opargs and targets and addresses.
Apparently, the dis module compiles the source file then analyzes the code object, and the compile() function is a 3.10 builtin.
I'll get my 3.10 environment straightened out, and maybe the output will be correct.

[mrolle45]

Hooray! It's not a bug, after all. It was my error :(.

I had copied py10.exe from my 3.10 root to my 3.7 root, which was on my %path%. That's why I got the 3.10 python interpreter but the 3.7 library.
I removed this interloper, and added the 3.10 root to %path%, after the 3.7 root. I normally use 3.7 for everything.

Now my session is:

Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis(open('example.py').read(), file=open('dis.dis10.txt', 'w'))
>>> dis
<module 'dis' from 'D:\\Python310\\lib\\dis.py'>
>>> dis.dis(open('example.py.txt').read(), file=open('dis.txt', 'w'))

The output is