gh-111744: Support opcode events in bdb

[gaogaotiantian]

This is the first step to solve #111744 . After some discussion in discord, most of the developers like the idea to have breakpoint() break immediately, rather than on next event. In order to achieve that, we need to use the opcode events.

This PR is just to lay the ground for opcode events. In general, the users can use opcode events now with a new set of API - set_stepinstr() and user_opcode(). The documentation is not updated on purpose - if we prefer to keep it internal now and just use it as an infra to support new breakpoint(), we can keep quiet about it. If we decide to make it available to public, I can write the docs.

• Issue: breakpoint does not enter pdb until the "next event" happens #111744

[gaogaotiantian]

Hi @brandtbucher , I know you've been busy recently. When you have some time, could you take a look at this? This is the prerequisite for a breakpoint that breaks immediately. That change might need more reviews as it's a breaking behavior so I kind of hope I can lay the foundation soon.

[brandtbucher]

Thanks a ton for your patience (I really dropped the ball on reviewing this)! I'd love to get your thoughts on a couple of things before merging:

[brandtbucher]

And just to jog my memory: self.__curframe is needed because we only want to change events at or above the frame where set_trace was called (these are the same frames where f_trace_lines was set), but we may be much deeper at this point. Right?

But why is it double-underscored again? Do we need the name-mangling? I see pdb.Pdb has a curframe attribute, so maybe it's to avoid confusion with that?

[gaogaotiantian]

Yes the reason for the double-underscore is to avoid the conflict with pdb.Pdb.curframe. And yes, we only want to set trace_opcodes in the "user frames", which means frames above (including) the caller frame into the pdb. So we need to record the entering frame.

The curframe attribute of pdb means the current frame being debugged. Do you prefer to change the the name __curframe to something like entering_frame?

[brandtbucher]

Yeah, that feels a lot less cryptic and scary. :) Especially considering we already have stopframe, returnframe, and botframe.

[brandtbucher]

Also, I played with this a bit while reviewing, and it's pretty neat. I know time's running out: do you think the rest of the breakpoint() change is feasible for 3.13? I'm available for review (I promise).

[brandtbucher]

Also, what's the relationship between this PR and GH-103050?

[gaogaotiantian]

#103050 was the bigger ambition to directly support instruction level debugging back then. It was not easy to push something that big at that time when I did not have some basic reputation of "this person knows what he's doing". That one was kind of abandoned and the most critical part (supporting opcode event in general) was adopted into this PR.

I will try to pursue it again in 3.14, alongwith some other more ambitious targets, but that's after the language summit :)

And to your question, I will finish the breakpoint change by the end of tomorrow and see if we can make it in 3.13.

[brandtbucher]

And to your question, I will finish the breakpoint change by the end of tomorrow and see if we can make it in 3.13.

Let me know if there's anything I can do to help, besides being available for review. I think it's a great change.

[gaogaotiantian]

I had the prototype working once it should not be too difficult. Although this might break some tests - now that it stopped after the instruction after breakpoint() - it might be the same line. I'll see if it causes issues.

[brandtbucher]

I'm ready to land this once __curframe is renamed. :)

[gaogaotiantian]

I used enterframe to match the style to others.

[brandtbucher]

Sorry, spotted one more thing:

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot iOS ARM64 Simulator 3.x has failed when building commit f34e965.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1380/builds/199) and take a look at the build logs.
4. Check if the failure is related to this commit (f34e965) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1380/builds/199

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 22, done.        
remote: Counting objects:   4% (1/22)        
remote: Counting objects:   9% (2/22)        
remote: Counting objects:  13% (3/22)        
remote: Counting objects:  18% (4/22)        
remote: Counting objects:  22% (5/22)        
remote: Counting objects:  27% (6/22)        
remote: Counting objects:  31% (7/22)        
remote: Counting objects:  36% (8/22)        
remote: Counting objects:  40% (9/22)        
remote: Counting objects:  45% (10/22)        
remote: Counting objects:  50% (11/22)        
remote: Counting objects:  54% (12/22)        
remote: Counting objects:  59% (13/22)        
remote: Counting objects:  63% (14/22)        
remote: Counting objects:  68% (15/22)        
remote: Counting objects:  72% (16/22)        
remote: Counting objects:  77% (17/22)        
remote: Counting objects:  81% (18/22)        
remote: Counting objects:  86% (19/22)        
remote: Counting objects:  90% (20/22)        
remote: Counting objects:  95% (21/22)        
remote: Counting objects: 100% (22/22)        
remote: Counting objects: 100% (22/22), done.        
remote: Compressing objects:  11% (1/9)        
remote: Compressing objects:  22% (2/9)        
remote: Compressing objects:  33% (3/9)        
remote: Compressing objects:  44% (4/9)        
remote: Compressing objects:  55% (5/9)        
remote: Compressing objects:  66% (6/9)        
remote: Compressing objects:  77% (7/9)        
remote: Compressing objects:  88% (8/9)        
remote: Compressing objects: 100% (9/9)        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 12 (delta 10), reused 3 (delta 2), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'f34e965e52b9bdf157b829371870edfde45b80bf'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f34e965e52 GH-111744: Support opcode events in bdb (GH-111834)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

../../Objects/frameobject.c:240:57: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_keys(PyObject *self, PyObject *__unused)
                                                        ^
../../Objects/frameobject.c:391:59: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_values(PyObject *self, PyObject *__unused)
                                                          ^
../../Objects/frameobject.c:419:58: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_items(PyObject *self, PyObject *__unused)
                                                         ^
../../Objects/frameobject.c:568:61: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_reversed(PyObject *self, PyObject *__unused)
                                                            ^
4 warnings generated.
../../Python/import.c:1057:13: warning: variable 'key' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
        if (_extensions_cache_init() < 0) {
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Python/import.c:1095:9: note: uninitialized use occurs here
    if (key != NULL) {
        ^~~
../../Python/import.c:1057:9: note: remove the 'if' if its condition is always false
        if (_extensions_cache_init() < 0) {
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Python/import.c:1063:5: note: variable 'key' is declared here
    void *key = NULL;
    ^
../../Python/import.c:1195:1: warning: unused function 'is_core_module' [-Wunused-function]
is_core_module(PyInterpreterState *interp, PyObject *name, PyObject *path)
^
2 warnings generated.
../../Python/pystate.c:1125:1: warning: unused function 'check_interpreter_whence' [-Wunused-function]
check_interpreter_whence(long whence)
^
1 warning generated.

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

../../Objects/frameobject.c:240:57: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_keys(PyObject *self, PyObject *__unused)
                                                        ^
../../Objects/frameobject.c:391:59: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_values(PyObject *self, PyObject *__unused)
                                                          ^
../../Objects/frameobject.c:419:58: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_items(PyObject *self, PyObject *__unused)
                                                         ^
../../Objects/frameobject.c:568:61: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
framelocalsproxy_reversed(PyObject *self, PyObject *__unused)
                                                            ^
4 warnings generated.
../../Python/import.c:1057:13: warning: variable 'key' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
        if (_extensions_cache_init() < 0) {
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Python/import.c:1095:9: note: uninitialized use occurs here
    if (key != NULL) {
        ^~~
../../Python/import.c:1057:9: note: remove the 'if' if its condition is always false
        if (_extensions_cache_init() < 0) {
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Python/import.c:1063:5: note: variable 'key' is declared here
    void *key = NULL;
    ^
1 warning generated.
../../Modules/_sqlite/connection.c:1307:14: warning: 'sqlite3_create_window_function' is only available on iOS 13.0 or newer [-Wunguarded-availability-new]
        rc = sqlite3_create_window_function(self->db, name, num_params, flags,
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator17.4.sdk/usr/include/sqlite3.h:5533:16: note: 'sqlite3_create_window_function' has been marked as being introduced in iOS 13.0 here, but the deployment target is iOS 12.0.0
SQLITE_API int sqlite3_create_window_function(
               ^
../../Modules/_sqlite/connection.c:1307:14: note: enclose 'sqlite3_create_window_function' in a __builtin_available check to silence this warning
        rc = sqlite3_create_window_function(self->db, name, num_params, flags,
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../Modules/_sqlite/connection.c:1315:14: warning: 'sqlite3_create_window_function' is only available on iOS 13.0 or newer [-Wunguarded-availability-new]
        rc = sqlite3_create_window_function(self->db, name, num_params, flags,
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator17.4.sdk/usr/include/sqlite3.h:5533:16: note: 'sqlite3_create_window_function' has been marked as being introduced in iOS 13.0 here, but the deployment target is iOS 12.0.0
SQLITE_API int sqlite3_create_window_function(
               ^
../../Modules/_sqlite/connection.c:1315:14: note: enclose 'sqlite3_create_window_function' in a __builtin_available check to silence this warning
        rc = sqlite3_create_window_function(self->db, name, num_params, flags,
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.

xcodebuild: error: 'iOSTestbed.arm64-iphonesimulator.1714834279/iOSTestbed.xcodeproj' does not exist.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys, json; result = json.load(sys.stdin); print(result['subsections']['_values'][1]['subsections']['_values'][0]['emittedOutput']['_value'])
                                                           ~~~~~~^^^^^^^^^^^^^^^
KeyError: 'subsections'
make: *** [testios] Error 1