Regression in 3.12 beta in json.dump deeply nested dict

[hauntsaninja]

This was reported here: https://discuss.python.org/t/has-sys-setrecursionlimit-behaviour-changed-in-python-3-12b/30205

The following program works fine on 3.11, but crashes with RecursionError on 3.12:

d = {}
for x in range(1_000):
    d = {'k': d}

import json, sys
sys.setrecursionlimit(100_000_000)
foo = json.dumps(d)

I confirmed this bisects to #96510

Linked PRs

• GH-107263: Increase C stack limit for most functions, except _PyEval_EvalFrameDefault() #107535
• [3.12] GH-107263: Increase C stack limit for most functions, except _PyEval_EvalFrameDefault() (GH-107535) #107618

[hauntsaninja]

cc @markshannon

[sunmy2019]

Related #105003

[hauntsaninja]

Also cc @Yhg1s out of caution, since we're close to RC.

At a minimum, I'm guessing we need a What's New

[hauntsaninja]

The relevant number is the C stack, so depending on your numbers, note you don't need to set sys.recursionlimit to see a regression. The following works in 3.11, but fails in 3.12:

d = {}
for x in range(810):
    d = {'k': d}

import json
foo = json.dumps(d)

[sunmy2019]

At a minimum, I'm guessing we need a What's New

I am in favor of adding a new API exposing this. Nothing else would fundamentally fix this.

[markshannon]

If we change the original example to:

d = {}
for x in range(100_000):
    d = {'k': d}

import json, sys
sys.setrecursionlimit(100_000_000)
foo = json.dumps(d)

It segfaults on 3.11.
I would argue that not segfaulting is not a regression, but an improvement.

Here are what we can do to lessen the impact of this change:

• Leaves things as they are.
• Tweak the C recursion limit, so that PyEval_EvalDefault() consumes more units than other functions so other C code can recurse at least 1000 deep. I'd up the limit to ~2200 making PyEval_EvalDefault() consume 3 units, and everything else 2.
• Allow the C recursion limit to be changed, which I don't think is a good idea for safety reasons.

Whatever we do, I'll add a section to the "what's new".

[markshannon]

Something else we could do is to change the C recursion limit from 800 to 1500 on optimized builds, where the C frames will be smaller.

A few tests will need changing to handle the differing recursion limits, but that's not a bad thing as it should make the tests more general.

[sunmy2019]

Allow the C recursion limit to be changed, which I don't think is a good idea for safety reasons.

Can you elaborate on that?

It segfaults on 3.11.

People can easily increase their stack size on Unix-like systems with ulimit

As OS provides this API, why would Python forbid users with real need?

Other virtual machines like V8 (JavaScript runtime) can increase stack size on Unix-like systems (while not on Windows).

Increasing stack size/recursion limit is a real need for some users, and this limit will be a problem without easy workaround. It breaks user's code without possibility of easy migration.