Skip to content

bpo-46845: Reduce dict size when all keys are Unicode. #31564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 1, 2022

Conversation

methane
Copy link
Member

@methane methane commented Feb 25, 2022

@methane
Copy link
Member Author

methane commented Feb 25, 2022

pyperformance (with --enable-optimizations --with-lto):

Result
Benchmark main-opt compact-opt
2to3 294 ms 292 ms: 1.01x faster
chameleon 7.78 ms 7.66 ms: 1.02x faster
chaos 81.0 ms 82.3 ms: 1.02x slower
crypto_pyaes 93.1 ms 94.3 ms: 1.01x slower
deltablue 4.59 ms 4.73 ms: 1.03x slower
dulwich_log 86.6 ms 85.3 ms: 1.02x faster
fannkuch 433 ms 426 ms: 1.02x faster
float 88.0 ms 86.5 ms: 1.02x faster
go 160 ms 158 ms: 1.01x faster
hexiom 7.40 ms 7.45 ms: 1.01x slower
json_dumps 13.8 ms 13.6 ms: 1.01x faster
json_loads 27.3 us 26.9 us: 1.02x faster
logging_format 7.38 us 7.33 us: 1.01x faster
logging_silent 118 ns 119 ns: 1.00x slower
logging_simple 6.72 us 6.66 us: 1.01x faster
mako 11.8 ms 11.5 ms: 1.02x faster
meteor_contest 116 ms 115 ms: 1.01x faster
nbody 101 ms 103 ms: 1.02x slower
pickle 13.1 us 13.0 us: 1.00x faster
pickle_dict 36.4 us 37.9 us: 1.04x slower
pickle_list 5.28 us 5.22 us: 1.01x faster
pidigits 216 ms 217 ms: 1.01x slower
pyflate 488 ms 490 ms: 1.00x slower
python_startup 28.5 ms 28.1 ms: 1.01x faster
python_startup_no_site 6.89 ms 6.80 ms: 1.01x faster
raytrace 347 ms 355 ms: 1.02x slower
regex_compile 157 ms 157 ms: 1.00x faster
regex_dna 233 ms 234 ms: 1.01x slower
regex_effbot 3.51 ms 3.46 ms: 1.01x faster
regex_v8 26.0 ms 25.5 ms: 1.02x faster
richards 54.8 ms 54.2 ms: 1.01x faster
scimark_fft 365 ms 368 ms: 1.01x slower
scimark_lu 130 ms 126 ms: 1.03x faster
scimark_monte_carlo 79.2 ms 77.7 ms: 1.02x faster
scimark_sor 137 ms 136 ms: 1.01x faster
scimark_sparse_mat_mult 5.44 ms 5.37 ms: 1.01x faster
spectral_norm 115 ms 116 ms: 1.01x slower
sqlalchemy_declarative 159 ms 158 ms: 1.01x faster
sqlalchemy_imperative 24.4 ms 24.2 ms: 1.01x faster
sqlite_synth 2.81 us 2.75 us: 1.02x faster
sympy_expand 566 ms 563 ms: 1.01x faster
sympy_integrate 23.9 ms 24.0 ms: 1.01x slower
sympy_sum 197 ms 196 ms: 1.00x faster
sympy_str 347 ms 344 ms: 1.01x faster
telco 7.50 ms 7.36 ms: 1.02x faster
unpack_sequence 52.3 ns 51.9 ns: 1.01x faster
unpickle 16.0 us 15.1 us: 1.06x faster
unpickle_list 5.42 us 5.28 us: 1.03x faster
unpickle_pure_python 274 us 271 us: 1.01x faster
xml_etree_generate 86.9 ms 86.3 ms: 1.01x faster
xml_etree_process 61.8 ms 61.6 ms: 1.00x faster
Geometric mean (ref) 1.01x faster

Benchmark hidden because not significant (8): django_template, html5lib, nqueens, pathlib, pickle_pure_python, tornado_http, xml_etree_parse, xml_etree_iterparse

@methane
Copy link
Member Author

methane commented Feb 25, 2022

sys.getsizeof(dict.fromkeys("abcdefg")) becomes 272 bytes from 352 bytes on 64bit platform.

maxrss(KB) of /usr/bin/time python -c 'import flask,sqlalchemy,fastapi'

  main compact -  ratio
default 46716 45300 1416 96.97%
-OO 43476 40844 2632 93.95%
-OO + nodebugrange 41424 39632 1792 95.67%

@methane
Copy link
Member Author

methane commented Feb 25, 2022

This PR includes fixing dict_merge, dict_keys, dict_values, and dict_items don't respect order of split table.
They just use _PyDict_Next() for now.

@methane methane merged commit 9833bb9 into python:main Mar 1, 2022
@methane methane deleted the dict-stringkeys branch March 1, 2022 23:09
@markshannon
Copy link
Member

Thanks for fixing the regressions in split table ordering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants