Segmentation Fault when multi-threading with different RPCs. #2599
Description
I can see multiple similar issues (#1847 , #2409) but I guess they're not merged in yet. The issue persists even with the non-async Web3 module with no middlewares. Here's my reproducible example where this issue happens every single time.
- Version: 5.30.0
- Python: 3.9.12
- OS: macOS Monterey 12.2.1 (21D62)
pip freezeoutput
aiohttp==3.8.1
aiosignal==1.2.0
aniso8601==9.0.1
async-timeout==4.0.2
attrs==22.1.0
base58==2.1.1
bitarray==1.2.2
cachetools==5.2.0
certifi @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_884c889c-96af-444f-bd6d-daddb5e9a462ykj3l5n_/croots/recipe/certifi_1655968814730/work/certifi
charset-normalizer==2.1.0
click==8.1.3
cytoolz==0.12.0
db-dtypes==1.0.3
eth-abi==2.2.0
eth-account==0.5.9
eth-hash==0.3.3
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-rlp==0.2.1
eth-typing==2.3.0
eth-utils==1.10.0
Flask==2.1.3
flask-restx==0.5.1
frozenlist==1.3.1
google-api-core==2.8.2
google-auth==2.10.0
google-auth-oauthlib==0.5.2
google-cloud-bigquery==3.3.0
google-cloud-bigquery-storage==2.14.1
google-cloud-core==2.3.2
google-crc32c==1.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
grpcio==1.47.0
grpcio-status==1.47.0
hexbytes==0.2.2
idna==3.3
importlib-metadata @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_croot-5pqd2z6f/importlib-metadata_1648710902288/work
ipfshttpclient==0.8.0a2
itsdangerous==2.1.2
Jinja2==3.1.2
jsonschema==4.9.1
keyring @ file:///Users/builder/miniconda3/envs/prefect/conda-bld/keyring_1638777422651/work
lru-dict==1.1.8
MarkupSafe==2.1.1
multiaddr==0.0.9
multidict==6.0.2
netaddr==0.8.0
numpy==1.23.1
oauthlib==3.2.0
packaging==21.3
pandas==1.4.3
pandas-gbq==0.17.7
parsimonious==0.8.1
proto-plus==1.20.6
protobuf==3.20.1
pyarrow==8.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycryptodome==3.15.0
pydata-google-auth==1.4.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2022.1
requests==2.28.1
requests-oauthlib==1.3.1
rlp==2.0.1
rsa==4.9
six==1.16.0
toolz==0.12.0
urllib3==1.26.11
varint==1.0.2
web3==5.30.0
websockets==9.1
Werkzeug==2.1.2
yarl==1.8.1
zipp @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_66c2c5f2-5dd5-4946-a16a-72af650ebd6cnmz4ou0f/croots/recipe/zipp_1652343960956/work
What was wrong?
Segfault when multi-threading with multiple RPCs:
- The code which produced the error
from concurrent.futures import ThreadPoolExecutor
from web3 import Web3
def get_block_number(rpc: str):
w3 = Web3(Web3.HTTPProvider(rpc))
block_number = w3.eth.get_block_number()
return block_number
def get_bn_chains():
with ThreadPoolExecutor(max_workers=len(rpcs)) as exc:
tasks = [exc.submit(get_block_number, rpc) for rpc in rpcs]
results = [r.result() for r in tasks]
print(f"results: {results}")
return results
if __name__ == '__main__':
rpcs = [
"https://api.mycryptoapi.com/eth",
"https://api.avax.network/ext/bc/C/rpc",
"https://rpc.ftm.tools",
"https://polygon-rpc.com",
"https://arb1.arbitrum.io/rpc",
"https://mainnet.optimism.io",
"https://bsc-dataseed1.binance.org",
"https://mainnet.aurora.dev",
"https://eth.bd.evmos.org:8545",
"https://http-mainnet.hecochain.com",
"https://evm.cronos.org",
"https://mainnet.boba.network",
"https://rpc.gnosischain.com",
"https://rpc.gnosischain.com",
"https://evm.kava.io",
"https://rpc.callisto.network"
]
print("first round")
get_bn_chains()
print("second round")
get_bn_chains()
- The full output of the error
/Users/arshankhanifar/miniconda3/envs/bt_defi_tx/bin/python /Users/arshankhanifar/tpc_aux_services/etl_services/bt_defi_tx/samples/find_segfault.py
first round
results: [15303145, 18385655, 44492119, 31667322, 19742334, 18360836, 20268554, 71583519, 2655226, 17696348, 4068760, 764415, 23595692, 23595687, 1025761, 10469308]
second round
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
- What type of node you were connecting to.
RPC http node, I found all of them on chainlist.
How can it be fixed?
Monkey-patching as shown here fixes it.
Here's my new code:
import threading
from concurrent.futures import ThreadPoolExecutor
from typing import Callable
import web3._utils.request
from web3 import Web3
lock = threading.Lock()
def locked(fn: Callable) -> Callable:
def inner(*args, **kwargs):
with lock:
return fn(*args, **kwargs)
return inner
web3._utils.request.cache_session = locked(web3._utils.request.cache_session)
web3._utils.request._get_session = locked(web3._utils.request._get_session)
def get_block_number(rpc: str):
w3 = Web3(Web3.HTTPProvider(rpc))
block_number = w3.eth.get_block_number()
return block_number
def get_bn_chains():
with ThreadPoolExecutor(max_workers=len(rpcs)) as exc:
tasks = [exc.submit(get_block_number, rpc) for rpc in rpcs]
results = [r.result() for r in tasks]
print(f"results: {results}")
return results
if __name__ == '__main__':
rpcs = [
"https://api.mycryptoapi.com/eth",
"https://api.avax.network/ext/bc/C/rpc",
"https://rpc.ftm.tools",
"https://polygon-rpc.com",
"https://arb1.arbitrum.io/rpc",
"https://mainnet.optimism.io",
"https://bsc-dataseed1.binance.org",
"https://mainnet.aurora.dev",
"https://eth.bd.evmos.org:8545",
"https://http-mainnet.hecochain.com",
"https://evm.cronos.org",
"https://mainnet.boba.network",
"https://rpc.gnosischain.com",
"https://rpc.gnosischain.com",
"https://evm.kava.io",
"https://rpc.callisto.network"
]
print("first round")
get_bn_chains()
print("second round")
get_bn_chains()
And here's the output for it:
/Users/arshankhanifar/miniconda3/envs/bt_defi_tx/bin/python /Users/arshankhanifar/tpc_aux_services/etl_services/bt_defi_tx/samples/find_segfault.py
first round
results: [15303155, 18385721, 44492225, 31667386, 19742473, 18360979, 20268599, 71583620, 2655299, 17696393, 4068829, 764419, 23595719, 23595719, 1025782, 10469321]
second round
results: [15303155, 18385721, 44492225, 31667386, 19742473, 18360980, 20268599, 71583621, 2655300, 17696394, 4068829, 764419, 23595719, 23595719, 1025783, 10469321]
Process finished with exit code 0