-
Notifications
You must be signed in to change notification settings - Fork 437
Profiling pprof converter is raising exception when sorting _locations #2962
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
jd
added a commit
to jd/dd-trace-py
that referenced
this issue
Nov 10, 2021
There was an ongoing issue wih the pprof encode mixing type between strings, ints and None when dealing with labels. This has been fixed in multiple places and in multiple occasion, but there were still holes. When encoding Events to pprof, they are grouped by event fields. Those fields might be str, int or something else. In order to group those events, all the fields used for grouping must be: 1. hashable (because we index in a dict) 2. sortable (because we sort event to have reproducible) 3. string (because we store those as labels in pprof) The problem is that most value type used in events are hashable (None, int or str) but sortability often fails when one of the field is None: you can't compare `int` with `None`, making sorted() call fails like in DataDog#2962. As we need string in the end, the fix is to convert everything to strings as soon as possible when grouping events: that makes sure conditions 1, 2 and 3 are validated ASAP and everything works. To do this, the code has been updated and typed to catch any future error. Fixes DataDog#2962
mergify bot
added a commit
that referenced
this issue
Nov 15, 2021
There was an ongoing issue wih the pprof encode mixing type between strings, ints and None when dealing with labels. This has been fixed in multiple places and in multiple occasion, but there were still holes. When encoding Events to pprof, they are grouped by event fields. Those fields might be str, int or something else. In order to group those events, all the fields used for grouping must be: 1. hashable (because we index in a dict) 2. sortable (because we sort event to have reproducible) 3. string (because we store those as labels in pprof) The problem is that most value type used in events are hashable (None, int or str) but sortability often fails when one of the field is None: you can't compare `int` with `None`, making sorted() call fails like in #2962. As we need string in the end, the fix is to convert everything to strings as soon as possible when grouping events: that makes sure conditions 1, 2 and 3 are validated ASAP and everything works. To do this, the code has been updated and typed to catch any future error. Fixes #2962 Co-authored-by: Brett Langdon <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
mergify bot
added a commit
that referenced
this issue
Dec 10, 2021
There was an ongoing issue wih the pprof encode mixing type between strings, ints and None when dealing with labels. This has been fixed in multiple places and in multiple occasion, but there were still holes. When encoding Events to pprof, they are grouped by event fields. Those fields might be str, int or something else. In order to group those events, all the fields used for grouping must be: 1. hashable (because we index in a dict) 2. sortable (because we sort event to have reproducible) 3. string (because we store those as labels in pprof) The problem is that most value type used in events are hashable (None, int or str) but sortability often fails when one of the field is None: you can't compare `int` with `None`, making sorted() call fails like in #2962. As we need string in the end, the fix is to convert everything to strings as soon as possible when grouping events: that makes sure conditions 1, 2 and 3 are validated ASAP and everything works. To do this, the code has been updated and typed to catch any future error. Fixes #2962 Co-authored-by: Brett Langdon <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 6520212)
mergify bot
added a commit
that referenced
this issue
Dec 10, 2021
…) (#3062) There was an ongoing issue wih the pprof encode mixing type between strings, ints and None when dealing with labels. This has been fixed in multiple places and in multiple occasion, but there were still holes. When encoding Events to pprof, they are grouped by event fields. Those fields might be str, int or something else. In order to group those events, all the fields used for grouping must be: 1. hashable (because we index in a dict) 2. sortable (because we sort event to have reproducible) 3. string (because we store those as labels in pprof) The problem is that most value type used in events are hashable (None, int or str) but sortability often fails when one of the field is None: you can't compare `int` with `None`, making sorted() call fails like in #2962. As we need string in the end, the fix is to convert everything to strings as soon as possible when grouping events: that makes sure conditions 1, 2 and 3 are validated ASAP and everything works. To do this, the code has been updated and typed to catch any future error. Fixes #2962 Co-authored-by: Brett Langdon <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 6520212) Co-authored-by: Julien Danjou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Which version of dd-trace-py are you using?
0.54.0rc1
Which version of pip are you using?
21.3.1
Which version of the libraries are you using?
aiofiles==0.7.0
appdirs==1.4.4
appnope==0.1.2
asgiref==3.4.1
astroid==2.5
async-lru==1.0.2
attrs==21.2.0
Babel==2.8.0
backcall==0.2.0
black==19.10b0
boto3==1.18.51
botocore==1.21.51
cachetools==4.2.4
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.6
click==8.0.1
colorama==0.4.4
commonmark==0.9.1
coverage==5.5
coveralls==3.2.0
datadog==0.42.0
ddtrace==0.54.0rc1
decorator==5.1.0
dnslib==0.9.16
dnspython==1.16.0
docker==5.0.0
docopt==0.6.2
dulwich==0.20.25
ecdsa==0.18.0b1
expiringdict==1.1.4
fastapi==0.68.1
Flask==2.0.1
greenlet==1.1.1
guppy3==3.1.1
h11==0.12.0
idna==3.2
importlib-metadata==4.8.1
inflect==5.3.0
iniconfig==1.1.1
ipython==7.28.0
isort==5.5.4
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
jmespath==0.10.0
launchdarkly-server-sdk==7.2.0
lazy-object-proxy==1.6.0
localstack==0.12.17.5
localstack-client==1.24
localstack-ext==0.12.17
lxml==4.6.3
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mccabe==0.6.1
microservice-python==4.1.7
packaging==21.0
parso==0.8.2
pathspec==0.9.0
pbr==5.6.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==1.0.0
prompt-toolkit==3.0.20
protobuf==3.18.0
ptyprocess==0.7.0
py==1.10.0
pyaes==1.6.1
pyasn1==0.4.8
pydantic==1.8.2
Pygments==2.10.0
pylint==2.6.0
pylint-ext==0.4.0
Pympler==0.9
pyparsing==3.0.0rc1
pyRFC3339==1.1
pytest==6.2.5
pytest-asyncio==0.15.1
pytest-black==0.3.12
pytest-cov==2.10.1
pytest-env==0.6.2
pytest-isort==2.0.0
pytest-timeout==1.4.2
pytest-watch==4.2.0
python-dateutil==2.8.2
python-jose==3.3.0
python-multipart==0.0.5
pytz==2021.1
PyYAML==5.4.1
raven==6.10.0
regex==2021.8.28
requests==2.26.0
requests-mock==1.8.0
rich==10.10.0
rsa==4.7.2
s3transfer==0.5.0
semver==2.13.0
sentry-sdk==1.4.1
six==1.16.0
SQLAlchemy==1.4.23
starlette==0.14.2
stevedore==3.4.0
tabulate==0.8.9
tenacity==8.0.1
toml==0.10.2
traitlets==5.1.0
typed-ast==1.4.3
typing-extensions==3.10.0.2
urllib3==1.26.7
uvicorn==0.15.0
uvloop==0.16.0
watchdog==2.1.5
wcwidth==0.2.5
websocket-client==1.2.1
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.6.0
How can we reproduce your problem?
I am running my application with
dd-trace-run
andDD_PROFILING_ENABLED=true
.I started seeing the error after adding a threading.Timer() that calls that calls the same function some time ago, like described below.
I have a class that has a load() function which populates a cache. In the end of the load() function I call another function that starts a threding.Timer(), like:
My timer runs every 15 minutes and the profiling scheduler runs every 1 minute. The exception is raised when both match running time. However it does not happen every time that there is a match.
What is the result that you get?
I am seeing following exception in my log:
What is the result that you expected?
No exception.
The text was updated successfully, but these errors were encountered: