Skip to content

Profiling pprof converter is raising exception when sorting _locations #2962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Truja opened this issue Oct 31, 2021 · 0 comments · Fixed by #2987
Closed

Profiling pprof converter is raising exception when sorting _locations #2962

Truja opened this issue Oct 31, 2021 · 0 comments · Fixed by #2987
Assignees

Comments

@Truja
Copy link

Truja commented Oct 31, 2021

Which version of dd-trace-py are you using?

0.54.0rc1

Which version of pip are you using?

21.3.1

Which version of the libraries are you using?

aiofiles==0.7.0
appdirs==1.4.4
appnope==0.1.2
asgiref==3.4.1
astroid==2.5
async-lru==1.0.2
attrs==21.2.0
Babel==2.8.0
backcall==0.2.0
black==19.10b0
boto3==1.18.51
botocore==1.21.51
cachetools==4.2.4
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.6
click==8.0.1
colorama==0.4.4
commonmark==0.9.1
coverage==5.5
coveralls==3.2.0
datadog==0.42.0
ddtrace==0.54.0rc1
decorator==5.1.0
dnslib==0.9.16
dnspython==1.16.0
docker==5.0.0
docopt==0.6.2
dulwich==0.20.25
ecdsa==0.18.0b1
expiringdict==1.1.4
fastapi==0.68.1
Flask==2.0.1
greenlet==1.1.1
guppy3==3.1.1
h11==0.12.0
idna==3.2
importlib-metadata==4.8.1
inflect==5.3.0
iniconfig==1.1.1
ipython==7.28.0
isort==5.5.4
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
jmespath==0.10.0
launchdarkly-server-sdk==7.2.0
lazy-object-proxy==1.6.0
localstack==0.12.17.5
localstack-client==1.24
localstack-ext==0.12.17
lxml==4.6.3
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mccabe==0.6.1
microservice-python==4.1.7
packaging==21.0
parso==0.8.2
pathspec==0.9.0
pbr==5.6.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==1.0.0
prompt-toolkit==3.0.20
protobuf==3.18.0
ptyprocess==0.7.0
py==1.10.0
pyaes==1.6.1
pyasn1==0.4.8
pydantic==1.8.2
Pygments==2.10.0
pylint==2.6.0
pylint-ext==0.4.0
Pympler==0.9
pyparsing==3.0.0rc1
pyRFC3339==1.1
pytest==6.2.5
pytest-asyncio==0.15.1
pytest-black==0.3.12
pytest-cov==2.10.1
pytest-env==0.6.2
pytest-isort==2.0.0
pytest-timeout==1.4.2
pytest-watch==4.2.0
python-dateutil==2.8.2
python-jose==3.3.0
python-multipart==0.0.5
pytz==2021.1
PyYAML==5.4.1
raven==6.10.0
regex==2021.8.28
requests==2.26.0
requests-mock==1.8.0
rich==10.10.0
rsa==4.7.2
s3transfer==0.5.0
semver==2.13.0
sentry-sdk==1.4.1
six==1.16.0
SQLAlchemy==1.4.23
starlette==0.14.2
stevedore==3.4.0
tabulate==0.8.9
tenacity==8.0.1
toml==0.10.2
traitlets==5.1.0
typed-ast==1.4.3
typing-extensions==3.10.0.2
urllib3==1.26.7
uvicorn==0.15.0
uvloop==0.16.0
watchdog==2.1.5
wcwidth==0.2.5
websocket-client==1.2.1
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.6.0

How can we reproduce your problem?

I am running my application with dd-trace-run and DD_PROFILING_ENABLED=true.

I started seeing the error after adding a threading.Timer() that calls that calls the same function some time ago, like described below.

I have a class that has a load() function which populates a cache. In the end of the load() function I call another function that starts a threding.Timer(), like:

    def load(self):
        # Do loading stuff

        # Set timer to reload data to cache
        self.start_cache_reload_timer()

    def start_cache_reload_timer(self):
        if self._timer is not None and self._timer.is_alive():
            self._timer.cancel()
        self._timer = threading.Timer(self._reload_period, self.load)
        self._timer.setName("CACHE_RELOAD_THREAD")
        self._timer.start()

My timer runs every 15 minutes and the profiling scheduler runs every 1 minute. The exception is raised when both match running time. However it does not happen every time that there is a match.

What is the result that you get?

I am seeing following exception in my log:

ERROR:ddtrace.profiling.scheduler:Unexpected error while exporting events. Please report this bug to https://github.com/DataDog/dd-trace-py/issues
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.7/site-packages/ddtrace/profiling/scheduler.py", line 53, in flush
exp.export(events, start, self._last_export)
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.7/site-packages/ddtrace/profiling/exporter/http.py", line 150, in export
profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)
File "ddtrace/profiling/exporter/pprof.pyx", line 671, in ddtrace.profiling.exporter.pprof.PprofExporter.export
File "ddtrace/profiling/exporter/pprof.pyx", line 320, in ddtrace.profiling.exporter.pprof._PprofConverter._build_profile
TypeError: '<' not supported between instances of 'NoneType' and 'str'

What is the result that you expected?

No exception.

@jd jd self-assigned this Nov 2, 2021
jd added a commit to jd/dd-trace-py that referenced this issue Nov 10, 2021
There was an ongoing issue wih the pprof encode mixing type between strings,
ints and None when dealing with labels. This has been fixed in multiple places
and in multiple occasion, but there were still holes.

When encoding Events to pprof, they are grouped by event fields. Those fields
might be str, int or something else. In order to group those events, all the
fields used for grouping must be:

1. hashable (because we index in a dict)
2. sortable (because we sort event to have reproducible)
3. string (because we store those as labels in pprof)

The problem is that most value type used in events are hashable (None, int or
str) but sortability often fails when one of the field is None: you can't
compare `int` with `None`, making sorted() call fails like in DataDog#2962.

As we need string in the end, the fix is to convert everything to strings as
soon as possible when grouping events: that makes sure conditions 1, 2 and 3
are validated ASAP and everything works.

To do this, the code has been updated and typed to catch any future error.

Fixes DataDog#2962
@mergify mergify bot closed this as completed in #2987 Nov 15, 2021
mergify bot added a commit that referenced this issue Nov 15, 2021
There was an ongoing issue wih the pprof encode mixing type between strings,
ints and None when dealing with labels. This has been fixed in multiple places
and in multiple occasion, but there were still holes.

When encoding Events to pprof, they are grouped by event fields. Those fields
might be str, int or something else. In order to group those events, all the
fields used for grouping must be:

1. hashable (because we index in a dict)
2. sortable (because we sort event to have reproducible)
3. string (because we store those as labels in pprof)

The problem is that most value type used in events are hashable (None, int or
str) but sortability often fails when one of the field is None: you can't
compare `int` with `None`, making sorted() call fails like in #2962.

As we need string in the end, the fix is to convert everything to strings as
soon as possible when grouping events: that makes sure conditions 1, 2 and 3
are validated ASAP and everything works.

To do this, the code has been updated and typed to catch any future error.

Fixes #2962

Co-authored-by: Brett Langdon <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
mergify bot added a commit that referenced this issue Dec 10, 2021
There was an ongoing issue wih the pprof encode mixing type between strings,
ints and None when dealing with labels. This has been fixed in multiple places
and in multiple occasion, but there were still holes.

When encoding Events to pprof, they are grouped by event fields. Those fields
might be str, int or something else. In order to group those events, all the
fields used for grouping must be:

1. hashable (because we index in a dict)
2. sortable (because we sort event to have reproducible)
3. string (because we store those as labels in pprof)

The problem is that most value type used in events are hashable (None, int or
str) but sortability often fails when one of the field is None: you can't
compare `int` with `None`, making sorted() call fails like in #2962.

As we need string in the end, the fix is to convert everything to strings as
soon as possible when grouping events: that makes sure conditions 1, 2 and 3
are validated ASAP and everything works.

To do this, the code has been updated and typed to catch any future error.

Fixes #2962

Co-authored-by: Brett Langdon <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit 6520212)
mergify bot added a commit that referenced this issue Dec 10, 2021
…) (#3062)

There was an ongoing issue wih the pprof encode mixing type between strings,
ints and None when dealing with labels. This has been fixed in multiple places
and in multiple occasion, but there were still holes.

When encoding Events to pprof, they are grouped by event fields. Those fields
might be str, int or something else. In order to group those events, all the
fields used for grouping must be:

1. hashable (because we index in a dict)
2. sortable (because we sort event to have reproducible)
3. string (because we store those as labels in pprof)

The problem is that most value type used in events are hashable (None, int or
str) but sortability often fails when one of the field is None: you can't
compare `int` with `None`, making sorted() call fails like in #2962.

As we need string in the end, the fix is to convert everything to strings as
soon as possible when grouping events: that makes sure conditions 1, 2 and 3
are validated ASAP and everything works.

To do this, the code has been updated and typed to catch any future error.

Fixes #2962

Co-authored-by: Brett Langdon <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit 6520212)

Co-authored-by: Julien Danjou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants