Skip to content

Profiling pprof converter is raising exception when sorting _locations #2962

@Truja

Description

@Truja

Which version of dd-trace-py are you using?

0.54.0rc1

Which version of pip are you using?

21.3.1

Which version of the libraries are you using?

aiofiles==0.7.0
appdirs==1.4.4
appnope==0.1.2
asgiref==3.4.1
astroid==2.5
async-lru==1.0.2
attrs==21.2.0
Babel==2.8.0
backcall==0.2.0
black==19.10b0
boto3==1.18.51
botocore==1.21.51
cachetools==4.2.4
certifi==2021.5.30
chardet==4.0.0
charset-normalizer==2.0.6
click==8.0.1
colorama==0.4.4
commonmark==0.9.1
coverage==5.5
coveralls==3.2.0
datadog==0.42.0
ddtrace==0.54.0rc1
decorator==5.1.0
dnslib==0.9.16
dnspython==1.16.0
docker==5.0.0
docopt==0.6.2
dulwich==0.20.25
ecdsa==0.18.0b1
expiringdict==1.1.4
fastapi==0.68.1
Flask==2.0.1
greenlet==1.1.1
guppy3==3.1.1
h11==0.12.0
idna==3.2
importlib-metadata==4.8.1
inflect==5.3.0
iniconfig==1.1.1
ipython==7.28.0
isort==5.5.4
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
jmespath==0.10.0
launchdarkly-server-sdk==7.2.0
lazy-object-proxy==1.6.0
localstack==0.12.17.5
localstack-client==1.24
localstack-ext==0.12.17
lxml==4.6.3
MarkupSafe==2.0.1
matplotlib-inline==0.1.3
mccabe==0.6.1
microservice-python==4.1.7
packaging==21.0
parso==0.8.2
pathspec==0.9.0
pbr==5.6.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==1.0.0
prompt-toolkit==3.0.20
protobuf==3.18.0
ptyprocess==0.7.0
py==1.10.0
pyaes==1.6.1
pyasn1==0.4.8
pydantic==1.8.2
Pygments==2.10.0
pylint==2.6.0
pylint-ext==0.4.0
Pympler==0.9
pyparsing==3.0.0rc1
pyRFC3339==1.1
pytest==6.2.5
pytest-asyncio==0.15.1
pytest-black==0.3.12
pytest-cov==2.10.1
pytest-env==0.6.2
pytest-isort==2.0.0
pytest-timeout==1.4.2
pytest-watch==4.2.0
python-dateutil==2.8.2
python-jose==3.3.0
python-multipart==0.0.5
pytz==2021.1
PyYAML==5.4.1
raven==6.10.0
regex==2021.8.28
requests==2.26.0
requests-mock==1.8.0
rich==10.10.0
rsa==4.7.2
s3transfer==0.5.0
semver==2.13.0
sentry-sdk==1.4.1
six==1.16.0
SQLAlchemy==1.4.23
starlette==0.14.2
stevedore==3.4.0
tabulate==0.8.9
tenacity==8.0.1
toml==0.10.2
traitlets==5.1.0
typed-ast==1.4.3
typing-extensions==3.10.0.2
urllib3==1.26.7
uvicorn==0.15.0
uvloop==0.16.0
watchdog==2.1.5
wcwidth==0.2.5
websocket-client==1.2.1
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.6.0

How can we reproduce your problem?

I am running my application with dd-trace-run and DD_PROFILING_ENABLED=true.

I started seeing the error after adding a threading.Timer() that calls that calls the same function some time ago, like described below.

I have a class that has a load() function which populates a cache. In the end of the load() function I call another function that starts a threding.Timer(), like:

    def load(self):
        # Do loading stuff

        # Set timer to reload data to cache
        self.start_cache_reload_timer()

    def start_cache_reload_timer(self):
        if self._timer is not None and self._timer.is_alive():
            self._timer.cancel()
        self._timer = threading.Timer(self._reload_period, self.load)
        self._timer.setName("CACHE_RELOAD_THREAD")
        self._timer.start()

My timer runs every 15 minutes and the profiling scheduler runs every 1 minute. The exception is raised when both match running time. However it does not happen every time that there is a match.

What is the result that you get?

I am seeing following exception in my log:

ERROR:ddtrace.profiling.scheduler:Unexpected error while exporting events. Please report this bug to https://github.com/DataDog/dd-trace-py/issues
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.7/site-packages/ddtrace/profiling/scheduler.py", line 53, in flush
exp.export(events, start, self._last_export)
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.7/site-packages/ddtrace/profiling/exporter/http.py", line 150, in export
profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)
File "ddtrace/profiling/exporter/pprof.pyx", line 671, in ddtrace.profiling.exporter.pprof.PprofExporter.export
File "ddtrace/profiling/exporter/pprof.pyx", line 320, in ddtrace.profiling.exporter.pprof._PprofConverter._build_profile
TypeError: '<' not supported between instances of 'NoneType' and 'str'

What is the result that you expected?

No exception.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions