TypeError: '<' not supported between instances of 'int' and 'NoneType'

[Apakottur]

Thanks for this awesome integration with Python.

We are using Sentry and Datadog (with profiling enabled) to monitor our system.
Occasionally, we get Datadog errors, like the following one:

sentry_sdk.integrations.logging:sentry_patched_callhandlers:86 - Unexpected error while exporting events.

Please report this bug to https://github.com/DataDog/dd-trace-py/issuesTraceback (most recent call last): |

File "/usr/local/lib/python3.10/threading.py", line 966, in _bootstrap

self._bootstrap_inner()

<function Thread._bootstrap_inner at 0x7fe4ded153f0>

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/threading.py", line 1009, in _bootstrap_inner

self.run()

<function PeriodicThread.run at 0x7fe4dd2312d0>

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/periodic.py", line 70, in run

self._target()

<bound method Scheduler.periodic of Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events...

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/scheduler.py", line 65, in periodic

self.flush()

<function Scheduler.flush at 0x7fe4dd28bb50>

Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events=32768, max_events={<class 'ddtrace....

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/scheduler.py", line 53, in flush

exp.export(events, start, self._last_export)

Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events=32768, max_events={<class 'ddtrace....

{<class 'ddtrace.profiling.collector.stack.StackSampleEvent'>: deque([StackSampleEvent(timestamp=1637081996232555786, samplin...

<function PprofHTTPExporter.export at 0x7fe4dbf95630>

PprofHTTPExporter(endpoint='http://localhost:8126', api_key=None, timeout=10.0, service=None, env=None, version=None, tags={'...

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/exporter/http.py", line 150, in export

profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)

{<class 'ddtrace.profiling.collector.stack.StackSampleEvent'>: deque([StackSampleEvent(timestamp=1637081996232555786, samplin...

PprofHTTPExporter(endpoint='http://localhost:8126', api_key=None, timeout=10.0, service=None, env=None, version=None, tags={'...

<class 'ddtrace.profiling.exporter.http.PprofHTTPExporter'>

File "ddtrace/profiling/exporter/pprof.pyx", line 535, in ddtrace.profiling.exporter.pprof.PprofExporter.export

File "ddtrace/profiling/exporter/pprof.pyx", line 406, in ddtrace.profiling.exporter.pprof.PprofExporter._group_stack_events

TypeError: '<' not supported between instances of 'int' and 'NoneType'

I don't know if it's related to Sentry or not, and I don't have anything except the traceback.

This happens rarely, and we are able to use both tools without issues.

Which version of dd-trace-py are you using?

ddtrace = "0.55.4"

Which version of pip are you using?

pip 21.2.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)

Which version of the libraries are you using?

aiobotocore==2.0.0
aiodns==3.0.0
aiohttp==3.8.1
aioitertools==0.8.0
aioredis==1.3.1
aiosignal==1.2.0
aiosmtplib==1.1.6
alembic==1.7.5
anyio==3.3.4
arq==0.22
asgiref==3.4.1
async-timeout==4.0.1
asyncpg==0.25.0
attrs==21.2.0
Babel==2.9.1
backports.entry-points-selectable==1.1.1
bcrypt==3.2.0
boto3==1.19.8
boto3-stubs==1.20.8
botocore==1.22.8
botocore-stubs==1.23.8
CacheControl==0.12.10
cached-property==1.5.2
cachy==0.3.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.7
cleo==0.8.1
click==8.0.3
clikit==0.6.2
coverage==6.1.2
crashtest==0.3.1
cryptography==35.0.0
ddtrace==0.55.4
distlib==0.3.3
dnspython==2.1.0
email-normalize==2.0.0
email-validator==1.1.3
execnet==1.9.0
filelock==3.3.2
freezegun==1.1.0
frozenlist==1.2.0
graphql-core==3.1.6
greenlet==1.1.2
gunicorn==20.1.0
h11==0.12.0
hiredis==2.0.0
html5lib==1.1
httpcore==0.14.3
httptools==0.2.0
httpx==0.21.1
idna==3.3
iniconfig==1.1.1
jeepney==0.7.1
Jinja2==3.0.3
jmespath==0.10.0
keyring==21.8.0
lockfile==0.12.2
loguru==0.5.3
Mako==1.1.6
MarkupSafe==2.0.1
migra==3.0.1621480950
more-itertools==8.11.0
moto==2.2.15
msgpack==1.0.2
multidict==5.2.0
mypy==0.910
mypy-extensions==0.4.3
packaging==21.3
passlib==1.7.4
pastel==0.2.1
pexpect==4.8.0
phonenumbers==8.12.37
pkginfo==1.7.1
platformdirs==2.4.0
pluggy==1.0.0
poetry==1.1.11
poetry-core==1.0.7
protobuf==3.19.1
psycopg2==2.9.2
ptyprocess==0.7.0
py==1.11.0
pycares==4.1.2
pycparser==2.21
pydantic==1.8.2
Pygments==2.10.0
PyJWT==2.3.0
pylev==1.4.0
pyparsing==3.0.6
pytest==6.2.5
pytest-asyncio==0.16.0
pytest-cov==3.0.0
pytest-forked==1.3.0
pytest-httpx==0.15.0
pytest-mock==3.6.1
pytest-timeout==2.0.1
pytest-xdist==2.4.0
python-dateutil==2.8.2
python-dotenv==0.19.2
python-multipart==0.0.5
pytz==2021.3
PyYAML==6.0
requests==2.26.0
requests-toolbelt==0.9.1
responses==0.16.0
rfc3986==1.5.0
s3transfer==0.5.0
schemainspect==3.0.1616029793
SecretStorage==3.3.1
sentinel==0.3.0
sentry-sdk==1.5.0
shellingham==1.4.0
shpyx==0.0.10
simplejson==3.17.6
six==1.16.0
sniffio==1.2.0
SQLAlchemy==1.4.27
sqlalchemy2-stubs==0.0.2a19
sqlbag==0.1.1617247075
starlette==0.17.1
strawberry-graphql==0.87.1
tenacity==8.0.1
toml==0.10.2
tomli==1.2.2
tomlkit==0.7.2
types-babel==2.9.1
types-python-dateutil==2.8.2
types-simplejson==3.17.1
typing-extensions==3.10.0.2
urllib3==1.26.7
uvicorn==0.15.0
uvloop==0.16.0
virtualenv==20.10.0
watchgod==0.7
webencodings==0.5.1
websockets==10.1
Werkzeug==2.0.2
wrapt==1.13.3
xmltodict==0.12.0
yarl==1.7.2

[jd]

Likely a duplicate of #2962
It should be fixed in the next release 🤞

[Apakottur]

Awesome, thanks for the response!
Feel free to close this if its a duplicate.

[jd]

Thank you — closing. Feel free to reopen if it's not fixed in the next release!

[Apakottur]

I just had this error again on 0.56.1:

Should we reopen the issue? Can't do it myself.

[jd]

It's scheduled for 0.57 but I'll backport it for 0.56 in #3062

[Apakottur]

@jd We just got this again on 0.57.3, should I open a new issue?

[adamos-kyriakou-wolt]

I can verify we're still getting these errors on the latest v0.57.3 version as well. @Apakottur any progress?

Weirdly enough we only started getting these after introducing https://github.com/claws/aioprometheus for capturing metrics against our prometheus instance

[Apakottur]

@adamos-kyriakou-wolt Not too much, we keep getting this error and just ignore it for now.
I also suspect that this has to do with other monitoring tools, like Sentry which we use in parallel.

One thing that I noticed is that we get this almost every time when we deploy our application (and the DD initialization code runs), but it also happens during the runtime of the application.

[adamos-kyriakou-wolt]

Same thing here @Apakottur, we're running both Sentry and DD over our service and get these errors during runtime. Let's hope @jd will have some news for us both 🙏