APM HTTP request information missing

[AhmadMS1988]

Describe the bug: ...
The request information like url.path and other information like headers and cookies are not found like what we were having in older elastic-apm version.

Environment (please complete the following information)

• OS: container: amazon linux 2, host bottlerocket 1.1.1
• Python version: 3.7.9
• Framework and version [e.g. Django 2.1]: Django 2.2.11, djangorestframework==3.10.3, requests==2.22.0, mod-wsgi==4.8.0
• APM Server version: 7.13.0, deployed with ECK 1.6.0
• Agent version: 6.1.3 and 6.2.1
• Kibana version: 7.13.0
• Elasticsearch version: 7.13.0

Additional context

Add any other context about the problem here.

• Agent config options

  Click to expand

  ELASTIC_APM = {
      'SERVER_URL': 'https://elastic-apm-http.elasticsearch:8200',
      'SECRET_TOKEN': '******',
      'SERVICE_NAME': 'my-service-name',
      'ENVIRONMENT': 'dev',
      'CAPTURE_BODY': 'all',
      'DEBUG': True, # Please not that DEBUG is true to match Django debug on the dev environment.
      'INSTRUMENT': True,
      'TRANSACTION_IGNORE_URLS': ['/metrics', '*/health_check'],
      'DJANGO_TRANSACTION_NAME_FROM_ROUTE': True,
      'VERIFY_SERVER_CERT': False
  }
  

• requirements.txt:

  Click to expand

  django==2.2.11
  mysqlclient==1.4.6
  djangorestframework==3.10.3
  requests==2.22.0
  pytz==2020.1
  django-slack==5.14.3
  SQLAlchemy==1.3.13
  marshmallow==3.3.0
  marshmallow-sqlalchemy==0.21.0
  boto3==1.17.86
  drf-nested-routers==0.91
  elastic-apm==5.10.1
  mod-wsgi==4.8.0
  django-prometheus==2.1.0
  elasticsearch-dsl==7.1.0
  requests-aws4auth==0.9
  

• apm-server.yml:

  Click to expand

  apiVersion: apm.k8s.elastic.co/v1
  kind: ApmServer
  metadata:
    name: elastic
    namespace: elasticsearch
  spec:
    version: 7.13.0
    count: 4
    config:
      logging.level: warning
      monitoring.enabled: true
      output.elasticsearch.workers: 1
      output.elasticsearch.bulk_max_size: 4096
      queue.mem.events: 8192
    elasticsearchRef:
      name: elastic
    kibanaRef:
      name: elastic
    podTemplate:
      metadata:
        labels:
          apm-name: elastic
          elastic-rule: apm
      spec:
        containers:
          - name: apm-server
            resources:
              requests:
                memory: 1Gi
                cpu: .25
              limits:
                memory: 2Gi
                cpu: .5
        affinity:
          podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: elastic-rule
                    operator: In
                    values:
                    - apm
                topologyKey: "kubernetes.io/hostname"
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: workload
                  operator: In
                  values:
                  - es
  

Please note that when I downgraded to elastic-apm==5.10.1 the request information are displayed again.
Is it a removed feature, or there is something extra needs to be configured for 6.2.1?
best regards

[beniwohli]

Hi @AhmadMS1988! Could you try and see if the information is there using version 6.1.2 of the agent? That would probably mean that #1117 introduced the bug

[AhmadMS1988]

@beniwohli thank you for your response.
I have tried every version down to 6.1.0 and did not work.
only 6.0.0 and below are working.

[beniwohli]

@AhmadMS1988 I have a hunch that this is something similar to what was reported in #1093. It's possible that somewhere, a new client instance is created, that isn't a DjangoClient, but a bare Client instance, which lacks the Django-specific methods to collect request data.

Do you per chance use elasticapm.handlers.logging.LoggingHandler and/or elasticapm.contrib.django.handlers.LoggingHandler in your logging configuration? If yes, could you post your (redacted) logging configuration here?

[AhmadMS1988]

Unfortunately no, but how can I enable this and where to get the output?
Also, in apache error, I have no exception.

[AhmadMS1988]

after enabling apm logging I found the following traceback:

during timing of request#012Traceback (most recent call last):#012 File "/var/python/venv/lib64/python3.7/site-packages/elasticapm/contrib/django/middleware/__init__.py", line 182, in process_response#012 lambda: self.client.get_data_from_request(request, constants.TRANSACTION), "request"#012 File "/var/python/venv/lib64/python3.7/site-packages/elasticapm/traces.py", line 852, in set_context#012 data = data()#012 File "/var/python/venv/lib64/python3.7/site-packages/elasticapm/contrib/django/middleware/__init__.py", line 182, in <lambda>#012 lambda: self.client.get_data_from_request(request, constants.TRANSACTION), "request"#012AttributeError: 'Client' object has no attribute 'get_data_from_request'

[beniwohli]

Hey @AhmadMS1988! That's helpful, as it confirms that this is indeed the same problem as reported in #1093.

I pushed a small change to a branch (#1158) that logs the stack trace when the client is set more than once. This could help debugging where the "naked" Client instance is created. You can install it by running something like

pip install https://github.com/beniwohli/apm-agent-python/archive/refs/heads/log-set_client-with-stack.zip\#egg\=elastic-apm\=\=100

It's logged at DEBUG level, using the elasticapm logger, so you might have to update your logging configuration to see the message.

If you could try to run this in a dev environment and report back any stack traces near the Client object is being set more than once message, that would help a lot!

[AhmadMS1988]

The attached logs are extracted using filebeat when enabling DEBUG level, using the elasticapm logger

apm-logs (1).csv

[beniwohli]

@AhmadMS1988 I think I see the problem. In announcement/components/sqs_component.py, line 33, you instantiate a new APM client instance. Can you instead try do something like this:

import elasticapm
self.apm_client = get_client()

That should avoid that the initial Django-specific APM client is replaced with a non-specific APM client.

[AhmadMS1988]

Brilliant, and worked as expected.
Thank you for your help :)