[gen-ai] gen_ai.agent semantic conventions need to be expanded further to include tools list

[PRATIBHA-Moogi]

Area(s)

area:gen-ai

What's missing?

Current gen_ai.agent attributes only include the below set of fields:

  - id: gen_ai.agent.id   # alternatives: assistant (openai)
    stability: development
    type: string
    brief: The unique identifier of the GenAI agent.
    examples: ['asst_5j66UpCpwteGg4YSxUnt7lPY']
  - id: gen_ai.agent.name
    stability: development
    type: string
    brief: Human-readable name of the GenAI agent provided by the application.
    examples: ["Math Tutor", "Fiction Writer"]
  - id: gen_ai.agent.description
    stability: development
    type: string
    brief: Free-form description of the GenAI agent provided by the application.
    examples: ["Helps with math problems", "Generates fiction stories"]

The current agentic application framework - OpenAI, langchain, langraph, autogen, llamaindex, crew.ai, most recent Google ADK include llm name and tools list as part of agent configuration parameters set. Agent's performance solely depends on which llm it has employed, what set of tools it is configured to use. Hence expressing them as a key attributes applied to agents observability data - metrics, logs, traces important for drawing higher lever of diagnostics such as if error count is found higher then is that due to certain specific llm used for agent's task execution or due to a specific set of tools employed in the agentic workflow ?

This issue proposes the addition of llm name and tools list part of gen_ai.agent.[] semantic conventions / attributes.

Below is the proposal:

  - id: gen_ai.agent.llm
    stability: development
    type: string
    brief: model name as per huggingface naming conventions.
    examples: ["ibm-granite/granite-3.2-2b-instruct"]

  - id: gen_ai.agent.tools
    stability: development
    type: string[]
    brief: list of tools name with which agent is configured.
    examples: ["fun1_name", "func2_name"]

Describe the solution you'd like

No response

[PRATIBHA-Moogi]

@gyliu513 one more proposal for expanding gen_ai.agent semantic conventions.

[gyliu513]

@PRATIBHA-Moogi agent already extends genai common attributes which include models https://github.com/open-telemetry/semantic-conventions/blob/main/model/gen-ai/spans.yaml#L185

For gen_ai.agent.tools, I recalled there are some discussion before, but seems we need to add this now, we can probably focus on this first.

@lmolkova ^^

[PRATIBHA-Moogi]

@gyliu513 As I understand gen_ai.request.model and other attributes applied at this level are meant for backend inference engines (vllm, tgis etc). Similarly gen_ai.tool.xx attributes will be applied at tool-calling server level. At Agent Application level one will use gen_ai.agent.xx attributes to tag trace,logs,metrics using observability SDKs. In order to provide a view such as tools performance on the basis of Agent employed llm, you need to tag Agent application observability data with llm name.

[gyliu513]

@PRATIBHA-Moogi Yes, I get your point, if you take a look at crewai agent code https://www.crewai.com/open-source , you will see they defined an LLM, but the LLM is exactly using the attributes that we defined here https://github.com/open-telemetry/semantic-conventions/blob/main/model/gen-ai/spans.yaml#L2 via trace.gen_ai.client.common_attributes.

If you want to add an attribute as below, it will be duplicate with https://github.com/open-telemetry/semantic-conventions/blob/main/model/gen-ai/spans.yaml#L8 gen_ai.request.model, comments?

- id: gen_ai.agent.llm
    stability: development
    type: string
    brief: model name as per huggingface naming conventions.
    examples: ["ibm-granite/granite-3.2-2b-instruct"]

[lmolkova]

I'd prefer to put the whole tool definition in the attribute. We can put it as a json blob for now and once/if spec allows (open-telemetry/opentelemetry-specification#4485) would record them as complex attributes. See #2010 and Modeling GenAI calls on telemetry in particular.

[PRATIBHA-Moogi]

Thanks @lmolkova looking into it.