New Evals for newrelic (#983)

Co-authored-by: Tomer Keshet <[email protected]>
Co-authored-by: mershal <[email protected]>
Co-authored-by: Robusta Runner <[email protected]>

Author: 4 people

holmes/plugins/prompts/_fetch_logs.jinja2

@@ -45,6 +45,8 @@ Tools to search and fetch logs from Datadog.
 * IMPORTANT: If a limit was applied, ALWAYS tell the user how many logs were shown vs total (e.g., "Showing latest <Y> of <Z> logs")
 * IMPORTANT: If any filters were applied, ALWAYS mention them explicitly
 {%- elif k8s_yaml_ts and k8s_yaml_ts.status == "enabled" -%}
+### Logs from newrelic
+* you can fetch logs from newrelic if this is toolset is enabled
 ### kubernetes/logs
 #### Kubernetes Logs Toolset
 Tools to search and fetch logs from Kubernetes.
@@ -65,4 +67,5 @@ DO NOT use `--tail` or `| tail` when calling `kubectl logs` because you may miss
 ** 'opensearch/logs'
 ** 'coralogix/logs'
 ** 'datadog/logs'
+** 'newrelic'
 {%- endif -%}

holmes/plugins/toolsets/newrelic/newrelic.jinja2

@@ -28,6 +28,8 @@ Note: Use `SHOW EVENT TYPES` to see all event types in the account, in addition
 #### 3. Querying Traces
 - Always validate first: run the base query without FACET (or a quick LIMIT) to confirm data exists; if results are empty, adjust filters or time range before proceeding.
 - Only attempt a FACET after confirming the field has values; if not, either try known alternatives or skip faceting entirely.
+- When investigating a trace also look at attributes
+- ***When investigating latency ALWAYS look to deliver the specific component or attribute in the span causing significant latnecy*** your investigation is not complete without this
 
 ### 3. Metrics/graphs
 - You always need to embed the partial result in your response. Include the "tool_name" and "random_key". For example: << {"type": "newrelicql", "tool_name": "execute_prometheus_range_query", "random_key": "92jf2hf"} >>

holmes/plugins/toolsets/newrelic/newrelic.py

@@ -222,7 +222,7 @@ def __init__(self):
             tools=[
                 ExecuteNRQLQuery(self),
             ],
-            experimental=True,
+            experimental=False,
             tags=[ToolsetTag.CORE],
         )
         template_file_path = os.path.abspath(

pyproject.toml

@@ -84,13 +84,15 @@ scripts_are_modules = true
 exclude = [
     "tests/llm/fixtures/.*",
     "dist/.*",
+    "site/.*",
 ]
 
 [tool.pytest.ini_options]
 markers = [
     "llm: Evaluate LLM behaviour (prompt, tools, etc.)",
     "datetime: Tests involving datetime functionality",
     "logs: Tests involving log processing",
+    "metrics: Tests involving metrics processing",
     "context_window: Tests involving context window handling",
     "synthetic: Tests using synthetic data",
     "network: Tests requiring network connectivity",