Introducing RedTeam (#39898)

* remove redundant quotes

* Fix typo

* pylint fix

* Update broken tests

* Include the grounding json in the manifest

* Fix typo

* Come on package

* Release 1.0.0b5

* Notice from Chang

* Remove adv_conv template parameters from the outputs

* Update chanagelog

* Experimental tags on adv scenarios

* Readme fix onbreaking change

* Add the category and both user and assistant context to the response of qr_json_lines

* Update changelog

* Rename _kwargs to _options

* _options as prefix

* update troubleshooting for simulator

* Rename according to suggestions

* Clean up readme

* more links

* Bugfix: zip_longest created null parameters

* Updated changelog

* zip does the job

* remove ununsed import

* Fix changelog merge

* Remove print statements

* Adding pyrit dependency

* updates

* updates

* Make pyrit extra

* Set up pyrit as extra correctly

* Limit the number of turns

* adding sample

* baseline sample with evals

* updates

* Update setup to use roman's branch

* callback chat target

* sample update

* add local callback chat target

* Revert to adding main pyrit as extra

* add option for simulation only

* budget first pass error on model target

* low budget working (without many shot)

* Adding RedTeamAgent

* updates for parallelism and cleanup

* add different converters, dispose of memory

* add typespec autogen files

* updates

* removing unused orchestrators and ensuring baseline always included

* first attempt to get attack objectives

* retrieve attack objectives sucessfully

* cachine to retrieve multiple objectives for each of the attack strategy

* Add generated client

* Add content

* Remove params argument from get method call

* scorecard and more targets

* I did mess up that merge oops

* Make attack objective generator mandatory

* Remove debug statement from _red_team_agent.py

* get rai call  working

* nits

* remove change not needed

* whitespace

* mock call to evaluate

* Remove caching for objectives, get num_objectives from the attack objective generator

* update the sample with attack objective generator

* scorecard updates

* remove baseline from detailed_joint_risk_attack_asr

* smaller changes

* Update all the content safety evalutors to have a pass/fail result and treshold

* Update groundedness service based

* Binary results for prompt based evaluators

* Update changelog

* new typespec gen and readme

* Newly generated client responds with objectives properly

* mlflow run

* Pass -> pass Fail -> fail

* Add thresholds to NLP evals

* mlflow updates

* init idea, waiting for tunnels or int to work

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_gleu/_gleu.py

Co-authored-by: Copilot <[email protected]>

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py

Co-authored-by: Copilot <[email protected]>

* print scorecard init call to evaluate and mock attackobjectives until int is fixed

* Make a call to jailbreak and prepend response

* flip call to evaluate instead of mock

* Binarization in rouge

* Adding threshold to all evaluators

* adding progress bar

* fixing and supressing errors

* more updates

* syntax error

* More syntax fifxes

* Typo fixes

* print a message if exception occurs for binary result calc

* Final typo

* start MLFlow run earlier

* Update built in evals test

* updates for call to evaluate

* draft of binarization

* RE add the previously removed _label

* Trying a fix for the test

* Why ar we checking len of keys instead of the keys themselves

* refactoring

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team_agent/red_team_agent_result.py

Co-authored-by: Nagkumar Arkalgud <[email protected]>

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team_agent/red_team_agent_result.py

Co-authored-by: Nagkumar Arkalgud <[email protected]>

* address feedback

* Update redundant comment and change to

* Yaay tests passed

* Fix bug

* uncomment recording

* updates

* Update

* use the values for accessing keys

* updates

* updates

* parsing risk category

* Handle data only scenario

* update

* Fix the keys for risk category to be lower case in result and show int url for ai studio

* Fix the way we render results

* updates

* updates

* Updates to use the new generated client

* updates

* version pin pyrit

* Update to include autogen files in setup

* Add init files and update setup

* Removed mocks

* risk category to output and update to asr calc

* revert unnecessary changes

* debug content filter error for open ai target

* minor updates

* mark startergies that are not supported

* update jailbreak retrieval

* update objective filtering logic

* fix safety eval unit tests

* remove empty utils subfolder

* init attempt to move things to diff files and add tests

* Fix the cspell error

* feat(security): Add RedTeamAgent for AI system vulnerability assessment

Implements a comprehensive RedTeamAgent feature for systematically
testing AI system security vulnerabilities using various attack strategies.

Key additions:
- Red Team Agent class with support for multiple attack strategies (Base64, ROT13, Jailbreak, etc.)
- Risk category assessment across Violence, SelfHarm, Sexual, and HateUnfairness domains
- MLflow integration for experiment tracking and result visualization
- Comprehensive scoring metrics including Attack Success Rate (ASR)
- Detailed test coverage for all major components
- Updated CHANGELOG.md with feature documentation

The RedTeamAgent helps security teams and AI developers evaluate system robustness
against potential attacks and provides detailed analytics on vulnerabilities.

* Revert all the changes to difficulty

* refactor(redteam): Simplify scorecard formatting output

Removed redundant 'Scorecard:' header and studio URL from output for cleaner display.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* test(redteam): Update formatting_utils tests

Updated test_formatting_utils.py to match the simplified scorecard format by removing assertions for removed elements.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* test(redteam): Fix Red Team Agent unit tests

Updated test_red_team_agent.py to properly mock logging, file handlers, and tempfile operations to support scan-specific output folders and MLflow integration.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* feat(redteam): Add output directory support to logging

Modified logging_utils.py to accept output_dir parameter for scan-specific log files.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

* making it work, thanks claude

* Allow custom attac objectives

* keeping a count of timeout, add a test

* Update init param and test

* update to aog and scan name

* Skip work in progress tests

* fix timeout and other small tweaks

* add init file to utils

* Make redteam agent tests optional CI stage

* trying to update ci to require installation of redteam extra

* update naming, unit tests

* minor updates

* update sample with release installation

* remove sample notebook

* Please pass CI analyze

* Again CI analyze please pass

* Again CI analyze please pass

---------

Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Miles Holland <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Claude <[email protected]>

Author: 7 people

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_exceptions.py

@@ -84,6 +84,7 @@ class ErrorTarget(Enum):
     UNKNOWN = "Unknown"
     CONVERSATION = "Conversation"
     TOOL_CALL_ACCURACY_EVALUATOR = "ToolCallAccuracyEvaluator"
+    RED_TEAM = "RedTeam"
 
 
 class EvaluationException(AzureError):

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/__init__.py

@@ -1,3 +1,3 @@
 # ---------------------------------------------------------
 # Copyright (c) Microsoft Corporation. All rights reserved.
-# ---------------------------------------------------------
+# ---------------------------------------------------------

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/_generated_rai_client.py

@@ -0,0 +1,38 @@
+# Azure AI Evaluation client library for Python
+
+## References
+
+- [How to generate client libraries](https://azure.github.io/typespec-azure/docs/howtos/generate-client-libraries/00howtogen/)
+
+# Getting started
+
+## Typespec setup
+
+- Install the typespec [compiler](https://typespec.io/docs/) by following the "Install tsp" section. You should be able to run `tsp` in your terminal once this is done.
+
+## Raiclient
+
+TODO: Have someone who did more than blindly stumble their way to a solution double check this, especially the step about modifying the tspconfig.
+
+[Source](https://github.com/Azure/azure-rest-api-specs/tree/gaugup/AddTypeSpecRAISvc/specification/ai/Azure.AI.Projects/RAISvc) Note, currently not on main. Use main once committed.
+
+- Navigate to `azure-rest-api-specs/specification/ai/Azure.AI.Projects/RAISvc` and make sure you're in the right repo branch.
+- The contents of `./tspconfig.yaml` are set to emit autorest files. Temporarily replace the file with the following:
+```yaml
+emit:
+  - "@azure-tools/typespec-python"
+options:
+  "@azure-tools/typespec-python":
+    package-name: "rai_client"
+    output-dir: "./rai_client"
+```
+- Run from  run ` tsp compile main.tsp`
+- Check the output. Autogenerated files should be in `tsp-output/@azure-tools/typespec-python/raiclient/`
+- Replace the contents of `azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient` with the newly generated code.
+- Cleanup lingering files and revert the config changed.
+
+# Key concepts
+# Examples
+# Troubleshooting
+# Next steps
+# Contributing

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/_safety_evaluation.py

@@ -0,0 +1,7 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from .raiclient import MachineLearningServicesClient
+
+__all__ = ["MachineLearningServicesClient"]

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/README.md

@@ -0,0 +1,34 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) Python Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+# pylint: disable=wrong-import-position
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from ._patch import *  # pylint: disable=unused-wildcard-import
+
+from ._client import MachineLearningServicesClient  # type: ignore
+from ._version import VERSION
+
+__version__ = VERSION
+
+try:
+    from ._patch import __all__ as _patch_all
+    from ._patch import *
+except ImportError:
+    _patch_all = []
+from ._patch import patch_sdk as _patch_sdk
+
+# Export GeneratedRAIClient as alias of MachineLearningServicesClient for backward compatibility 
+
+__all__ = [
+    "MachineLearningServicesClient",
+]
+__all__.extend([p for p in _patch_all if p not in __all__])  # pyright: ignore
+
+_patch_sdk()

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/__init__.py

@@ -0,0 +1,128 @@
+# pylint: disable=line-too-long,useless-suppression
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) Python Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+from copy import deepcopy
+from typing import Any, TYPE_CHECKING
+from typing_extensions import Self
+
+from azure.core import PipelineClient
+from azure.core.pipeline import policies
+from azure.core.rest import HttpRequest, HttpResponse
+
+from ._configuration import MachineLearningServicesClientConfiguration
+from ._serialization import Deserializer, Serializer
+from .operations import RAISvcOperations
+
+if TYPE_CHECKING:
+    from azure.core.credentials import TokenCredential
+
+
+class MachineLearningServicesClient:
+    """MachineLearningServicesClient.
+
+    :ivar rai_svc: RAISvcOperations operations
+    :vartype rai_svc: raiclient.operations.RAISvcOperations
+    :param endpoint: Supported Azure-AI endpoints. Required.
+    :type endpoint: str
+    :param subscription_id: The ID of the target subscription. Required.
+    :type subscription_id: str
+    :param resource_group_name: The name of the Resource Group. Required.
+    :type resource_group_name: str
+    :param workspace_name: The name of the AzureML workspace or AI project. Required.
+    :type workspace_name: str
+    :param credential: Credential used to authenticate requests to the service. Required.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :keyword api_version: The API version to use for this operation. Default value is
+     "2022-11-01-preview". Note that overriding this default value may result in unsupported
+     behavior.
+    :paramtype api_version: str
+    """
+
+    def __init__(
+        self,
+        endpoint: str,
+        subscription_id: str,
+        resource_group_name: str,
+        workspace_name: str,
+        credential: "TokenCredential",
+        **kwargs: Any
+    ) -> None:
+        _endpoint = "{endpoint}/raisvc/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}"
+        self._config = MachineLearningServicesClientConfiguration(
+            endpoint=endpoint,
+            subscription_id=subscription_id,
+            resource_group_name=resource_group_name,
+            workspace_name=workspace_name,
+            credential=credential,
+            **kwargs
+        )
+        _policies = kwargs.pop("policies", None)
+        if _policies is None:
+            _policies = [
+                policies.RequestIdPolicy(**kwargs),
+                self._config.headers_policy,
+                self._config.user_agent_policy,
+                self._config.proxy_policy,
+                policies.ContentDecodePolicy(**kwargs),
+                self._config.redirect_policy,
+                self._config.retry_policy,
+                self._config.authentication_policy,
+                self._config.custom_hook_policy,
+                self._config.logging_policy,
+                policies.DistributedTracingPolicy(**kwargs),
+                policies.SensitiveHeaderCleanupPolicy(**kwargs) if self._config.redirect_policy else None,
+                self._config.http_logging_policy,
+            ]
+        self._client: PipelineClient = PipelineClient(base_url=_endpoint, policies=_policies, **kwargs)
+
+        self._serialize = Serializer()
+        self._deserialize = Deserializer()
+        self._serialize.client_side_validation = False
+        self.rai_svc = RAISvcOperations(self._client, self._config, self._serialize, self._deserialize)
+
+    def send_request(self, request: HttpRequest, *, stream: bool = False, **kwargs: Any) -> HttpResponse:
+        """Runs the network request through the client's chained policies.
+
+        >>> from azure.core.rest import HttpRequest
+        >>> request = HttpRequest("GET", "https://www.example.org/")
+        <HttpRequest [GET], url: 'https://www.example.org/'>
+        >>> response = client.send_request(request)
+        <HttpResponse: 200 OK>
+
+        For more information on this code flow, see https://aka.ms/azsdk/dpcodegen/python/send_request
+
+        :param request: The network request you want to make. Required.
+        :type request: ~azure.core.rest.HttpRequest
+        :keyword bool stream: Whether the response payload will be streamed. Defaults to False.
+        :return: The response of your network call. Does not do error handling on your response.
+        :rtype: ~azure.core.rest.HttpResponse
+        """
+
+        request_copy = deepcopy(request)
+        path_format_arguments = {
+            "endpoint": self._serialize.url("self._config.endpoint", self._config.endpoint, "str", skip_quote=True),
+            "subscriptionId": self._serialize.url("self._config.subscription_id", self._config.subscription_id, "str"),
+            "resourceGroupName": self._serialize.url(
+                "self._config.resource_group_name", self._config.resource_group_name, "str"
+            ),
+            "workspaceName": self._serialize.url("self._config.workspace_name", self._config.workspace_name, "str"),
+        }
+
+        request_copy.url = self._client.format_url(request_copy.url, **path_format_arguments)
+        return self._client.send_request(request_copy, stream=stream, **kwargs)  # type: ignore
+
+    def close(self) -> None:
+        self._client.close()
+
+    def __enter__(self) -> Self:
+        self._client.__enter__()
+        return self
+
+    def __exit__(self, *exc_details: Any) -> None:
+        self._client.__exit__(*exc_details)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient/__init__.py

@@ -0,0 +1,87 @@
+# coding=utf-8
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for license information.
+# Code generated by Microsoft (R) Python Code Generator.
+# Changes may cause incorrect behavior and will be lost if the code is regenerated.
+# --------------------------------------------------------------------------
+
+from typing import Any, TYPE_CHECKING
+
+from azure.core.pipeline import policies
+
+from ._version import VERSION
+
+if TYPE_CHECKING:
+    from azure.core.credentials import TokenCredential
+
+
+class MachineLearningServicesClientConfiguration:  # pylint: disable=too-many-instance-attributes,name-too-long
+    """Configuration for MachineLearningServicesClient.
+
+    Note that all parameters used to create this instance are saved as instance
+    attributes.
+
+    :param endpoint: Supported Azure-AI endpoints. Required.
+    :type endpoint: str
+    :param subscription_id: The ID of the target subscription. Required.
+    :type subscription_id: str
+    :param resource_group_name: The name of the Resource Group. Required.
+    :type resource_group_name: str
+    :param workspace_name: The name of the AzureML workspace or AI project. Required.
+    :type workspace_name: str
+    :param credential: Credential used to authenticate requests to the service. Required.
+    :type credential: ~azure.core.credentials.TokenCredential
+    :keyword api_version: The API version to use for this operation. Default value is
+     "2022-11-01-preview". Note that overriding this default value may result in unsupported
+     behavior.
+    :paramtype api_version: str
+    """
+
+    def __init__(
+        self,
+        endpoint: str,
+        subscription_id: str,
+        resource_group_name: str,
+        workspace_name: str,
+        credential: "TokenCredential",
+        **kwargs: Any
+    ) -> None:
+        api_version: str = kwargs.pop("api_version", "2022-11-01-preview")
+
+        if endpoint is None:
+            raise ValueError("Parameter 'endpoint' must not be None.")
+        if subscription_id is None:
+            raise ValueError("Parameter 'subscription_id' must not be None.")
+        if resource_group_name is None:
+            raise ValueError("Parameter 'resource_group_name' must not be None.")
+        if workspace_name is None:
+            raise ValueError("Parameter 'workspace_name' must not be None.")
+        if credential is None:
+            raise ValueError("Parameter 'credential' must not be None.")
+
+        self.endpoint = endpoint
+        self.subscription_id = subscription_id
+        self.resource_group_name = resource_group_name
+        self.workspace_name = workspace_name
+        self.credential = credential
+        self.api_version = api_version
+        self.credential_scopes = kwargs.pop("credential_scopes", ["https://ml.azure.com/.default"])
+        kwargs.setdefault("sdk_moniker", "rai_client/{}".format(VERSION))
+        self.polling_interval = kwargs.get("polling_interval", 30)
+        self._configure(**kwargs)
+
+    def _configure(self, **kwargs: Any) -> None:
+        self.user_agent_policy = kwargs.get("user_agent_policy") or policies.UserAgentPolicy(**kwargs)
+        self.headers_policy = kwargs.get("headers_policy") or policies.HeadersPolicy(**kwargs)
+        self.proxy_policy = kwargs.get("proxy_policy") or policies.ProxyPolicy(**kwargs)
+        self.logging_policy = kwargs.get("logging_policy") or policies.NetworkTraceLoggingPolicy(**kwargs)
+        self.http_logging_policy = kwargs.get("http_logging_policy") or policies.HttpLoggingPolicy(**kwargs)
+        self.custom_hook_policy = kwargs.get("custom_hook_policy") or policies.CustomHookPolicy(**kwargs)
+        self.redirect_policy = kwargs.get("redirect_policy") or policies.RedirectPolicy(**kwargs)
+        self.retry_policy = kwargs.get("retry_policy") or policies.RetryPolicy(**kwargs)
+        self.authentication_policy = kwargs.get("authentication_policy")
+        if self.credential and not self.authentication_policy:
+            self.authentication_policy = policies.BearerTokenCredentialPolicy(
+                self.credential, *self.credential_scopes, **kwargs
+            )