Skip to content

Commit 897e055

Browse files
stephentoubstephentoub
authored
Update M.E.AI READMEs (#5925)
- Add additional information to the M.E.AI.Abstractions README - Move information from the M.E.AI README into one specific to tests; the README doubles as the nuget package README, where the extra test information isn't relevant
1 parent b19cf20 commit 897e055

File tree

3 files changed

+210
-80
lines changed
  • src/Libraries
  • test/Libraries/Microsoft.Extensions.AI.Integration.Tests

3 files changed

+210
-80
lines changed

‎src/Libraries/Microsoft.Extensions.AI.Abstractions/README.md‎

Lines changed: 144 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -18,21 +18,22 @@ Or directly in the C# project file:
1818
</ItemGroup>
1919
```
2020

21+
To also have access to higher-level utilities for working with such components, instead reference the [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI)
22+
package. Libraries providing implementations of the abstractions will typically only reference `Microsoft.Extensions.AI.Abstractions`, whereas most consuming applications and services
23+
will reference the `Microsoft.Extensions.AI` package (which itself references `Microsoft.Extensions.AI.Abstractions`) along with one or more libraries that provide concrete implementations
24+
of the abstractions.
25+
2126
## Usage Examples
2227

2328
### `IChatClient`
2429

25-
The `IChatClient` interface defines a client abstraction responsible for interacting with AI services that provide chat capabilities. It defines methods for sending and receiving messages comprised of multi-modal content (text, images, audio, etc.), either as a complete set or streamed incrementally. Additionally, it allows for retrieving strongly-typed services that may be provided by the client or its underlying services.
30+
The `IChatClient` interface defines a client abstraction responsible for interacting with AI services that provide "chat" capabilities. It defines methods for sending and receiving messages comprised of multi-modal content (text, images, audio, etc.), with responses being either as a complete result or streamed incrementally. Additionally, it allows for retrieving strongly-typed services that may be provided by the client or its underlying services.
2631

2732
#### Sample Implementation
2833

2934
.NET libraries that provide clients for language models and services may provide an implementation of the `IChatClient` interface. Any consumers of the interface are then able to interoperate seamlessly with these models and services via the abstractions.
3035

31-
Here is a sample implementation of an `IChatClient` to show the general structure. You can find other concrete implementations in the following packages:
32-
33-
- [Microsoft.Extensions.AI.AzureAIInference](https://aka.ms/meai-azaiinference-nuget)
34-
- [Microsoft.Extensions.AI.OpenAI](https://aka.ms/meai-openai-nuget)
35-
- [Microsoft.Extensions.AI.Ollama](https://aka.ms/meai-ollama-nuget)
36+
Here is a sample implementation of an `IChatClient` to show the general structure.
3637

3738
```csharp
3839
using System.Runtime.CompilerServices;
@@ -99,6 +100,12 @@ public class SampleChatClient : IChatClient
99100
}
100101
```
101102

103+
As further examples, you can find other concrete implementations in the following packages (but many more such implementations for a large variety of services are available on NuGet):
104+
105+
- [Microsoft.Extensions.AI.AzureAIInference](https://aka.ms/meai-azaiinference-nuget)
106+
- [Microsoft.Extensions.AI.OpenAI](https://aka.ms/meai-openai-nuget)
107+
- [Microsoft.Extensions.AI.Ollama](https://aka.ms/meai-ollama-nuget)
108+
102109
#### Requesting a Chat Response: `GetResponseAsync`
103110

104111
With an instance of `IChatClient`, the `GetResponseAsync` method may be used to send a request and get a response. The request is composed of one or more messages, each of which is composed of one or more pieces of content. Accelerator methods exist to simplify common cases, such as constructing a request for a single piece of text content.
@@ -127,6 +134,22 @@ Console.WriteLine(await client.GetResponseAsync(
127134
]));
128135
```
129136

137+
The `ChatResponse` that's returned from `GetResponseAsync` exposes a `ChatMessage` representing the response message. It may be added back into the history in order to provide this response back to the service in a subsequent request, e.g.
138+
139+
```csharp
140+
List<ChatMessage> history = [];
141+
while (true)
142+
{
143+
Console.Write("Q: ");
144+
history.Add(new(ChatRole.User, Console.ReadLine()));
145+
146+
ChatResponse response = await client.GetResponseAsync(history);
147+
148+
Console.WriteLine(response);
149+
history.Add(response.Message);
150+
}
151+
```
152+
130153
#### Requesting a Streaming Chat Response: `GetStreamingResponseAsync`
131154

132155
The inputs to `GetStreamingResponseAsync` are identical to those of `GetResponseAsync`. However, rather than returning the complete response as part of a `ChatResponse` object, the method returns an `IAsyncEnumerable<ChatResponseUpdate>`, providing a stream of updates that together form the single response.
@@ -142,9 +165,30 @@ await foreach (var update in client.GetStreamingResponseAsync("What is AI?"))
142165
}
143166
```
144167

145-
#### Tool calling
168+
Such a stream of response updates may be combined into a single response object via the `ToChatResponse` and `ToChatResponseAsync` helper methods, e.g.
169+
170+
```csharp
171+
List<ChatMessage> history = [];
172+
List<ChatResponseUpdate> updates = [];
173+
while (true)
174+
{
175+
Console.Write("Q: ");
176+
history.Add(new(ChatRole.User, Console.ReadLine()));
177+
178+
updates.Clear();
179+
await foreach (var update in client.GetStreamingResponseAsync(history))
180+
{
181+
Console.Write(update);
182+
updates.Add(update);
183+
}
184+
185+
history.Add(updates.ToChatResponse().Message));
186+
}
187+
```
188+
189+
#### Tool Calling
146190

147-
Some models and services support the notion of tool calling, where requests may include information about tools that the model may request be invoked in order to gather additional information, in particular functions. Rather than sending back a response message that represents the final response to the input, the model sends back a request to invoke a given function with a given set of arguments; the client may then find and invoke the relevant function and send back the results to the model (along with all the rest of the history). The abstractions in Microsoft.Extensions.AI include representations for various forms of content that may be included in messages, and this includes representations for these function call requests and results. While it's possible for the consumer of the `IChatClient` to interact with this content directly, `Microsoft.Extensions.AI` supports automating these interactions. It provides an `AIFunction` that represents an invocable function along with metadata for describing the function to the AI model, along with an `AIFunctionFactory` for creating `AIFunction`s to represent .NET methods. It also provides a `FunctionInvokingChatClient` that both is an `IChatClient` and also wraps an `IChatClient`, enabling layering automatic function invocation capabilities around an arbitrary `IChatClient` implementation.
191+
Some models and services support the notion of tool calling, where requests may include information about tools (in particular .NET methods) that the model may request be invoked in order to gather additional information. Rather than sending back a response message that represents the final response to the input, the model sends back a request to invoke a given function with a given set of arguments; the client may then find and invoke the relevant function and send back the results to the model (along with all the rest of the history). The abstractions in Microsoft.Extensions.AI include representations for various forms of content that may be included in messages, and this includes representations for these function call requests and results. While it's possible for the consumer of the `IChatClient` to interact with this content directly, `Microsoft.Extensions.AI` supports automating these interactions. It provides an `AIFunction` that represents an invocable function along with metadata for describing the function to the AI model, along with an `AIFunctionFactory` for creating `AIFunction`s to represent .NET methods. It also provides a `FunctionInvokingChatClient` that both is an `IChatClient` and also wraps an `IChatClient`, enabling layering automatic function invocation capabilities around an arbitrary `IChatClient` implementation.
148192

149193
```csharp
150194
using System.ComponentModel;
@@ -215,6 +259,8 @@ IChatClient client = new ChatClientBuilder(new SampleChatClient(new Uri("http://
215259
Console.WriteLine((await client.GetResponseAsync("What is AI?")).Message);
216260
```
217261

262+
Alternatively, the `LoggingChatClient` and corresponding `UseLogging` method provide a simple way to write log entries to an `ILogger` for every request and response.
263+
218264
#### Options
219265

220266
Every call to `GetResponseAsync` or `GetStreamingResponseAsync` may optionally supply a `ChatOptions` instance containing additional parameters for the operation. The most common parameters that are common amongst AI models and services show up as strongly-typed properties on the type, such as `ChatOptions.Temperature`. Other parameters may be supplied by name in a weakly-typed manner via the `ChatOptions.AdditionalProperties` dictionary.
@@ -231,7 +277,7 @@ Console.WriteLine(await client.GetResponseAsync("What is AI?")); // will request
231277
Console.WriteLine(await client.GetResponseAsync("What is AI?", new() { ModelId = "llama3.1" })); // will request "llama3.1"
232278
```
233279

234-
#### Pipelines of Functionality
280+
#### Pipelines of Chat Functionality
235281

236282
All of these `IChatClient`s may be layered, creating a pipeline of any number of components that all add additional functionality. Such components may come from `Microsoft.Extensions.AI`, may come from other NuGet packages, or may be your own custom implementations that augment the behavior in whatever ways you need.
237283

@@ -249,7 +295,7 @@ var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder()
249295
.AddConsoleExporter()
250296
.Build();
251297

252-
// Explore changing the order of the intermediate "Use" calls to see that impact
298+
// Explore changing the order of the intermediate "Use" calls to see the impact
253299
// that has on what gets cached, traced, etc.
254300
IChatClient client = new ChatClientBuilder(new OllamaChatClient(new Uri("http://localhost:11434"), "llama3.1"))
255301
.UseDistributedCache(new MemoryDistributedCache(Options.Create(new MemoryDistributedCacheOptions())))
@@ -408,7 +454,7 @@ another overload of `Use` exists that accepts a delegate for each.
408454

409455
#### Dependency Injection
410456

411-
`IChatClient` implementations will typically be provided to an application via dependency injection (DI). In this example, an `IDistributedCache` is added into the DI container, as is an `IChatClient`. The registration for the `IChatClient` employs a builder that creates a pipeline containing a caching client (which will then use an `IDistributedCache` retrieved from DI) and the sample client. Elsewhere in the app, the injected `IChatClient` may be retrieved and used.
457+
While not required, `IChatClient` implementations will often be provided to an application via dependency injection (DI). In this example, an `IDistributedCache` is added into the DI container, as is an `IChatClient`. The registration for the `IChatClient` employs a builder that creates a pipeline containing a caching client (which will then use an `IDistributedCache` retrieved from DI) and the sample client. Elsewhere in the app, the injected `IChatClient` may be retrieved and used.
412458

413459
```csharp
414460
using Microsoft.Extensions.AI;
@@ -429,6 +475,81 @@ Console.WriteLine(await chatClient.GetResponseAsync("What is AI?"));
429475

430476
What instance and configuration is injected may differ based on the current needs of the application, and multiple pipelines may be injected with different keys.
431477

478+
#### Stateless vs Stateful Clients
479+
480+
"Stateless" services require all relevant conversation history to sent back on every request, while "stateful" services keep track of the history and instead
481+
require only additional messages be sent with a request. The `IChatClient` interface is designed to handle both stateless and stateful AI services.
482+
483+
If you know you're working with a stateless service (currently the most common form), responses may be added back into a message history for sending back to the server.
484+
```csharp
485+
List<ChatMessage> history = [];
486+
while (true)
487+
{
488+
Console.Write("Q: ");
489+
history.Add(new(ChatRole.User, Console.ReadLine()));
490+
491+
ChatResponse response = await client.GetResponseAsync(history);
492+
493+
Console.WriteLine(response);
494+
history.Add(response.Message);
495+
}
496+
```
497+
498+
For stateful services, you may know ahead of time an identifier used for the relevant conversation. That identifier can be put into `ChatOptions.ChatThreadId`:
499+
```csharp
500+
ChatOptions options = new() { ChatThreadId = "my-conversation-id" };
501+
while (true)
502+
{
503+
Console.Write("Q: ");
504+
505+
ChatResponse response = await client.GetResponseAsync(Console.ReadLine(), options);
506+
507+
Console.WriteLine(response);
508+
}
509+
```
510+
511+
Some services may support automatically creating a thread ID for a request that doesn't have one. In such cases, you can transfer the `ChatResponse.ChatThreadId` over
512+
to the `ChatOptions.ChatThreadId` for subsequent requests, e.g.
513+
```csharp
514+
ChatOptions options = new();
515+
while (true)
516+
{
517+
Console.Write("Q: ");
518+
519+
ChatResponse response = await client.GetResponseAsync(Console.ReadLine(), options);
520+
521+
Console.WriteLine(response);
522+
options.ChatThreadId = response.ChatThreadId;
523+
}
524+
```
525+
526+
If you don't know ahead of time whether the service is stateless or stateful, both can be accomodated by checking the response `ChatThreadId`
527+
and acting based on its value. Here, if the response `ChatThreadId` is set, then that value is propagated to the options and the history
528+
cleared so as to not resend the same history again. If, however, the `ChatThreadId` is not set, then the response message is added to the
529+
history so that it's sent back to the service on the next turn.
530+
```csharp
531+
List<ChatMessage> history = [];
532+
ChatOptions options = new();
533+
while (true)
534+
{
535+
Console.Write("Q: ");
536+
history.Add(new(ChatRole.User, Console.ReadLine()));
537+
538+
ChatResponse response = await client.GetResponseAsync(history);
539+
540+
Console.WriteLine(response);
541+
options.ChatThreadId = response.ChatThreadId;
542+
if (response.ChatThreadId is not null)
543+
{
544+
history.Clear();
545+
}
546+
else
547+
{
548+
history.Add(response.Message);
549+
}
550+
}
551+
```
552+
432553
### IEmbeddingGenerator
433554

434555
The `IEmbeddingGenerator<TInput,TEmbeddding>` interface represents a generic generator of embeddings, where `TInput` is the type of input values being embedded and `TEmbedding` is the type of generated embedding, inheriting from `Embedding`.
@@ -476,7 +597,7 @@ public class SampleEmbeddingGenerator(Uri endpoint, string modelId) : IEmbedding
476597
}
477598
```
478599

479-
#### Creating an embedding: `GenerateAsync`
600+
#### Creating an Embedding: `GenerateAsync`
480601

481602
The primary operation performed with an `IEmbeddingGenerator` is generating embeddings, which is accomplished with its `GenerateAsync` method.
482603

@@ -492,7 +613,17 @@ foreach (var embedding in await generator.GenerateAsync(["What is AI?", "What is
492613
}
493614
```
494615

495-
#### Middleware
616+
Accelerator extension methods also exist to simplify common cases, such as generating an embedding vector from a single input, e.g.
617+
```csharp
618+
using Microsoft.Extensions.AI;
619+
620+
IEmbeddingGenerator<string, Embedding<float>> generator =
621+
new SampleEmbeddingGenerator(new Uri("http://coolsite.ai"), "my-custom-model");
622+
623+
ReadOnlyMemory<float> vector = generator.GenerateEmbeddingVectorAsync("What is AI?");
624+
```
625+
626+
#### Pipelines of Functionality
496627

497628
As with `IChatClient`, `IEmbeddingGenerator` implementations may be layered. Just as `Microsoft.Extensions.AI` provides delegating implementations of `IChatClient` for caching and telemetry, it does so for `IEmbeddingGenerator` as well.
498629

‎src/Libraries/Microsoft.Extensions.AI/README.md‎

Lines changed: 0 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -25,70 +25,3 @@ Please refer to the [README](https://www.nuget.org/packages/Microsoft.Extensions
2525
## Feedback & Contributing
2626

2727
We welcome feedback and contributions in [our GitHub repo](https://github.com/dotnet/extensions).
28-
29-
## Running the integration tests
30-
31-
If you're working on this repo and want to run the integration tests, e.g., those in `Microsoft.Extensions.AI.OpenAI.Tests`, you must first set endpoints and keys. You can either set these as environment variables or - better - using .NET's user secrets feature as shown below.
32-
33-
### Configuring OpenAI tests (OpenAI)
34-
35-
Run commands like the following. The settings will be saved in your user profile.
36-
37-
```
38-
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
39-
dotnet user-secrets set OpenAI:Mode OpenAI
40-
dotnet user-secrets set OpenAI:Key abcdefghijkl
41-
```
42-
43-
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
44-
45-
```
46-
dotnet user-secrets set OpenAI:ChatModel gpt-4o-mini
47-
dotnet user-secrets set OpenAI:EmbeddingModel text-embedding-3-small
48-
```
49-
50-
### Configuring OpenAI tests (Azure OpenAI)
51-
52-
Run commands like the following. The settings will be saved in your user profile.
53-
54-
```
55-
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
56-
dotnet user-secrets set OpenAI:Mode AzureOpenAI
57-
dotnet user-secrets set OpenAI:Endpoint https://YOUR_DEPLOYMENT.openai.azure.com/
58-
dotnet user-secrets set OpenAI:Key abcdefghijkl
59-
```
60-
61-
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
62-
63-
```
64-
dotnet user-secrets set OpenAI:ChatModel gpt-4o-mini
65-
dotnet user-secrets set OpenAI:EmbeddingModel text-embedding-3-small
66-
```
67-
68-
Your account must have models matching these names.
69-
70-
### Configuring Azure AI Inference tests
71-
72-
Run commands like the following. The settings will be saved in your user profile.
73-
74-
```
75-
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
76-
dotnet user-secrets set AzureAIInference:Endpoint https://YOUR_DEPLOYMENT.azure.com/
77-
dotnet user-secrets set AzureAIInference:Key abcdefghijkl
78-
```
79-
80-
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
81-
82-
```
83-
dotnet user-secrets set AzureAIInference:ChatModel gpt-4o-mini
84-
dotnet user-secrets set AzureAIInference:EmbeddingModel text-embedding-3-small
85-
```
86-
87-
### Configuring Ollama tests
88-
89-
Run commands like the following. The settings will be saved in your user profile.
90-
91-
```
92-
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
93-
dotnet user-secrets set Ollama:Endpoint http://localhost:11434/
94-
```

‎test/Libraries/Microsoft.Extensions.AI.Integration.Tests/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
## Running the integration tests
2+
3+
To run the integration tests, e.g., those in `Microsoft.Extensions.AI.OpenAI.Tests`, you must first set endpoints and keys. You can either set these as environment variables or - better - using .NET's user secrets feature as shown below.
4+
5+
### Configuring OpenAI tests (OpenAI)
6+
7+
Run commands like the following. The settings will be saved in your user profile.
8+
9+
```
10+
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
11+
dotnet user-secrets set OpenAI:Mode OpenAI
12+
dotnet user-secrets set OpenAI:Key abcdefghijkl
13+
```
14+
15+
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
16+
17+
```
18+
dotnet user-secrets set OpenAI:ChatModel gpt-4o-mini
19+
dotnet user-secrets set OpenAI:EmbeddingModel text-embedding-3-small
20+
```
21+
22+
### Configuring OpenAI tests (Azure OpenAI)
23+
24+
Run commands like the following. The settings will be saved in your user profile.
25+
26+
```
27+
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
28+
dotnet user-secrets set OpenAI:Mode AzureOpenAI
29+
dotnet user-secrets set OpenAI:Endpoint https://YOUR_DEPLOYMENT.openai.azure.com/
30+
dotnet user-secrets set OpenAI:Key abcdefghijkl
31+
```
32+
33+
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
34+
35+
```
36+
dotnet user-secrets set OpenAI:ChatModel gpt-4o-mini
37+
dotnet user-secrets set OpenAI:EmbeddingModel text-embedding-3-small
38+
```
39+
40+
Your account must have models matching these names.
41+
42+
### Configuring Azure AI Inference tests
43+
44+
Run commands like the following. The settings will be saved in your user profile.
45+
46+
```
47+
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
48+
dotnet user-secrets set AzureAIInference:Endpoint https://YOUR_DEPLOYMENT.azure.com/
49+
dotnet user-secrets set AzureAIInference:Key abcdefghijkl
50+
```
51+
52+
Optionally also run the following. The values shown here are the defaults if you don't specify otherwise:
53+
54+
```
55+
dotnet user-secrets set AzureAIInference:ChatModel gpt-4o-mini
56+
dotnet user-secrets set AzureAIInference:EmbeddingModel text-embedding-3-small
57+
```
58+
59+
### Configuring Ollama tests
60+
61+
Run commands like the following. The settings will be saved in your user profile.
62+
63+
```
64+
cd test/Libraries/Microsoft.Extensions.AI.Integration.Tests
65+
dotnet user-secrets set Ollama:Endpoint http://localhost:11434/
66+
```

0 commit comments

Comments
 (0)