From fd490fe4a4e0f8f665856169bb5365b9dfdd7ade Mon Sep 17 00:00:00 2001 From: Domenic Denicola Date: Mon, 21 Apr 2025 16:10:25 +0900 Subject: [PATCH 1/4] Remove append() aborting and add more detail on append See discussions in https://github.com/webmachinelearning/prompt-api/issues/92#issuecomment-2811773119 onward. --- README.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 40cbb75..6f66739 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,10 @@ analyzeButton.onclick = async (e) => { }; ``` +The promise returned by `append()` will reject if the prompt cannot be appended (e.g., too big, invalid modalities for the session, etc.), or will fulfill once the prompt has been validated, processed, and appended to the session. + +Note that `append()` can also cause [overflow](#tokenization-context-window-length-limits-and-overflow), in which case it will evict the oldest non-system prompts from the session and fire the `"quotaoverflow"` event. + ### Configuration of per-session parameters In addition to the `systemPrompt` and `initialPrompts` options shown above, the currently-configurable model parameters are [temperature](https://huggingface.co/blog/how-to-generate#sampling) and [top-K](https://huggingface.co/blog/how-to-generate#top-k-sampling). The `params()` API gives the default and maximum values for these parameters. @@ -429,7 +433,7 @@ The ability to manually destroy a session allows applications to free up memory ### Aborting a specific prompt -Specific calls to `prompt()`, `promptStreaming()`, or `append()` can be aborted by passing an `AbortSignal` to them: +Specific calls to `prompt()` or `promptStreaming()` can be aborted by passing an `AbortSignal` to them: ```js const controller = new AbortController(); @@ -440,11 +444,11 @@ const result = await session.prompt("Write me a poem", { signal: controller.sign Note that because sessions are stateful, and prompts can be queued, aborting a specific prompt is slightly complicated: -* If the prompt is still queued behind other prompts in the session, then it will be removed from the queue. -* If the prompt is being currently responded to by the model, then it will be aborted, and the prompt/response pair will be removed from the conversation history. +* If the prompt is still queued behind other prompts in the session, then it will be removed from the queue, and the returned promise will be rejected with an `"AbortError"` `DOMException`. +* If the prompt is being currently responded to by the model, then it will be aborted, the prompt/response pair will be removed from the conversation history, and the returned promise will be rejected with an `"AbortError"` `DOMException`. * If the prompt has already been fully responded to by the model, then attempting to abort the prompt will do nothing. -Since `append()`ed prompts are not responded to immediately, they can be aborted until a subsequent call to `prompt()` or `promptStreaming()` happens and that response has been finished. +(Note: `append()` calls cannot currently be aborted. We could consider adding that in the future, if people have strong use cases. But see [this discussion](https://github.com/webmachinelearning/prompt-api/issues/92#issuecomment-2811773119) for all the complexities involved in designing such a system.) ### Tokenization, context window length limits, and overflow @@ -616,10 +620,7 @@ interface LanguageModel : EventTarget { LanguageModelPrompt input, optional LanguageModelPromptOptions options = {} ); - Promise append( - LanguageModelPrompt input, - optional LanguageModelAppendOptions options = {} - ); + Promise append(LanguageModelPrompt input); Promise measureInputUsage( LanguageModelPrompt input, @@ -671,10 +672,6 @@ dictionary LanguageModelPromptOptions { AbortSignal signal; }; -dictionary LanguageModelAppendOptions { - AbortSignal signal; -}; - dictionary LanguageModelCloneOptions { AbortSignal signal; }; From c2783ad7ed6e3e6da4d9d2fe66c993c52c82d79a Mon Sep 17 00:00:00 2001 From: Domenic Denicola Date: Tue, 22 Apr 2025 15:22:13 +0900 Subject: [PATCH 2/4] Actually, allow aborting, but with specific semantics --- README.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6f66739..5f05403 100644 --- a/README.md +++ b/README.md @@ -445,10 +445,17 @@ const result = await session.prompt("Write me a poem", { signal: controller.sign Note that because sessions are stateful, and prompts can be queued, aborting a specific prompt is slightly complicated: * If the prompt is still queued behind other prompts in the session, then it will be removed from the queue, and the returned promise will be rejected with an `"AbortError"` `DOMException`. -* If the prompt is being currently responded to by the model, then it will be aborted, the prompt/response pair will be removed from the conversation history, and the returned promise will be rejected with an `"AbortError"` `DOMException`. +* If the prompt is being currently responded to by the model, then it will be aborted, the prompt/response pair will be removed from the session, and the returned promise will be rejected with an `"AbortError"` `DOMException`. * If the prompt has already been fully responded to by the model, then attempting to abort the prompt will do nothing. -(Note: `append()` calls cannot currently be aborted. We could consider adding that in the future, if people have strong use cases. But see [this discussion](https://github.com/webmachinelearning/prompt-api/issues/92#issuecomment-2811773119) for all the complexities involved in designing such a system.) +Similarly, the `append()` operation can also be aborted. In this case the behavior is: + +* If the append is queued behind other appends in the session, then it will be removed from the queue, and the returned promise will be rejected with an `"AbortError"` `DOMException`. +* If the append operation is currently ongoing, then it will be aborted, any part of the prompt that was appended so far will be removed from the session, and the returned promise will be rejected with an `"AbortError"` `DOMException`. +* If the append operation is complete, then attempting to abort it will do nothing. This includes all the following states: + * The append operation is complete, but another append operation is processing. + * The append operation is complete, and a prompt generation step is processing. + * The append operation is complete, and a prompt generation step has used it to produce a result. ### Tokenization, context window length limits, and overflow @@ -620,7 +627,10 @@ interface LanguageModel : EventTarget { LanguageModelPrompt input, optional LanguageModelPromptOptions options = {} ); - Promise append(LanguageModelPrompt input); + Promise append( + LanguageModelPrompt input, + optional LanguageModelAppendOptions options = {} + ); Promise measureInputUsage( LanguageModelPrompt input, @@ -672,6 +682,10 @@ dictionary LanguageModelPromptOptions { AbortSignal signal; }; +dictionary LanguageModelAppendOptions { + AbortSignal signal; +}; + dictionary LanguageModelCloneOptions { AbortSignal signal; }; From 725f66df7af5d00bace88a95ba746ac2bf552ab5 Mon Sep 17 00:00:00 2001 From: Domenic Denicola Date: Tue, 22 Apr 2025 15:24:04 +0900 Subject: [PATCH 3/4] Note on overflow --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5f05403..91b9926 100644 --- a/README.md +++ b/README.md @@ -452,11 +452,13 @@ Similarly, the `append()` operation can also be aborted. In this case the behavi * If the append is queued behind other appends in the session, then it will be removed from the queue, and the returned promise will be rejected with an `"AbortError"` `DOMException`. * If the append operation is currently ongoing, then it will be aborted, any part of the prompt that was appended so far will be removed from the session, and the returned promise will be rejected with an `"AbortError"` `DOMException`. -* If the append operation is complete, then attempting to abort it will do nothing. This includes all the following states: +* If the append operation is complete (i.e., the returned promise has resolved), then attempting to abort it will do nothing. This includes all the following states: * The append operation is complete, but another append operation is processing. * The append operation is complete, and a prompt generation step is processing. * The append operation is complete, and a prompt generation step has used it to produce a result. +Finally, note that if either prompting or appending has caused an [overflow](#tokenization-context-window-length-limits-and-overflow), aborting the operation does not re-introduce the overflowed messages into the session. + ### Tokenization, context window length limits, and overflow A given language model session will have a maximum number of tokens it can process. Developers can check their current usage and progress toward that limit by using the following properties on the session object: From 7eebeba8700a5b9616e7afef2176ddb48fca4bba Mon Sep 17 00:00:00 2001 From: Domenic Denicola Date: Thu, 24 Apr 2025 11:01:48 +0900 Subject: [PATCH 4/4] Revise per lozy219's suggestion --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 91b9926..8b49e54 100644 --- a/README.md +++ b/README.md @@ -453,7 +453,7 @@ Similarly, the `append()` operation can also be aborted. In this case the behavi * If the append is queued behind other appends in the session, then it will be removed from the queue, and the returned promise will be rejected with an `"AbortError"` `DOMException`. * If the append operation is currently ongoing, then it will be aborted, any part of the prompt that was appended so far will be removed from the session, and the returned promise will be rejected with an `"AbortError"` `DOMException`. * If the append operation is complete (i.e., the returned promise has resolved), then attempting to abort it will do nothing. This includes all the following states: - * The append operation is complete, but another append operation is processing. + * The append operation is complete, but a prompt generation step has not yet triggered. * The append operation is complete, and a prompt generation step is processing. * The append operation is complete, and a prompt generation step has used it to produce a result.