Some method of re-warming up an unloaded session

[domenic]

Currently in Chromium, any created LanguageModelSession stays alive until it is explicitly destroyed, or the page creating it is unloaded, or similar.

However, we're likely going to add some mechanism that auto-unloads the session after some period of inactivity.

This could lead to the following workflow:

• User opens page which uses the prompt API, which creates a LanguageModelSession and uses it.
• The user happily continues engaging with the page, which calls prompt(), and each call gets relatively fast responses.
• User walks away from their computer for an hour.
• User re-engages with the page, which calls prompt(). But this time, the prompt() call takes much longer, as it has to re-load the session into memory.

We're considering adding some new method, e.g. warmup(), which sites could use to restore an unloaded session into memory ahead of time.

Note that calling this method right after session creation, or right before calling prompt(), is useless. It's only useful if there's a third point in time, when the developer knows that a prompt will happen soon but not immediately, that this method would be useful. Then, the developer can call warmup() at that time, and the browser can use the time between warmup() and prompt() to do work ahead of time that would otherwise delay responding to the prompt. An example of such a time would be if the user focuses a textbox in a chat interface.

(This method may be similar to Apple's Foundation Models API's prewarm() method, and maybe we should just reeuse that name. However, the docs for that are a bit unclear, as it says "Consider calling this method when you need to immediately use the session", which seems like it's not aligned with what we're envisioning here. If you're going to immediately use the session, then you can just call prompt() and the reloading of the session into memory will happen as part of that.)

[Thomas101]

How would this work in a multi-tab environment? I assume at the moment there's only ever one "active" session in memory at any given time. If one tab is idle and has a cold session, whilst another has a hot session, is there a case here where the cold session calling warmup() could steal the slot and cause the hot one to become cold?

Would it be better to emit some event from the session when it gets unloaded from memory, then devs could either clone it or use the proposed warmup call?