Add PSReadlineOption 'PredictionTimeout' for configurable Predictive Intellisense timeout

[JustinGrote]

Prerequisites

• Write a descriptive title.

Description of the new feature/enhancement

As a PowerShell User
I want to receive prediction results from external sources like large language models (ChatGPT/Copilot)
And that may take longer than 20 milliseconds

Since the original design of Predictive Intellisense, the world has changed in the form of LLMs like Github CoPilot that can predict code for you based on a prompt. However, the round trip to perform this action will likely be in the several hundred ms if not multiple seconds, which is more than the default 20ms in the PowerShell CommandPrediction subsystem.
https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/Subsystem/PredictionSubsystem/CommandPrediction.cs#L75

That default is configurable however, so empower users with the option to choose to extend the timeout, understanding they are trading "slower" prompt responses for potentially more detailed ones.

I ran into this while trying to write a copilot predictor, thanks @SeeminglyScience for the heads up.

CC @daxian-dbw

Proposed technical implementation details (optional)

Set-PSReadlineOption -PredictionTimeout 5000

If this option is set, it will call the PredictInputAsync method with the overload that allows specifying a timeout.

[the-mentor]

+1

[daxian-dbw]

For the current design, the new input from user typing is not rendered until after the prediction is done. So, simply increasing the timeout means the user will likely see noticeable lagging when typing, which is a bad experience we want to avoid.

I think what we truly want is the asynchronous rendering of the prediction results, in a way that doesn't affect the user typing, similar to how intellisense works in VSCode editor (see #3581 (comment)). I will treat this issue as a duplicate of #3581.

[JustinGrote]

@daxian-dbw understood, however since the prediction timeout is an option, the user opts into it. By not changing this setting, the user gets the default experience, however this enables those who are willing to wait the ability do so.

I agree that a larger scoped feature that optimizes the typing experience for fast cancel of prediction and perhaps a minimum time threshold for typing for resolving should be present, but does that preclude this feature from being merged? What does it hurt?

[daxian-dbw]

The 20ms timeout used in PSReadLine was intentionally chosen to not cause noticeable lagging in user typing. That was a compromise based on the assumption at the time that AI models will be running locally, such as the Az Predictor.

Given that we will be investigating and eventually investing in the asynchronous rendering, which is likely a big design change, I don't think it's a good idea to expose a sort-of broken interactive experience by making the timeout configurable. The lagging in typing when working with online AI services will make it not really usable.

[JustinGrote]

@daxian-dbw thank you, I understand your position and while I disagree, I appreciate the considerations you need to make. I just hope we can capitalize on the momentum of AI assistants and make PowerShell a first class citizen in that realm and not miss the boat. Thanks!

[daxian-dbw]

Thanks for your understanding. That's exactly what I am hoping for too!