Custom Sampling Pipelines #348
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…her options with an entirely custom pipeline. - Added a `Sample` method to `LLamaContext` which uses a custom pipeline - Modified all executors to use the custom pipeline if it exists
- Added an example usage to one of the tests
…better written in code. - Added BaseSamplingPipeline which provides a base impl of `ISamplingPipeline` - Added `DefaultSamplingPipeline` which mimics normal llama.cpp sampling
AsakusaRinne
approved these changes
Dec 9, 2023
|
That reminds me of something related that I thought about and I might come back to in a future PR. At the moment the LLamaContext basically has several different modes, depending on what parameters you pass:
So using this new pipeline system we could explicitly split it into 4 separate pipelines, construct them inside LLamaContext and use them as necessary. Would split some of the code out of LLamaContext into a more re-usable form. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduced an entirely new way to sample models.
The current mechanism is hardcoded (see here) and can only be configured by tweaking values in InferenceParams.
This means a couple of things are not possible:
This PR introduces:
ISamplingPipeline. If theSamplingPipelineproperty in the inference params is non-null this pipeline will be used.BaseSamplingPipelineis an abstract implementation ifISamplingPipelinewhich makes it a little easier to implement.DefualtSamplingPipelinewhich is a demo implementation that does standard sampling.You can see an example of it in use here: https://github.com/SciSharp/LLamaSharp/pull/348/files#diff-16c496b3d63b9606d57c5d2a5592059223a9e4bc7ab29416ea775d5356888df7R37