Lossless Large Language Model Acceleration via Self-Speculative Decoding

[KerfuffleV2]

Link: https://arxiv.org/abs/2309.08168

Basically this is like the existing speculative decoding stuff except it doesn't use a separate speculation model but instead runs only some of the main model's layers to generate the draft. The big advantage is the "draft" model's output will definitely be in sync with the main model and you don't need to load in a whole separate model: the existing model can be reused.

Unfortunately, they don't really include specific information about which layers to skip are optimal so that's something we'd have to find out ourselves. The first step to that might be extending the inference API to allow passing a list of the layers to run and an example that could run perplexity on various permutations of layers.

[BarfingLemurs]

Unfortunately, they don't really include specific information about which layers to skip are optimal so that's something we'd have to find out ourselves.

:( :(

couldn't we just ask paper authors?
maybe some of them are on github?

[KerfuffleV2]

Dumb as it sounds, that hadn't even occurred to me. :) I created an issue in their repo, we'll see what happens.

The need to perform our own research would still exist, but knowing one set of good/optimal results would still be extremely helpful.

[Galunid]

Repo with code (not yet available): https://github.com/dilab-zju/self-speculative-decoding

[KerfuffleV2]

I asked the authors and got a very helpful reply. See: #3565 (comment)