Implement speculative decoding

[wsxiaoys]

Code location: https://github.com/TabbyML/tabby/blob/main/crates/llama-cpp-bindings/src/engine.cc
Reference: https://github.com/ggerganov/llama.cpp/blob/master/examples/speculative/speculative.cpp#L47

Implement speculative decoding to speed up certain models.

[Squadrick]

I've started working on this. I'll have a draft CL with the implementation to make sure I have the logic right. Might need some help on what the interface changes to TextInferenceEngine will look like to make this possible.

Also, right now, the decoding in entirely greedy, should I continue to use greedy decoding for the speculative model as well?

[wsxiaoys]

Thanks for claiming the feature!

Also, right now, the decoding in entirely greedy, should I continue to use greedy decoding for the speculative model as well?

Yes - greedy decoding shall be good for now