how to achieve the Guidance acceleration

[xujli]

I just want to know how do you achieve the Guidance acceleration in transformers, only cache? Could you point out the source code I'd like to appreciate it.

[slundberg]

It works by sending things in batch to the compute backend when we can. How this batching happens is specific to the backend but the if statement that detect if the next token is forced (and can so be batched) is at:

guidance/guidance/models/_model.py

Line 731 in 26554b0

if is_forced:

hope that helps!

[xujli]

If the next token is forced, the backend will directly return the token ID instead of get_logits, thanks for your reply. Thus, the actual acceleration is the parallel attention score calculation when we feed multiple new tokens caused by the template and avoid the logits processing for the template tokens, Am I right?