Usage examples.

[murrellb]

Checking equivalence to HuggingFace transformers:

using HuggingFaceTransformers, Jjama3, JSON3

path = "/localpathto/Qwen2.5-Coder-0.5B-Instruct"

config = JSON3.read(read(path*"/config.json", String));
Jmodel = load_llama3_from_safetensors(path*"/model.safetensors", config);

model = from_pretrained(CausalLM, path);
tokenizer = from_pretrained(AutoTokenizer, path)
prompt = "Explain, in as much detail as possible, the Julia ecosystem for deep learning."

messages = [
    Dict("role" => "system", "content" => "You are Bob, a helpful assistant."),
    Dict("role" => "user", "content" => prompt)
]

toks = apply_chat_template(
    tokenizer,
    messages,
    add_generation_prompt=true
)

gen = HuggingFaceTransformers.generate(
    model,
    toks,
    max_length=100,
    do_sample=false,
    repetition_penalty=1.0 #Very important for equivalent sampling, because Jjama3 doesn't have this yet.
);

jjamagen = Jjama3.generate(Jmodel, toks .+ 1,
        max_new_tokens=100-length(toks),
        end_token = HuggingFaceTransformers.encode(tokenizer, "<|im_end|>")[1]+1);

println(HuggingFaceTransformers.decode(tokenizer, gen));
println(HuggingFaceTransformers.decode(tokenizer, jjamagen .- 1));

(gen .+ 1) == jjamagen