attention weights for attention entropy

[EdwardDali]

is there a way to expose attention weights? the purpose is to calculate attention entropy in order to experiment with different sampling strategies. thanks!

[iamlemec]

You would need to use some ggml_* functions search through the graph nodes to find the last tensor named "kq". However, correct me if I'm wrong, but I don't think the graph object is meaningfully exposed right now. Here's an example of how I do it in gadget: gadget/compute.py#L136.

[EdwardDali]

do you have any GGUF model already created with attention heads explosed? I want to do some tests for https://github.com/EdwardDali/EntropixLab
Preferably small model, 0.5B, 1B

[iamlemec]

It doesn't depend on the GGUF, it actually depends on the inference implementation. With llama.cpp the internal compute graph will produce a tensor named "kq" that has the attention scores. This graph is different from the contents of the Llama class that I think you've show below. It's an internal object that isn't exposed to the Python interface, so it's pretty difficult to access via regular means.

@abetlen has anyone expressed interest in being able to view internally generated tensors in the compute graph? Just wondering if this is a thing worth implementing.

[EdwardDali]

if xjdr entropix proves to be a thing this will go hot

[EdwardDali]

Exploring model structure to find GGML graph...
_Llama__backend_initialized: <class 'bool'>
_batch: <class 'llama_cpp._internals.LlamaBatch'>
_c_tensor_split: <class 'NoneType'>
_candidates: <class 'llama_cpp._internals.LlamaTokenDataArray'>
_chat_handlers: <class 'dict'>
_create_completion: <class 'method'>
_ctx: <class 'llama_cpp._internals.LlamaContext'>
_init_sampler: <class 'method'>
_input_ids: <class 'numpy.ndarray'>
_lora_adapter: <class 'NoneType'>
_mirostat_mu: <class 'ctypes.c_float'>
_model: <class 'llama_cpp._internals.LlamaModel'>
_n_ctx: <class 'int'>
_n_vocab: <class 'int'>
_rpc_servers: <class 'NoneType'>
_sampler: <class 'llama_cpp._internals.LlamaSampler'>
_scores: <class 'numpy.ndarray'>
_seed: <class 'int'>
_stack: <class 'contextlib.ExitStack'>
_token_eos: <class 'int'>
_token_nl: <class 'int'>
cache: <class 'NoneType'>
chat_format: <class 'str'>
chat_handler: <class 'NoneType'>
close: <class 'method'>
context_params: <class 'llama_cpp.llama_cpp.llama_context_params'>
b_base: <class 'NoneType'>
b_needsfree: <class 'int'>
fields: <class 'list'>
_objects: <class 'NoneType'>
abort_callback: <class 'ctypes.CFUNCTYPE..CFunctionType'>
argtypes: <class 'tuple'>
b_base: <class 'llama_cpp.llama_cpp.llama_context_params'>
b_needsfree: <class 'int'>
flags: <class 'int'>
_objects: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
argtypes: <class 'tuple'>
errcheck: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
b_base: <class 'member_descriptor'>
b_needsfree: <class 'member_descriptor'>
_objects: <class 'member_descriptor'>
type: <class 'str'>
value: <class 'getset_descriptor'>
abort_callback_data: <class 'NoneType'>
attention_type: <class 'int'>
cb_eval: <class 'ctypes.CFUNCTYPE..CFunctionType'>
argtypes: <class 'tuple'>
b_base: <class 'llama_cpp.llama_cpp.llama_context_params'>
b_needsfree: <class 'int'>
flags: <class 'int'>
_objects: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
argtypes: <class 'tuple'>
errcheck: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
b_base: <class 'member_descriptor'>
b_needsfree: <class 'member_descriptor'>
_objects: <class 'member_descriptor'>
type: <class 'str'>
value: <class 'getset_descriptor'>
cb_eval_user_data: <class 'NoneType'>
defrag_thold: <class 'float'>
embeddings: <class 'bool'>
flash_attn: <class 'bool'>
logits_all: <class 'bool'>
n_batch: <class 'int'>
n_ctx: <class 'int'>
n_seq_max: <class 'int'>
n_threads: <class 'int'>
n_threads_batch: <class 'int'>
n_ubatch: <class 'int'>
offload_kqv: <class 'bool'>
pooling_type: <class 'int'>
rope_freq_base: <class 'float'>
rope_freq_scale: <class 'float'>
rope_scaling_type: <class 'int'>
type_k: <class 'int'>
type_v: <class 'int'>
yarn_attn_factor: <class 'float'>
yarn_beta_fast: <class 'float'>
yarn_beta_slow: <class 'float'>
yarn_ext_factor: <class 'float'>
yarn_orig_ctx: <class 'int'>
create_chat_completion: <class 'method'>
create_chat_completion_openai_v1: <class 'method'>
create_completion: <class 'method'>
create_embedding: <class 'method'>
ctx: <class 'int'>
detokenize: <class 'method'>
draft_model: <class 'NoneType'>
embed: <class 'method'>
eval: <class 'method'>
eval_logits: <class 'collections.deque'>
eval_tokens: <class 'collections.deque'>
from_pretrained: <class 'method'>
generate: <class 'method'>
input_ids: <class 'numpy.ndarray'>
kv_overrides: <class 'NoneType'>
last_n_tokens_size: <class 'int'>
load_state: <class 'method'>
logits_to_logprobs: <class 'function'>
longest_token_prefix: <class 'function'>
lora_base: <class 'NoneType'>
lora_path: <class 'NoneType'>
lora_scale: <class 'float'>
metadata: <class 'dict'>
model: <class 'int'>
model_params: <class 'llama_cpp.llama_cpp.llama_model_params'>
b_base: <class 'NoneType'>
b_needsfree: <class 'int'>
fields: <class 'list'>
_objects: <class 'NoneType'>
check_tensors: <class 'bool'>
kv_overrides: <class 'llama_cpp.llama_cpp.LP_llama_model_kv_override'>
b_base: <class 'llama_cpp.llama_cpp.llama_model_params'>
b_needsfree: <class 'int'>
_objects: <class 'NoneType'>
type: <class '_ctypes.PyCStructType'>
contents: <Error: NULL pointer access>
main_gpu: <class 'int'>
n_gpu_layers: <class 'int'>
progress_callback: <class 'ctypes.CFUNCTYPE..CFunctionType'>
argtypes: <class 'tuple'>
b_base: <class 'llama_cpp.llama_cpp.llama_model_params'>
b_needsfree: <class 'int'>
flags: <class 'int'>
_objects: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
argtypes: <class 'tuple'>
errcheck: <class 'NoneType'>
restype: <class '_ctypes.PyCSimpleType'>
b_base: <class 'member_descriptor'>
b_needsfree: <class 'member_descriptor'>
_objects: <class 'member_descriptor'>
type: <class 'str'>
value: <class 'getset_descriptor'>
progress_callback_user_data: <class 'NoneType'>
rpc_servers: <class 'NoneType'>
split_mode: <class 'int'>
tensor_split: <class 'llama_cpp.llama_cpp.LP_c_float'>
b_base: <class 'llama_cpp.llama_cpp.llama_model_params'>
b_needsfree: <class 'int'>
_objects: <class 'NoneType'>
type: <class 'ctypes.PyCSimpleType'>
contents: <Error: NULL pointer access>
use_mlock: <class 'bool'>
use_mmap: <class 'bool'>
vocab_only: <class 'bool'>
model_path: <class 'str'>
n_batch: <class 'int'>
n_ctx: <class 'method'>
n_embd: <class 'method'>
n_threads: <class 'int'>
n_threads_batch: <class 'int'>
n_tokens: <class 'int'>
n_vocab: <class 'method'>
numa: <class 'int'>
pooling_type: <class 'method'>
reset: <class 'method'>
sample: <class 'method'>
save_state: <class 'method'>
scores: <class 'numpy.ndarray'>
set_cache: <class 'method'>
set_seed: <class 'method'>
spm_infill: <class 'bool'>
tensor_split: <class 'NoneType'>
token_bos: <class 'method'>
token_eos: <class 'method'>
token_nl: <class 'method'>
tokenize: <class 'method'>
tokenizer: <class 'method'>
tokenizer: <class 'llama_cpp.llama_tokenizer.LlamaTokenizer'>
_abc_impl: <class '_abc._abc_data'>
_model: <class 'llama_cpp._internals.LlamaModel'>
decode: <class 'method'>
detokenize: <class 'method'>
encode: <class 'method'>
from_ggml_file: <class 'method'>
tokenize: <class 'method'>
verbose: <class 'bool'>
Could not find compute_graph in the model structure.
Could not find 'kq' tensor in the graph.