Memory requirements using transformers

[motaatmo]

Hi,
I'm trying to use guidance for a data extraction task. I'm currently using the transformer models (as for reasons that I don't fully understand llama.cpp does not work on the machine, something with Cuda driver mismatch).
I can access two A100, yet I'm running out of VRAM with llama 2 13B, context lengths around 3k, and "device_map" set to "auto". Which I don't understand. Debugging is a PITA as the machine can only be accessed using SLURM/sbatch (no interactive options).
Does anybody have a similar setup and can give me some hints on what I might try?
Greetings!