Optillm for Aider

[Mushoz]

Has anyone tried using optillm with Aider? It contains a builtin benchmark as well, but so far, optillm isn't adding any performance for the approaches that I benchmarked with Qwen2.5-Coder-32B running locally. Was wondering if anyone else had any better luck with the OptiLLM + Aider combination.

[codelion]

It should be straightforward to use optillm with any model in Aider as they support OpenAI compatible endpoints - https://aider.chat/docs/llms/openai-compat.html As for the aider-bechmark I do not think it is publically available? In any case the benchmark tests ability of models to edit code effectively. It is likely a model fine-tuned on that task would do better. I had trained a model for similar task recently - https://huggingface.co/patched-codes/Llama-3.2-1B-FastApply

[Mushoz]

Aider's benchmark is available publicly. Documentation on how to run it can be found here: https://github.com/Aider-AI/aider/blob/main/benchmark/README.md

It doesn't really specifically target the actual editing, but is more general as it asks the model to implement certain requirements. I know how to use OptiLLM with Aider, but I am experiencing worse performance with the approaches I benchmarked (cot_reflection, re2 and plansearch). Was curious to see if anyone was able to enhance Aider's performance through OptiLLM.

[codelion]

I had a look at the benchmark README, based on my experience plansearch should have done well as it performed good on the LiveCodeBench. What base model are you trying out aider benchmark with?

[Mushoz]

I am using Qwen2.5-Coder-32B, with and without OptiLLM. I did see those LiveCodeBench results in OptiLLM's readme, but the results are nearly identical pass@1. It's only at pass@5 and even more so at pass@10 where it pulls ahead. Since Aider only has 2 shots, this might be the reason why I am not seeing improvements.

[codelion]

Oh yes, most of the techniques implemented in optillm only scale when giving more inference time compute.