Podcastfy not working with local model

[Davide-gtr]

Hello, when I try to use Podcastfy with local model (TinyLlama for example), it don't generate the right formatting, and sometimes it don't generate anything.

I don't have this issue with OpenAI model (GPT-4o), so can it be related to the context window ? I know TinyLlama has only 2048 tokens context window.

Or is it something else ?

Thank you in advance for your help and for your tool.

[souzatharsis]

Hi thanks for your question. Unfortunately, there are no guarantees when it comes to open and particularly small models. However, I do acknowledge that integration with outlines to guarantee structured output would greatly increase reliability as demonstrated here: https://www.souzatharsis.com/tamingLLMs/notebooks/structured_output.html#outlines 1. Potential instability: Local models may produce less consistent or stable outputs compared to well-tested private models oftentimes producing transcripts that cannot be used for podcast generation (TTS) out-of-the-box

…

On Fri, Dec 20, 2024 at 3:32 PM Davide Garcia Civiero < ***@***.***> wrote: Hello, when I try to use Podcastfy with local model (TinyLlama for example), it don't generate the right formatting, and sometimes it don't generate anything. I don't have this issue with OpenAI model (GPT-4o), so can it be related to the context window ? I know TinyLlama has only 2048 tokens context window. Or is it something else ? Thank you in advance for your help and for your tool. — Reply to this email directly, view it on GitHub <#215>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTMY3KPSNZO4ZT6X5WR2GT2GRPCLAVCNFSM6AAAAABT7UFXHCVHI2DSMVQWIX3LMV43ASLTON2WKOZSG42TGMRWGE2DKMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Davide-gtr]

Hi,
Yes I understand the limits of local models, I also tested with Llama 3.1 model but I have the same issue.
I will definitely take a look at the outlines library, maybe I can enhance the formatting.
Thank's a lot for the answers!