fix: respect provider choice from UI

[TommyHolmberg]

This change ensures the streamText function respects the provider choice made in the UI.

I am unable to test this thoroughly because I don't have API keys for the providers, as I use local models.

Even locally, this still doesn't work, but it's a step in the right direction. The URL calls are correct, but they aren't reaching the local LiteLLM server for some reason.

[nandan121]

I merged this change into my fork, and it works as expected. Thanks Tommy

I have tested using same model name, gpt-4o-mini, and openAI vs OPENAI_LIKE...

[nandan121]

This change ensures the streamText function respects the provider choice made in the UI.

I am unable to test this thoroughly because I don't have API keys for the providers, as I use local models.

Even locally, this still doesn't work, but it's a step in the right direction. The URL calls are correct, but they aren't reaching the local LiteLLM server for some reason.
Hello Tommy,
I could get it working for liteLLM. I used your https://github.com/TommyHolmberg/bolt.new-any-llm/tree/respect-provider-choice , my constants file was https://github.com/nandan121/bolt.new-any-llm/blob/417307bdb5edf2b4a5da92c9d1c6ec15b8f68923/app/utils/constants.ts

I tried one step at a time with the free API from Mistral to try. https://docs.litellm.ai/docs/providers/mistral It worked fine. The issue in many cases was that bolt uses about 3500 tokens each time, and so some free models came up with error.

[coleam00]

This is great, thank you @TommyHolmberg! I'm going to test this myself this week and merge it in.

[TommyHolmberg]

This change ensures the streamText function respects the provider choice made in the UI.
I am unable to test this thoroughly because I don't have API keys for the providers, as I use local models.
Even locally, this still doesn't work, but it's a step in the right direction. The URL calls are correct, but they aren't reaching the local LiteLLM server for some reason.
Hello Tommy,
I could get it working for liteLLM. I used your https://github.com/TommyHolmberg/bolt.new-any-llm/tree/respect-provider-choice , my constants file was https://github.com/nandan121/bolt.new-any-llm/blob/417307bdb5edf2b4a5da92c9d1c6ec15b8f68923/app/utils/constants.ts

I tried one step at a time with the free API from Mistral to try. https://docs.litellm.ai/docs/providers/mistral It worked fine. The issue in many cases was that bolt uses about 3500 tokens each time, and so some free models came up with error.

Great! Shortly after your comment I solved the problem on my setup too. It works fine with litellm now.

I hope it still works with all of the regular providers as well. I don't see a reason why it shouldn't, but you never know until it's been tested.

[nandan121]

It seems that the message has "Provider" in it, and it needs to be removed

[TommyHolmberg]

It seems that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

[nandan121]

It seems that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expecms that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expected(no model name or provider name).

BUT, my suggestion would be to have it in the chat... It is nice and does not take space.

I would suggest that in https://github.com/coleam00/bolt.new-any-llm/pull/188/files#diff-dd246a18ac09a3e1a293988bf2f81b4f246f4c4aa03db23db48f79371c378981 I would probably have it in my version.

return content.replace(modificationsRegex, '').replace(MODEL_REGEX, '$1').replace(PROVIDER_REGEX, ' ($1)\n\n').trim();

could be better. I will allow the @coleam00 to make that decision.

[TommyHolmberg]

It seems that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expecms that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expected(no model name or provider name).

BUT, my suggestion would be to have it in the chat... It is nice and does not take space.

I would suggest that in https://github.com/coleam00/bolt.new-any-llm/pull/188/files#diff-dd246a18ac09a3e1a293988bf2f81b4f246f4c4aa03db23db48f79371c378981 I would probably have it in my version.

return content.replace(modificationsRegex, '').replace(MODEL_REGEX, '$1').replace(PROVIDER_REGEX, ' ($1)\n\n').trim();

could be better. I will allow the @coleam00 to make that decision.

I think it looks great! And good catch with those line breaks. I was wondering why the message boxes looked off.

[coleam00]

I appreciate the back and forth here @TommyHolmberg and @nandan121! Could you clarify if this is ready to merge now?

[TommyHolmberg]

I added Using: before the model and provider to clarify what it's about. The best would be if we could get the model and provider into the response message, but that requires a bit of work and it doesnt belong in this PR.

@coleam00 In my opinion it is ready to merge.

[nandan121]

I appreciate the back and forth here @TommyHolmberg and @nandan121! Could you clarify if this is ready to merge now?

Thanks @coleam00 and @TommyHolmberg . I tested it once, it looks great. All is working and ready to merge.

[coleam00]

Fantastic, thank you @TommyHolmberg and @nandan121!

[mehmetsahin06]

It seems that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expecms that the message has "Provider" in it, and it needs to be removed

I completely overlooked that. I found the sanitize method for it and could fix it rather easily. Try the PR now.

Tried, and it works as expected(no model name or provider name).

BUT, my suggestion would be to have it in the chat... It is nice and does not take space.

I would suggest that in https://github.com/coleam00/bolt.new-any-llm/pull/188/files#diff-dd246a18ac09a3e1a293988bf2f81b4f246f4c4aa03db23db48f79371c378981 I would probably have it in my version.

return content.replace(modificationsRegex, '').replace(MODEL_REGEX, '$1').replace(PROVIDER_REGEX, ' ($1)\n\n').trim();

could be better. I will allow the @coleam00 to make that decision.

how to use openai-like gpt-4o mini

[nandan-dhamma]

@mehmetsahin06 , it will be good if you can explain what you want to do, or are trying to do? And a copy of your constants.ts. Make sure you have the latest pull from github