Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Json output for different models varies #1885

Open
PranavPuranik opened this issue Sep 19, 2024 · 2 comments · May be fixed by #1884 or #1978
Open

Json output for different models varies #1885

PranavPuranik opened this issue Sep 19, 2024 · 2 comments · May be fixed by #1884 or #1978
Labels
bug Something isn't working enhancement New feature or request

Comments

@PranavPuranik
Copy link
Contributor

🐛 Describe the bug

Some models output json response that starts with ```

This doesn't get loaded in json.loads() and throws an error.

A better prompt could solve it for now, but we should look for model specific prompt solutions.

@PranavPuranik PranavPuranik linked a pull request Sep 19, 2024 that will close this issue
18 tasks
@Morgandri1
Copy link

Morgandri1 commented Sep 19, 2024

IMO this 'flaw' could be useful for parsing generated JSON and code, especially when there is some inconsistency between model output. Personally I can see a lot of benefit using the triple backtick as a flag for something to be parsed, seeing as these models arent designed explicitly to output parsable data. even better when the model uses correct MD syntax and lists the content type after the opening ticks. This could make finding data tokens in outputs easier and (hopefully) less likely to be parsed incorrectly when read. i.e, if closing ticks arent found, the block could be disregarded as it is unlikely to contain valid data.

@spike-spiegel-21
Copy link
Collaborator

Duplicate -> #1854

@Dev-Khant Dev-Khant added bug Something isn't working enhancement New feature or request labels Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
4 participants