fix: [AG-178] AI Gateway bugs, 3.9.0 rollup

[tysoekong]

Summary

AG-178

Bug fix rollup from 3.9.0.RC-1

Checklist

• The Pull Request has tests
• Doesn't need changelog - unreleased features
• Doesn't need new docs - bug fixes

Issue reference

Fixes everything in AG-178.

[fffonion]

this is still needed, when normalize-json-response is not enabled on a request (for example, using semantic-cache without ai proxy)

[tysoekong]

@oowl What do you recommend?

Just duplicate the code for now "QAD" with a if not (namespace-ai-proxy) then run_the_code end ?

[tysoekong]

Because actually, metadata extraction itself should be move to its own filter too...

[fffonion]

it's fine to run this code twice, the later one will just update/overwrite the previous, same as how we do to
headers and body.

metadata extraction itself should be move to its own filter too...

Ideally yes, but we are not in the real filter pipeline but still in Kong's plugin iterator.
Right now, move this to llm/shared to a utility function and to be called by both filters will be fine. Let's do this after 3.9.

[fffonion]

So ideally:
for ai-proxy
parse-json-response | get metadata | normalize-json-response | get metadata
for ai-semantic-cache etc, without ai-proxy
parse-json-response | get metadata

Even if we do this, in real world, the plugin iterator still execute all "filters" for one plugin before executing the other, so if you have ai-proxy + ai-semantic-cache it would actually be:

parse-json-response | get metadata | parse-json-response (skipped) | get metadata | normalize-json-response | get metadata

so I think it's fine to just have get metadata part of the parse and normalize so we would have
parse-json-response| parse-json-response (skipped) | normalize-json-response

[tysoekong]

I'll add it back and just double check the tests!

[tysoekong]

Oh you already did it! Okay am very confused now.

[tysoekong]

I've properly fixed the flakey tests

[tysoekong]

@fffonion @oowl Finally... good to go.