Form-data request is not being streamed

[LaeraFelipe]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.7.1

Current Behavior

We're sending a request with the form data content type, and apparently the request is only forwarded to the upstream server when kong has finished reading all the content there.

We've tried disabling request_buffering in the route and in the nginx settings using the configuration:

nginx_proxy_request_buffering=off

But nothing seems to work. Can anyone help us?

Expected Behavior

The request should be streamed in real time.

Steps To Reproduce

1 - Configure a route
2 - Send a request with content type: application/octet-stream.

Anything else?

No response

[hanshuebner]

Buffering of request data can be configured per route using the request_buffering parameter. If you want to disable request buffering globally, you can try to set nginx_proxy_proxy_request_buffering to off.

[git-hulk]

Should be proxy_request_buffering instead of nginx_proxy_request_buffering?

[LaeraFelipe]

@hanshuebner Thanks for the reply!

We've tried both configurations, both per route and globally. But the behavior remains the same: KONG seems to read the entire request before forwarding it to the upstream server.

I imagine that the correct thing to do with the request_buffering settings disabled would be to chain the request immediately to the upstream server.

[hanshuebner]

I've done a little bit of digging and found that request and response buffering can only be turned off for HTTP/1.1 requests. It seems that there was a change in #10204 to fix this. Can you confirm that the problem you have is related to HTTP/2.0 requests?

[LaeraFelipe]

Requests are made in HTTP 1.1.
Here are the response headers with and without the request_buffering option. Note that when it's enabled, the upstream_latency is reduced by a few milliseconds and the proxy_latency is increased. But the time remains high in both conditions.

request_buffering = false

request_buffering = true

[brentos]

@LaeraFelipe are there any other plugins running for these requests? Certain plugins that read the request body will force buffering back on.

[LaeraFelipe]

@brentos I'm using ACL and Basic Authentication.

[LaeraFelipe]

I imagine these plugins only need to read the headers. Am I right?

[brentos]

Yeah, these shouldn't force buffering on.

[fffonion]

Please also share relevent client code that send the request, or a curl snippet, for example, does client send Content-Length and/or Transfer-Encoding headers?

[LaeraFelipe]

I'm sending these headers:
{ 'content-type': 'multipart/form-data; boundary=--------------------------300558992205452359757022', 'x-census-user': 'default', 'x-api-key': '*********', 'x-census-groups': 'default,test', 'x-metadata-file': '************', host: 'laeratests.kong-census-hml.skyone.tools:8100', authorization: 'Basic MTIzOjEyMw==', 'content-length': 4667436755 }
And body as form-data.

[LaeraFelipe]

Is there something wrong with the headers?

[mayankmurari]

@LaeraFelipe Was your Issue resolved? I am also seeing same behaviour where even after setting annotation konghq.com/request-buffering: "false" request seems to be getting buffered for multipart/form-data requests.