-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support chunked requests when APIcast talks directly to upstream (not proxy in the middle) #1405
Conversation
It is required to not read request body for chunked requests to be propagated as chunked requests. Only read the request body when POST query arguments (of the MIME type application/x-www-form-urlencoded) and thus query arguments could be in the request body. Note: chunked requests are propagated to upstream with Transfer-Encoding chunked only when client request sends more than one chunk
45da7a9
to
656e26d
Compare
656e26d
to
87f67fc
Compare
@@ -95,6 +95,8 @@ location @upstream { | |||
proxy_pass $proxy_pass; | |||
|
|||
proxy_http_version 1.1; | |||
proxy_request_buffering off; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if turning this off globally is a good idea (Slowloris DDoS attack) May be we can have another location block @upstream-unbufferd instead?
There is an interesting discussion on GitLab https://gitlab.com/gitlab-org/gitlab/-/issues/325095
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am happy to discuss about potential issues with this approach.
Slowloris DDoS attack is about opening too many TCP connections with unfinished HTTP requests. You are suggesting that the APIcast gateway should protect upstream from opening connections against upstream until the entire request is read (and buffered). This what I understand, let me know if I am wrong. If that is true, then https://issues.redhat.com/browse/THREESCALE-9542 (requests with transfer encoding chunked) cannot be supported for all scenarios. For the current scenario (no proxy), when the request is buffered by nginx, I do not know how to tell nginx to stick with the chunked requests on the upstream HTTP session. Nginx has always added the Content-Length to the upstream request when it knows the request length. If we knew how to change Nginx behavior, request would be buffered (preventing from the DDoS attack) and chunked requests would be propagated to upstream with the same encoding.
I assumed we wanted to respect chunked requests with big size bodies (with Gb length) to avoid buffering (and writing to temp file)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we have another location @upstream-unbufferd
, what would be the routing logic to forward to @upstream
or @upstream-unbuffered
? For requests with "small" body, no matter chunked or not, nginx will read and know the length of the body. Then it will send the request to upstream with the "content-length" header. Thus, it does not matter @upstream
or @upstream-unbuffered
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry if I confused you, but what I suggest is to allow disable proxy_request_buffering per location instead of globally.
The reason are:
- DDoS attack
- This is a breaking change, so I think it's better to be able to configure this change per location than to go and upgrade and everything suddenly stops working.
Also, I'm just throwing ideas around, so please ignore me if I'm talking non-sense here 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This location is dynamic so by doing this via a policy you are not enabling it globally. I don't know what was the outcome of the last meeting we had but did we figure out if it was possible to just implement transfer-encoding as a policy? Leave existing behaviour for buffered requests as it is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So from the last meeting, we concluded that Transfer-Encoding
header is hop-by-hop header, so the gateway is not enforced to propagate that header or transfer encoding itself. What it can be configured (exactly the same as nginx does), is the request buffering. It can be on/off.
@eguzki thinks the issue with the policy can be the order where the "request buffering" policy exists. If it is in an incorrect order, it might not work as expected. And he also thinks that we should have a single location @upstream
and control the proxy_request_buffering
via the Lua code. However, something like proxy_request_buffering $proxy_request_buffering
won't work as nginx complained on boot saying that proxy_request_buffering
should be on
or off
.
My proposal is to have a second @upstream-unbuffered
block inside https://github.com/3scale/APIcast/blob/master/gateway/conf.d/apicast.conf
and inside the policy we set the request context, something like request_buffered = false
and here https://github.com/3scale/APIcast/blob/master/gateway/src/apicast/upstream.lua#L250 we can check the context and rewrite the location_name
to @upstream-unbuffered
909f6a0
to
73cd1a9
Compare
superseded by #1408 |
What
As part of https://issues.redhat.com/browse/THREESCALE-9542 this is fixing the support for chunked requests when APIcast talks directly to upstream (not proxy between APIcast and upstream).
The fix is disabling request buffering, which means nginx will forward request body ASAP. Does this mean that when the downstream request in chunked (
Transfer-Encoding: chunked
), the request to upstream service will also be chunked? Well, it depends. If nginx can read the entire body in one chunk, the upstream request will not be chunked andContent-Length
header will be added. On the other hand, if after reading the first chunk, nginx cannot know the body length, it will build an upstream request with transfer encoding chunked. This usually happens when the second chunk is delayed.The changes to support chunked requests affect only
gateway/src/apicast/configuration/service.lua
andgateway/conf.d/apicast.conf
. The additional files changed in this PR are only to create dev environments to reproduce use cases.Dev Note: In order to implement chunked requests, request body should not be read by the gateway. If the body is read, buffering happens and the entire body will be read before sending request to upstream. When the request body is read, Nginx will know the request body length, hence will favor upstream request with the
Content-Length
header instead of the chunked one. However, APIcast reads the body looking for request params (forContent-Type: application/x-www-form-urlencoded
requests) because 3scale mapping rules may match parameters in the body. Thus, for requests withContent-Type: application/x-www-form-urlencoded
, the request body will be read (and buffered) before sending request to upstream and the request to upstream will always be withContent-Length
header.Verification steps
That request should return
200 OK
.Note that upstream echo API is reporting that the request included
Content-Length
header and the expected body.First add to
/etc/hosts
filepost
hostname having${APICAST_IP}
value. Resulting in:Then send request
Note that the upstream service got transfer encoding chunked request.
Note the chunked encoding of the request with the length bytes preceding each chunk.