Short write errors on http.handlers.reverse_proxy

[bpizzi]

Hi,

I'm building coraza-caddy/v2 against caddy 2.7.5 (inside a caddy:2.7.5-alpine docker image):

xcaddy build --with github.com/corazawaf/coraza-caddy/v2

Then I'm running this image inside a docker container and configuring it to trigger the waf on requests that are reverse proxied to a remote inside a docker network (actually an apache/php-fpm stack inside another containter):

        handle * {
                coraza_waf {
                    load_owasp_crs
                    directives `
                    Include @coraza.conf-recommended
                    Include @crs-setup.conf.example
                    Include @owasp_crs/*.conf
                    SecRuleEngine On
                    `
                }

                reverse_proxy http://com_apache_1 {
                        header_up X-Real-IP {remote_host}
                }
        }

It works well for some of my remotes, but one is always triggering short write errors on http.handlers.reverse_proxy (I've redacted some ip/host below):

{"level":"error","ts":1704360844.105692,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"com_apache_1:80","duration":0.625059646,"request":{"remote_ip":"","remote_port":"60865","client_ip"
:"","proto":"HTTP/2.0","method":"GET","host":"www..com","uri":"/.html","headers":{"X-Forwarded-For":[""],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"X-Real-Ip":[""],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["www..com"],"User
-Agent":[""],"From":[""]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"pro
to":"h2","server_name":""}},"error":"short write"}

In that case, there's no http.handlers.waf errors.

I've also tried and found the almost same behavior with caddy 2.7.6 and xcaddy build --with github.com/corazawaf/coraza-caddy@master:

• with caddy 2.7.5 and coraza-caddy/v2, the remote html content is partially returned to the browser (ie. the html file is not complete, the end is missing),
• with caddy 2.7.6 and coraza-caddy@master, the browser gets nothing at all (html file is empty),
• both cases triggers a http.handlers.reverse_proxy/short write in the logs.

The the best of my knowledge, the only difference with the remote not working and the others is the size of the html page: the one triggering short write is 914kb, whereas the (non broken) others are below 500kb.

Maybe I'm missing something in the configuration for handling a large content? Or it could be not related at all to the content size, in which case I would appreciate some direction where I could investigate.

Thanks!

[jptosso]

Hey!
Please try xcaddy build --with github.com/corazawaf/coraza-caddy/v2@latest

[bpizzi]

Hey! Please try xcaddy build --with github.com/corazawaf/coraza-caddy/v2@latest

Holy cow, I was not expecting an answer that fast ;)

Thanks for the hint, I just did that, caddy 2.7.6 and coraza-caddy/v2@latest, and it's not better.
The browser returns Secure Connection Failed, and the logs "logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response", ..., "error":"short write".

[jptosso]

@jcchavezs @M4tteoP any chance this is related to ResponseBodyLimit buffer?

@bpizzi can you confirm the response content-length?

[M4tteoP]

We faced the short write in corazawaf/coraza#852 and #107 (it was also a fix for this that should be present when running with @latest 🤔). It was always following an interruption: do you see any logs about rules triggered before this errror?

[bpizzi]

@bpizzi can you confirm the response content-length?
It was always following an interruption: do you see any logs about rules triggered before this errror?

Yep so I made some additional testing.

With all versions of caddy/coraza-caddy, with no coraza_waf directive in Caddyfile, the content is fully retrieved:

With Caddy 2.7.6 and coraza-caddy/v2@latest, the https connection fails in firefox with Secure Connection Failed, so no way to check the response content-lenght, and we have short write in logs.

Getting back to Caddy 2.7.3 and coraza-caddy/v2 (not @latest), the https connection do not fail, but the content is partially retrieved:

That's with:

                coraza_waf {
                    load_owasp_crs
                    directives `
                    Include @coraza.conf-recommended
                    Include @crs-setup.conf.example
                    Include @owasp_crs/*.conf
                    SecRuleEngine On
                    `
                }

Now if I switch SecRuleEngine to Off then I get rid of short write and have the full content.
If I switch it to DetectionOnly then we're back to the partial content, to the exact same limited length each time (524.29kb).

And I'm pretty sure there's no http.handlers.waf triggered in any of the logs in both cases.
The thing is, this is a production server, and not having hostname filled (#75) doesn't help, but that's another discussion ;)
However I guess I could find some time to work on a docker-compose.yml for trying to reproduce it offline, if you think it would help.

Otherwise I can confirm that I'm able to trigger the waf on other remotes with the same caddy/coraza-caddy stack (while having the full ~200kb content when the waf is not triggered):

{"level":"error","ts":1704374418.3372767,"logger":"http.handlers.waf","msg":"[client \\"] Coraza: Access denied (phase 2). XSS Attack Detected via libinjection [file \"@owasp_crs/REQUEST-941-APPLICATION-ATTACK-XSS.conf\"] [line \"4453\"] [id \"941100\"] [rev \"\"] [msg \"XSS Attack Detected via libinjection\"] [data \"Matched Data: XSS data found within ARGS:search: <script>alert('CRS Sandbox Release')</>\"] [severity \"critical\"] [ver \"OWASP_CRS/4.0.0-rc1\"] [maturity \"0\"] [accuracy
\"0\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-xss\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/152/242\"] [hostname \"\"] [uri \"/?search=%3Cscript%3Ealert(%27CRS+Sandbox+Release%27)%3C/%3E\"] [unique_id \"WovVznFcxWcMWkbh\"]\n"}

[cs8425]

I got similar issue here.
with config:

SecRuleEngine On/DetectionOnly
SecResponseBodyAccess On
SecResponseBodyLimitAction ProcessPartial
SecResponseBodyLimit 524288

when upstream server return a chunked response with size > 512 KB,
client will get error net::ERR_INCOMPLETE_CHUNKED_ENCODING,
and there is only a short write error in caddy log (http.handlers.reverse_proxy aborting with incomplete response).
not test for non-chunked response yet.

If I disable SecResponseBodyAccess or increase SecResponseBodyLimit,
the response return without any issue.
Seems that we may have some issue with SecResponseBodyLimitAction ProcessPartial?

[9mlcy7]

coraza/http/interceptor.go:74, 触发 BodyLimitActionReject 或者 BodyLimitActionProcessPartial 会返回 0，导致 /usr/local/go/src/net/http/httputil/reverseproxy.go:459 在调用 dst.Write 后对比 nr\nw 抛出的 short write

参考 corazawaf/coraza#1123 (commits) 修改