Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpxy accepts HTTP2 when force_http11_upstream is set, request fails #184

Open
akostadinov opened this issue Sep 15, 2024 · 15 comments
Open

Comments

@akostadinov
Copy link
Contributor

akostadinov commented Sep 15, 2024

Don't know if this is related to #77 or not. But I have an upstream that only supports HTTP 1.1

Like this:

[apps.myapp]
server_name = 'myrpxy.example.com'
reverse_proxy = [{ upstream = [{ location = 'OpenWrt', tls = true }] }]
tls = { https_redirection = true, acme = true }
upstream_options = [ "force_http11_upstream" ]

But when running curl:

curl -v https://myrpxy.example.com
* Host myrpxy.example.com:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.124
*   Trying 192.168.1.124:443...
* Connected to myrpxy.example.com (192.168.1.124) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=myrpxy.example.com
*  start date: Sep  2 20:47:06 2024 GMT
*  expire date: Dec  1 20:47:05 2024 GMT
*  subjectAltName: host "myrpxy.example.com" matched cert's "myrpxy.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://myrpxy.example.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: myrpxy.example.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: myrpxy.example.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 500 
< date: Sun, 15 Sep 2024 22:39:18 GMT
< 
* Connection #0 to host myrpxy.example.com left intact

So basically TLS handshake appears to incorrectly accept h2 when it should accept http/1.1 because of the upstream server.

P.S. Request works with curl --http1.1 but there is no such option for normal browsers.

@junkurihara
Copy link
Owner

junkurihara commented Sep 16, 2024

force_http11_upstream option just converts incoming requests of HTTP/1.1, 2 or 3 to that of HTTP/1.1, and does not offers only HTTP/1.1 directly to clients.

In my understanding, your setting seems as below.

client -> (https over http/2) -> rpxy -> (https over http/1.1) -> backend app

But it is weird that your application returns 500 when clients send request with HTTP/2. The setting seems that rpxy converts HTTP/2 to HTTP/1.1 towards your backend app. So could you check logs of rpxy and your backend app?

@akostadinov
Copy link
Contributor Author

akostadinov commented Sep 16, 2024

Sorry, I missed to paste rpxy log because the error message in rpxy log seemed like http2 forwarding to http1 was not possible. I think the relevant part is:

2024-09-16T20:38:45.038091Z DEBUG rpxy rpxy_lib::proxy::proxy_main:191: HTTP/2 or 1.1: SNI in ClientHello: "myserver.example.com"
2024-09-16T20:38:45.060956Z DEBUG rpxy rpxy_lib::proxy::proxy_main:82: Request incoming: current # 2
2024-09-16T20:38:45.082350Z DEBUG rpxy rpxy_lib::backend::upstream:88: Found upstream: "/"
2024-09-16T20:38:45.083015Z DEBUG rpxy rpxy_lib::message_handler::handler_manipulate_messages:69: Generate request to be forwarded
2024-09-16T20:38:45.083664Z DEBUG rpxy rpxy_lib::backend::upstream:231: Upstream of index 0 is chosen.
2024-09-16T20:38:45.083690Z DEBUG rpxy rpxy_lib::backend::upstream:232: Context to LB (Cookie in Request): None
2024-09-16T20:38:45.083697Z DEBUG rpxy rpxy_lib::backend::upstream:233: Context from LB (Set-Cookie in Response): None
2024-09-16T20:38:45.083710Z DEBUG rpxy rpxy_lib::message_handler::handler_main:163: Request to be forwarded: [uri https://OpenWrt/, method: GET, version HTTP/2.0, headers {"user-agent": "curl/8.6.0", "accept": "*/*", "x-forwarded-for": "192.168.1.203", "x-forwarded-proto": "https", "x-forwarded-port": "8443", "x-real-ip": "192.168.1.203", "x-forwarded-ssl": "on", "x-original-uri": "https://myserver.example.com/", "proxy": "", "host": "myserver.example.com"}]
2024-09-16T20:38:45.308347Z  WARN rpxy hyper_util::client::legacy::client:283: Connection is HTTP/1, but request requires HTTP/2
2024-09-16T20:38:45.310803Z ERROR rpxy rpxy_lib::message_handler::handler_main:74: Failed to get response from backend: Failed to fetch from upstream: client error (UserUnsupportedVersion)
2024-09-16T20:38:45.313777Z  INFO rpxy rpxy_lib::message_handler::http_log:76: myserver.example.com <- 192.168.1.203:56784 -- GET / HTTP/2.0 -- 500 Internal Server Error -- https://myserver.example.com "curl/8.6.0", "192.168.1.203" "https://OpenWrt/"
2024-09-16T20:38:45.350427Z DEBUG rpxy rpxy_lib::proxy::proxy_main:112: Request processed: current # 1

More specifically

Connection is HTTP/1, but request requires HTTP/2

P.S. the upstream service is OpenWRT whatever its configuration interface uses as a web server. But should be pretty simple software because these devices are resources constraint. I expect something that adheres to as few modern specifications as possible.

@akostadinov
Copy link
Contributor Author

OpenWRT uses uhttpd by default. But I assume the result would be the same with any other web server that doesn't support HTTP2.

Another thing I notice though is that this only is a problem with upstream = [{ location = 'whatever.example.com', tls = true }]. When upstream is not using TLS, the requests appear to be forwarded as HTTP1 🤔

@junkurihara
Copy link
Owner

Thanks for the investigation! Actually, seems weird. I will check but needs time for a while. For now I am guessing this is due to the configuration of hyper-client sending requests towards backend apps.

@junkurihara
Copy link
Owner

junkurihara commented Sep 20, 2024

I just tested on localhost and my rpxy hosted server.

Flow:

curl -> (http 2 or 3) -> localhost rpxy -> (http 1.1 w/ force_http11_upstream) -> rpxy/nginx hosted site on the Internet

rpxy settings:

[[apps.localhost.reverse_proxy]]
upstream = [
  { location = 'whoami-1.rpxy.net', tls = true },
]
load_balance = "sticky"
upstream_options = [
   "set_upstream_host",
   "force_http11_upstream",
]

Curl:

$ curl https://localhost:4433 -v -i --insecure --http2 -I
$ curl https://localhost:4433 -v -i --insecure --http3 -I

The log are as below, which are actually for proxied requests to rpxy but ones to nginx results same.

HTTP/3

2024-09-20T02:21:22.746202Z  INFO rpxy rpxy_lib::proxy::proxy_h3:36: QUIC/HTTP3 connection established from [::1]:58047 localhost
~~~~~~~
2024-09-20T02:21:22.747057Z DEBUG rpxy rpxy_lib::message_handler::utils_request:69: HTTP/3 is currently unsupported for request to upstream.
2024-09-20T02:21:22.747067Z DEBUG rpxy rpxy_lib::message_handler::handler_main:163: Request to be forwarded: [uri https://whoami-1.rpxy.net/, method: HEAD, version HTTP/1.1, headers {"user-agent": "curl/8.10.1", "accept": "*/*", "x-forwarded-for": "::1", "x-forwarded-proto": "https", "x-forwarded-port": "4433", "x-real-ip": "::1", "x-forwarded-ssl": "on", "x-original-uri": "https://localhost:4433/", "proxy": "", "host": "whoami-1.rpxy.net"}]
~~~~~~
2024-09-20T02:21:22.832814Z  INFO rpxy rpxy_lib::message_handler::http_log:76: localhost <- [::1]:58047 -- HEAD / HTTP/3.0 -- 200 OK -- https://localhost "curl/8.10.1", "::1" "https://whoami-1.rpxy.net/"

HTTP/2

2024-09-20T02:21:31.218937Z DEBUG rpxy rpxy_lib::proxy::proxy_main:191: HTTP/2 or 1.1: SNI in ClientHello: "localhost"
~~~~~
2024-09-20T02:21:31.223477Z DEBUG rpxy rpxy_lib::message_handler::handler_main:163: Request to be forwarded: [uri https://whoami-1.rpxy.net/, method: HEAD, version HTTP/1.1, headers {"user-agent": "curl/8.10.1", "accept": "*/*", "x-forwarded-for": "::1", "x-forwarded-proto": "https", "x-forwarded-port": "4433", "x-real-ip": "::1", "x-forwarded-ssl": "on", "x-original-uri": "https://localhost:4433/", "proxy": "", "host": "whoami-1.rpxy.net"}]
~~~~~
2024-09-20T02:21:31.247187Z  INFO rpxy rpxy_lib::message_handler::http_log:76: localhost <- [::1]:57083 -- HEAD / HTTP/2.0 -- 200 OK -- https://localhost "curl/8.10.1", "::1" "https://whoami-1.rpxy.net/"

So, mine works flawlessly actually. very weird... I will investigate more

@akostadinov
Copy link
Contributor Author

I see. I tried this. The difference is that in your example, you don't use HTTPS to the downstream, only to the upstream.

So this worked for me:

[apps.testdomain]
server_name = 'exampledomain.ddnsgeek.com'
tls = { https_redirection = true, acme = true }
reverse_proxy = [{ upstream = [{ location = 'whoami-1.rpxy.net', tls = true }] }]
upstream_options = [
   "force_http11_upstream",
]

And the result was:

🐚 curl -v http://exampledomain.ddnsgeek.com/
* Host exampledomain.ddnsgeek.com:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.124
*   Trying 192.168.1.124:80...
* Connected to exampledomain.ddnsgeek.com (192.168.1.124) port 80
> GET / HTTP/1.1
> Host: exampledomain.ddnsgeek.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< date: Fri, 20 Sep 2024 11:07:01 GMT
< server: rpxy
< transfer-encoding: chunked
< 
* Connection #0 to host exampledomain.ddnsgeek.com left intact

And rpxy logs:

2024-09-20T11:07:15.582830Z DEBUG rpxy rpxy_lib::message_handler::handler_main:163: Request to be forwarded: [uri https://whoami-1.rpxy.net/, method: GET, version HTTP/1.1, headers {"host": "exampledomain.ddnsgeek.com", "pragma": "no-cache", "upgrade-insecure-requests": "1", "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1", "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-encoding": "gzip, deflate", "accept-language": "en-US,en;q=0.9", "cache-control": "no-cache", "x-forwarded-for": "104.252.186.112", "x-forwarded-proto": "http", "x-forwarded-port": "8080", "x-real-ip": "104.252.186.112", "x-forwarded-ssl": "off", "x-original-uri": "/", "proxy": ""}]
2024-09-20T11:07:15.836817Z  INFO rpxy rpxy_lib::message_handler::http_log:76: exampledomain.ddnsgeek.com <- 104.252.186.112:59844 -- GET / HTTP/1.1 -- 400 Bad Request --  "Mozilla/5.0 (iPhone; CPU iPhone OS 16_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1", "104.252.186.112" "https://whoami-1.rpxy.net/"

But if I add https to the downstream as well with:

tls = { https_redirection = true, acme = true }

Then the result is:

$ curl -v https://exampledomain.ddnsgeek.com/
* Host exampledomain.ddnsgeek.com:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.124
*   Trying 192.168.1.124:443...
* Connected to exampledomain.ddnsgeek.com (192.168.1.124) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=exampledomain.ddnsgeek.com
*  start date: Sep 20 10:03:18 2024 GMT
*  expire date: Dec 19 10:03:17 2024 GMT
*  subjectAltName: host "exampledomain.ddnsgeek.com" matched cert's "exampledomain.ddnsgeek.com"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://exampledomain.ddnsgeek.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: exampledomain.ddnsgeek.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: exampledomain.ddnsgeek.com
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 400 
< date: Fri, 20 Sep 2024 11:04:32 GMT
< server: rpxy
< alt-svc: h3=":443"; ma=3600
< 
* Connection #0 to host exampledomain.ddnsgeek.com left intact

And rpxy log is:

2024-09-20T11:04:36.709830Z DEBUG rpxy rpxy_lib::message_handler::handler_main:163: Request to be forwarded: [uri https://whoami-1.rpxy.net/dns-query?dns=tM0BAAABAAAAAAAAA3d3dwNhYmMDY29tAAABAAE, method: GET, version HTTP/1.1, headers {"host": "exampledomain.ddnsgeek.com", "user-agent": "python-requests/2.22.0", "content-type": "application/dns-message", "accept": "*/*", "accept-encoding": "gzip, deflate", "x-forwarded-for": "165.227.64.216", "x-forwarded-proto": "https", "x-forwarded-port": "8443", "x-real-ip": "165.227.64.216", "x-forwarded-ssl": "on", "x-original-uri": "/dns-query?dns=tM0BAAABAAAAAAAAA3d3dwNhYmMDY29tAAABAAE", "proxy": ""}]
2024-09-20T11:04:36.961545Z  INFO rpxy rpxy_lib::message_handler::http_log:76: exampledomain.ddnsgeek.com <- 165.227.64.216:42856 -- GET /dns-query?dns=tM0BAAABAAAAAAAAA3d3dwNhYmMDY29tAAABAAE HTTP/1.1 -- 400 Bad Request --  "python-requests/2.22.0", "165.227.64.216" "https://whoami-1.rpxy.net/dns-query?dns=tM0BAAABAAAAAAAAA3d3dwNhYmMDY29tAAABAAE"

In both cases request passes through because the upstream apparently supports HTTP2. But in the second example the upstream request is HTTP2 instead of HTTP1.1 which should have been enforced.

@junkurihara
Copy link
Owner

junkurihara commented Sep 20, 2024

Haha! I found the reason!

This

[apps.testdomain]
server_name = 'exampledomain.ddnsgeek.com'
tls = { https_redirection = true, acme = true }
reverse_proxy = [{ upstream = [{ location = 'whoami-1.rpxy.net', tls = true }] }]
upstream_options = [
   "force_http11_upstream",
]

must be changed to

[apps.testdomain]
server_name = 'exampledomain.ddnsgeek.com'
tls = { https_redirection = true, acme = true }
reverse_proxy = [
  {
    upstream = [
      { location = 'whoami-1.rpxy.net', tls = true }
    ],
    upstream_options = [
       "force_http11_upstream",
    ]
 }
]

This is because options for upstream requests should be configured for each backend app! In your original configuration, options for upstream request were configured for the domain!

@junkurihara
Copy link
Owner

So I also really confirmed that we need more friendly configuration documentation. Current config-example.toml and README.md are really unfriendly.

@akostadinov
Copy link
Contributor Author

It doesn't look like a valid configuration. When I copy/paste the above configuration snippet in my toml I see

2024-09-20T11:59:13.669337Z  WARN main hot_reload:154: Failed to reload watch target

@junkurihara
Copy link
Owner

junkurihara commented Sep 20, 2024

Well it should be a typo. Anyways upstream and upstream_options should be on the same level as TOML format in reverse_proxy directive, like:

[apps."whoami-1.rpxy.net"]
server_name = 'whoami-1.rpxy.net'
reverse_proxy = [
  { upstream = [
    { location = 'whoami-1:8000', tls = false },
  ], upstream_options = ['keep_original_host'] }
]
tls = { https_redirection = true, acme = true }

The above is actually the configuration for whoami-1.rpxy.net.

@junkurihara
Copy link
Owner

Or, please prepare the separated reverse_proxy field as:

[apps.testdomain]
server_name = 'exampledomain.ddnsgeek.com'
tls = { https_redirection = true, acme = true }


[[apps.testdomain.reverse_proxy]]
upstream = [{ location = 'whoami-1.rpxy.net', tls = true }]
upstream_options = [
   "force_http11_upstream",
]

@akostadinov
Copy link
Contributor Author

This separate configuration works! Really thanks a lot!

But I think it is not a typo. There seems to be some magic related to parsing of stuff. This fails:

[apps."testdomain"]
server_name = 'somedomain.ddnsgeek.com'
reverse_proxy = [
  {
    upstream = [
    { location = 'OpenWrt', tls = true }
  ], upstream_options = ["force_http11_upstream"]
  }
]
tls = { https_redirection = true, acme = true }

This works:

[apps."testdomain"]
server_name = 'secrethq.ddnsgeek.com'
reverse_proxy = [
  { upstream = [
    { location = 'OpenWrt', tls = true },
  ], upstream_options = ["force_http11_upstream"] }
]
tls = { https_redirection = true, acme = true }

The difference is that I've put {} on separate lines. Perhaps brakets or indentation are not irrelevant in toml, I know nothing about it. Maybe this is by design, apologies if I should have known better.

But thanks again that it is now working as expected!

@akostadinov
Copy link
Contributor Author

So this can be closed. Unless you want to keep it open to implement auto-detection of HTTP2 support. Or for whatever reason.

@junkurihara
Copy link
Owner

junkurihara commented Sep 21, 2024

Good to hear that!

Honestly, the auto-detection of HTTP/2 (or 3) is quite tough. The client library using to issue request to backends supports auto negotiation. It works as follows: initial requests are usually sent over HTTP/1.1 and then subsequent requests would be updated to HTTP/2 or 3 according to responses's ALT-SVC field.

In `rpxy, that is totally overridden and the same HTTP version as original requests would be used by default (unless the backend connection is over plantext HTTP) to remove such the overhead of upgrading. This is from my concept that the backend connection should be controlled by the deployer (but not sticked to this).

If we try to auto-detect, we need to periodically issue a kind of keep-alive message. It might be useful to check the aliveness of backend apps from rpxy side though... I am not sure if such keep-aliving is always better for any kind of backends in any deployment situations.

@akostadinov
Copy link
Contributor Author

It makes sense for performance reasons to make user specify the connection type. But it has to be well documented what would be the default behavior. I didn't see this when reading the readme or the example config.

HTTP2 upgrading when using HTTPS on the upstream should be straightforward because that negotiation happens within the TLS handshake. I think that auto-detection is no issue. More of an issue is the HTTP3 upgrade.

Actually the ideal approach would be for rpxy to negotiate HTTP version automatically but then keep the highest negotiated version in memory so that subsequent connections will directly use that version instead of the auto-negotiation algorithm.

For the time being though, just documenting this would be super helpful for new users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants