Forward proxy implementation

[evgenyrud]

I need to implement forward proxy which receives http requests from existing client which can not change

1. Receive request from client
2. Modify json in body
3. Forward request to another upstream server with different ip, port and URL path

Also upstream server might send http post/get messages back to client

Implemented ModifyPostDataPlugin and proxy.py routed request to upstream server and got reply back.
It works only with command like:
curl -v -x localhost:8899 http://127.0.0.1:8000/connect -d '{"key1": "value1", "key2": "value2"}'

But since my client doesn't know what is proxy server and can't be changed - my request should be like
curl -v -x http://127.0.0.1:8899/connect -d '{"key1": "value1", "key2": "value2"}' and this doesn't work.
and proxy should convert JSON and forward request to upstream server

How my goal can be achieved? tried to implement with different plugins but with no luck.

Please help

Thanks,
Evgeny

[abhinavsingh]

@evgenyrud Hi. Afaik, -x http://127.0.0.1:8899/connect is an invalid protocol use-case because -x documentation says -x, --proxy [protocol://]host[:port]. As you can see, proxy requests don't work on paths e.g. /connect cannot be part of the proxy address.

Let me know if I have understood it incorrectly.

Best

[evgenyrud]

Hi,

1. Can proxy.py forward messages to backend server when client makes http request like
   curl -v -http://127.0.0.1:8899/connect -d '{"key1": "value1", "key2": "value2"}' (without actually specifying who is proxy server with -x parameter)

and ip and port of backend server will be set in proxy.py plugin?

2. Which plugin shall be implemented to support message forwarding in both directions:
   a. Client to backend server
   b. Backend server to client

I need to be able to manipulate on body (json) and url to forward message

Thanks,
Evgeny

[evgenyrud]

Is there some option on one request to proxy create several requests to backend server and then reply to client?

[abhinavsingh]

Hi @evgenyrud

Yes, what you are trying to achieve can be implemented with proxy.py, just that, since your requirement are specific to your use-case, you will have to work around it. Here is how you can achieve the same:

1. You want your clients to send direct request to the proxy.py web server
   (not proxy server as there is no -x as we discussed).
2. Then, within the web server, you will process the data as you wish and return back a response.
3. Start by writing a custom web server plugin which will internally act much like the reverse proxy plugin.
4. This custom plugin will accept client request,
   do the needful (e.g. connect with upstream for a response)
   and finally send it back data to the client.

Let me know if this makes sense and is useful for your use case.

Best
Abhinav

[evgenyrud]

It makes sense. Thanks

[evgenyrud]

Hi,

I need to store some key from upstream server reply and use it with each next http request to be sent to upstream server.
What would be the best solution?

I see that on every http request new plugin instance is created so neither static nor global plugin variable don't work for me.

I'm implementing custom web server plugin.

Thanks

[abhinavsingh]

1. You can certainly use global variables to share states between 2 instances of a plugin
2. But not, such global variables are not thread safe by default. You must use thread safe data structure for this.
3. Alternately, you can also write a some file on disk (or redis, memcache, some KV store) and have a lock around it for sharing data

[evgenyrud]

Hi, I thought about file as alternative. Regarding global variable, where shall I define it? I tried to define it in plugin class with default value None and set it in handle_route method and then return reply to client. The next time when http request comes from client I check the variable and it’s value is None. Thanks, Evgeny

…

On Tue, 16 May 2023 at 9:34 Abhinav Singh ***@***.***> wrote: 1. You can certainly use global variables to share states between 2 instances of a plugin 2. But not, such global variables are not thread safe by default. You must use thread safe data structure for this. 3. Alternately, you can also write a some file on disk (or redis, memcache, some KV store) and have a lock around it for sharing data — Reply to this email directly, view it on GitHub <#1332 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN7SR6TE33SMWGRZM2KDL3DXGMNW5ANCNFSM6AAAAAAXUEHBCI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[abhinavsingh]

Yes file based is a good approach here instead of dealing with global variables 🗡️