Basics of a caching HTTP(S) proxy

[bra-fsn]

I need to hack up something which selectively caches responses (and returns them for subsequent requests under certain conditions), so if it's not a big deal, could you point me to the right direction on this topic?

What I need is:

1. capture the original request, get its parameters (like the URL, HTTP method, headers etc)
2. if conditions stand in Added request and response logging for debugging reasons #1, I need to return a response (mostly binary files) from a local file object (files can be big, so it's important that they aren't read into their entirety into memory and served from an open file object which is then closed after the transaction)
3. if conditions are not fulfilled, I should let the original request through unmodified
4. on a response from the remote server I need to inspect and store it if needed. Again: responses may not fit in memory, so it's essential that they are spooled to files.
5. if Doesn't work on safari? #4 is not possible, I need to make the HTTP (GET) requests myself, equivalent with step python3 support #2. In this case, my question is: what is the preferred way to do blocking IO from there? If that callback runs in its own thread, that's fine, but with anything else, blocking there will cause problems and I need advice on what to use/how to make a blocking HTTP request (to have the response in a file in a streaming way).

Pointers, or code snippets for the above steps are both welcome!

Thanks,

[abhinavsingh]

@bra-fsn Can you look into --cache-by-content-type flag. I recently added support for caching responses by their content type. When you use cache response plugin with interception enabled, you will be able to see almost all the content being downloaded via HTTP(s) requests.

Serving them back is a little trickier and has not been added in the core. You can at-least use this as a starter and implement on top of it. Part of this functionality is implemented here:

https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/cache/cache_responses.py#L74-L79

PS: Sorry not much doc on this. First enable TLS interception flags, then CacheResponsePlugin flags and finally also add the --cache-by-content-type. You should see json, xml, html, images, videos etc being downloaded in the --cache-dir flag path.

Let me know how it goes.