Proposal to support finer grained behavior control

[mfschwartz]

I'm considering implementing a new HTTP behavior that would make vaurien capable of finer-grained error case simulation, which we could use for deeper integration testing of gsutil - for example, rather than just randomly picking a 5xx error code x% of the time, I'd like to be able to configure the proxy to pass (say) 100KB without errors, then send a particular error code, then pass the rest of the upload/download without errors. Or more complicated sequences, like 100KB without errors, then a code that forces gsutil to start the upload/download over from scratch, then (after the restart) 200KB without errors followed by an error code that requires retrying but starting at the same offset, and then the rest without errors - etc.

I'm thinking the config could be specified using a JSON object such as:
{ traffic_spec: [ pass: 100k, error: {code:503, text: "slow down"}, pass: * ] }

In the future someone could extend this functionality to mix in other behaviors (e.g., so that while bytes are being passed delays or packet drops are included).

What do you think of this idea?

[tarekziade]

Ah sorry, I just answered by e-mail :)

I like the idea a lot - I think the hardest part is to make sure we can express the failures scenarios in a simple way in the configuration, because what you're describing is starting to get quite complex.

If you can propose an implementation in a PR, I guess it's a good step forward to see how it can look

Thanks a lot !

[mfschwartz]

Hi Tarek,

I realized more work will be needed beyond what I described above:

Right now vaurien only supports perturbing the transmission between the client and a single backend service -- e.g., the command:

./vauenv/bin/python vauenv/lib/python2.7/site-packages/vaurien/run.py --protocol http --proxy localhost:8000 --backend www.googleapis.com:443 --behavior 10:error

will cause requests sent to localhost:8000 to return 5xx errors for 10% of the traffic, and pass the rest through to www.googleapis.com:443. But gsutil needs to send requests to *.googleapis.com (for storage) as well as to accounts.google.com (for authentication), so this simple setup won't work. Instead I think we'd need to make vaurien understand real HTTP proxying, with the client sending absolute-form (http://tools.ietf.org/html/rfc7230#section-5.3.2) requests and the proxy doing the needed DNS lookup, rewriting the request and sending it on to the downstream server, and similarly passing responses back from the server.

So, now I'm wondering whether it makes more sense to add such proxy functionality to vaurien vs to start with a proxy (Squid) and add programmable behavioral controls.

Your thoughts/comments welcome.

Thanks,

Mike