add regression tests for mitmproxy support

[msridhar]

We need regression tests for our mitmproxy support that run on Travis. Ideally this support would also test the certificate installation process. Not sure if this is possible.

[csujedihy]

mitmproxy crash when accessing mtalk.google.com. Just a heads up.

[msridhar]

@csujedihy can you try with the latest installation instructions in the README? In particular, you should use mitmproxy version 0.11.3. Before trying, please do pip uninstall mitmproxy and rm -rf ~/.mitmproxy. You should also remove any mitmproxy certificates you placed in the Keychain (if on Mac, otherwise the appropriate Linux thing) and reinstall freshly-generated certificates.

[csujedihy]

My mitmdump's version is 0.18.2.

[csujedihy]

I got it. You modified the README recently.

[msridhar]

Yes, just this morning 😄 mitmproxy 0.17 might work, 0.18.2 definitely doesn't. This is the motivation for this issue

[msridhar]

I have confirmed that jalangi2analyses do not work out of the box with mitmproxy 0.17 installed. This may just be updating which APIs get invoked, but I also see various errors in the console about the client not trusting the proxy's certificate. So there may be a mitmproxy problem here.

[msridhar]

@christofferqa have you gotten mitmproxy 0.17 to work with HTTPS sites on Mac?

[christofferqa]

Yes. Have you marked the mitmproxy certificate as trusted in Keychain Access?

[csujedihy]

0.17-0.18.2 will crash when visiting mtalk.google.com via HTTPS.

[msridhar]

@csujedihy does 0.11.3 work on mtalk.google.com? Maybe we should report this one upstream.

[christofferqa]

I don't experience the problems you are having with 0.17.1 (https://github.com/mitmproxy/mitmproxy/tree/v0.17.1). My steps were as follows (assuming mitmproxy has been installed properly).

1. Start mitmproxy.

   mitmdump --quiet --anticache -s "scripts/proxy.py --inlineIID --inlineSource --analysis src/js/sample_analyses/ChainedAnalyses.js --analysis src/js/runtime/analysisCallbackTemplate.js"
   

2. Open Google Chrome with the proper proxy flags.

   Mac OS:

   open -a 'Google Chrome' --args --proxy-server="127.0.0.1:8080" --proxy-bypass-list=""
   

   Ubuntu:

   google-chrome-stable --proxy-server="127.0.0.1:8080" --proxy-bypass-list=""
   

3. Open https://hangouts.google.com. (I tried opening mtalk.google.com, but it is not reachable for me.)

Below is a screenshot of Jalangi2 running on Google Hangouts (on HTTPS). (Uploaded to http://users-cs.au.dk/cqa/jalangi2/hangouts.png.)

Below is a screenshot of Google Hangouts (on HTTPS), when visited using mitmproxy 0.17.1, but without the Jalangi2 instrumentation. (Uploaded to http://users-cs.au.dk/cqa/jalangi2/hangouts-no-instr.png.)

[msridhar]

Christoffer, can you clarify the exact installation steps you use for mitmproxy?

…

On Sun, Dec 11, 2016, 9:00 AM christofferqa ***@***.***> wrote: I don't experience the problems you are having with 0.17.1 ( https://github.com/mitmproxy/mitmproxy/tree/v0.17.1). My steps were as follows (assuming mitmproxy has been installed properly). 1. Start mitmproxy. mitmdump --quiet --anticache -s "scripts/proxy.py --inlineIID --inlineSource --analysis src/js/sample_analyses/ChainedAnalyses.js --analysis src/js/runtime/analysisCallbackTemplate.js" 2. Open Google Chrome with the proper proxy flags. Mac OS: open -a 'Google Chrome' --args --proxy-server="127.0.0.1:8080" --proxy-bypass-list="" Ubuntu: google-chrome-stable --proxy-server="127.0.0.1:8080" --proxy-bypass-list="" 3. Open https://hangouts.google.com. (I tried opening mtalk.google.com, but it is not reachable for me.) Below is a screenshot of Jalangi2 running on Google Hangouts (on HTTPS). (Uploaded to http://users-cs.au.dk/cqa/jalangi2/hangouts.png.) [image: Jalangi2 on Google Hangouts] <https://camo.githubusercontent.com/2c44733c416d20ba75cadc5e577656634061ca1f/687474703a2f2f75736572732d63732e61752e646b2f6371612f6a616c616e6769322f68616e676f7574732e706e67> Below is a screenshot of Google Hangouts (on HTTPS), when visited using mitmproxy 0.17.1, but *without* the Jalangi2 instrumentation. (Uploaded to http://users-cs.au.dk/cqa/jalangi2/hangouts-no-instr.png.) [image: Jalangi2 on Google Hangouts] <https://camo.githubusercontent.com/10d2a929f1edff35e27598df2c89099aa2c72ce0/687474703a2f2f75736572732d63732e61752e646b2f6371612f6a616c616e6769322f68616e676f7574732d6e6f2d696e7374722e706e67> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#124 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyUUtPjGTU0WFN6N2jgqSJqeNMcaXWks5rHCwegaJpZM4LEos5> .

[christofferqa]

Yes, but I'll need access to a Mac, where mitmproxy has not previously been installed, to be sure that I don't have some hidden dependency installed. It may take a few days.

[msridhar]

When you do get to a Mac, let us know if it's running El Capitan or Sierra. My issues have been on Sierra; maybe that's the problem.

…

On Sun, Dec 11, 2016 at 10:41 AM christofferqa ***@***.***> wrote: Yes, but I'll need access to a Mac, where mitmproxy has not previously been installed, to be sure that I don't have some hidden dependency installed. It may take a few days. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#124 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyUVUeJmb-NBrHs1jA8SjJbEJ4R9A2ks5rHEPGgaJpZM4LEos5> .

[christofferqa]

Complete jalangi2 install instructions for OS X 10.10. Tested on an entirely fresh virtual machine, installed from the recovery partition of my Mac (see http://cs.au.dk/~cqa/jalangi2/vm.png).

1. Install Homebrew

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. Install git, node 4.x, python

$ brew install git
$ brew install node@4
$ brew install python

3. Update pip and install mitmproxy 0.17

$ pip install --upgrade pip
$ pip install mitmproxy==0.17

4. Install Google Chrome from https://www.google.com/chrome

5. Install mitmproxy certificates

   A. Start mitmproxy and open http://mitm.it/ in Chrome, with the proper proxy flags (make sure not to have a running instance of Chrome!)

   $ mitmdump -p 8081
   $ open -a 'Google Chrome' --args http://mitm.it/ --proxy-server="127.0.0.1:8081" --proxy-bypass-list=""
   

   B. Click on the Apple icon in order to download "mitmproxy-ca-cert.pem", and then open "mitmproxy-ca-cert.pem" to install the certificate

   C. Open Keychain Access, go to the "Certificates" menu, right-click on the "mitmproxy" certificate and select "Get Info". In the window that opens, click on the "Trust" item, and set "When using this certificate:" to "Always Trust".

6. Clone jalangi2 and install dependencies

$ git clone https://github.com/Samsung/jalangi2.git
$ (cd jalangi2; npm install)

7. Start jalangi2

$ cd jalangi2
$ mitmdump -p 8081 --quiet --anticache -s "scripts/proxy.py --inlineIID --inlineSource --analysis src/js/sample_analyses/ChainedAnalyses.js --analysis src/js/runtime/analysisCallbackTemplate.js"

8. Open Chrome with the proper proxy flags (make sure not to have a running instance of Chrome!), and visit your favorite HTTPS website

$ open -a 'Google Chrome' --args --proxy-server="127.0.0.1:8081" --proxy-bypass-list=""

[msridhar]

FWIW, mitmproxy is now known to work using the master branch, and we've updated the relevant documentation. We still need regression tests; renamed the issue accordingly