lots of various improvements.. #283
Conversation
add: * LICENSE * crx and xpi build scripts * Firefox ID in manifest - for installing an unsigned xpi in FF Developer edition fix: * webpack config to support building on Windows change: * name of output directory from 'dist' to 'PrivacyPass' * npm 'clean' script to include: - typescript output 'lib' directory - crx and xpi files generated by new build scripts * npm 'sjcl' script to explicitly call perl, rather than depending upon the shell to parse the shebang line
use chrome.i18n to internationalize strings in UI
* update message passing between userscript and background page
* update redux store used by popup
summary:
========
* messages are currently only passed in one direction:
- from userscript to background page (w/ response to callback)
- messages:
1. {clear: true}
- removes all passes for all providers from local storage
- no response
2. {tokensCount: true, providerID: number}
- providerID is:
* optional;
if not included, then the response includes all providers
* the same integer that identifies each unique provider
1 == Cloudflare
2 == hCaptcha
etc..
- response: {[key: string]: number}
* each key is a `${providerID}`
* each value is a count of the number of passes in wallet
- for example:
{"1": 30, "2": 5}
* the data structure used by the redux store to hold state
is identical to the response from the background page
to messages that request 'tokensCount'
* refactor the 'voprf' crypto module into a reusable class,
so each provider can own a distinct instance
* add the dependency: 'keccak'
- v2 of the extension included a snapshot of this library
- v3 removed it, but there is still code that wants to use it
* presumably, this branch of code will get called
when the "hCaptcha" provider is reimplemented (..coming soon)
* somewhere between a bug fix and a performance optimization:
- pass an instance of the 'voprf' crypto class
from provider(s) to the Token constructor
- otherwise, each Token initializes its own instance
of the 'voprf' crypto class using default parameters
* hCaptcha provider is added, but not yet functional
status:
=======
* "handleBeforeRequest()" detects when
a captcha is solved on an issuing domain
* "issue()" makes a subsequent request
for tokens to be signed
* the response data does include "signatures",
which can be properly parsed:
{
sigs: string[];
proof: string;
version: string = "1.0";
}
to do:
======
* the code currently expects that the commitment
for the version to be formatted the same as Cloudflare:
{H: string; expiry: string; sig: string;}
* however, the JSON data file:
https://raw.githubusercontent.com/privacypass/ec-commitments/master/commitments-p256.json
shapes the data for version HC["1.0"] differently:
{H: string; G: string;}
* annecdotally, the part that hurts most..
is that this data file includes a version HC["1.01"]
that is shaped in a way that is consistent and would work
* hCaptcha provider is added, but not yet functional
status:
=======
* "handleBeforeRequest()" detects when
a captcha is solved on an issuing domain
* "issue()" makes a subsequent request
for tokens to be signed
* the response data does include "signatures",
which can be properly parsed:
{
sigs: string[];
proof: string;
version: string = "1.0";
prng?: string = undefined;
}
* the version "1.0" commitment is shaped:
{H: string; G: string;}
which, unlike Cloudflare, does not require verification
* since "prng" is not defined,
it should default to "shake",
as was the methodology used by the v2 extension
to do:
======
* the "shake" prng has a missing dependency: Buffer
|
here are releases that include unsigned packed extensions.. for browsers that allow them to be installed ex:
|
|
|
After all thank you for the patch 😄 |
| @@ -38,7 +38,7 @@ $ npm ci | |||
| $ npm run build | |||
| ``` | |||
|
|
|||
| After that, the `dist` folder will contain all files required by the extension. | |||
There was a problem hiding this comment.
Why did you change the name to PrivacyPass? I think the name PrivacyPass doesn't indicate that the files in the directory are generated files.
There was a problem hiding this comment.
because when Chrome is used to pack an extension, both the name of the .crx and the .pem are expected to be the same as the input directory (all siblings)..
.gitignore allows the .pem to be permanently homed where Chrome expects it to be.. without ever being commit to version control.. and likewise, the generated .crx and .xpi files won't either
this naming convention is associated with the scripts in the .bin directory.. if you already have tools that you use to package your extensions, then.. you:
- won't want/need the
.bindirectory- though I find these scripts useful
- won't want to rename the output directory.. as you tooling will already be configured to use the old path
There was a problem hiding this comment.
another option.. which might be preferable.. would be to:
- move the
.bindirectory todist/.bin - configure webpack to output to
dist/PrivacyPass - configure
.gitignoreto exclude:dist/PrivacyPassanddist/PrivacyPass.[pem|crx|xpi]
so all output.. and my scripts to pack the output into extensions.. would be confined under dist
There was a problem hiding this comment.
update:
That last suggestion seemed (to me) as a much cleaner option.. so I just pushed a commit to make those changes.
|
I think many directory names and files names are unnecessary too long. For example, |
| const cached = this.getStoredTokens(); | ||
| this.setStoredTokens(cached.concat(tokens)); | ||
| // delay the request to issue tokens until next tick of the event loop | ||
| setTimeout( |
There was a problem hiding this comment.
Why do you change to setTimeout over here?
There was a problem hiding this comment.
because (if memory serves) that's a blocking API method.. and there's no need for the browser to wait for the secondary request to finish; using a next tick allows the triggered code to run asynchronously.
maybe there's a good reason to not do it this way, but it struck me as a better idea while I was touching that portion of code.
There was a problem hiding this comment.
actually.. my answer just now was wrong..
(async () => {...})()wouldn't block..
I think my reason for adding the timeout in the first place had to do with hCaptcha. I don't entirely understand how their process works.. but it seems (from trial and error) very clear that the intercepted request needs to complete before the triggered request to issue tokens can be made.. otherwise it fails. So.. i don't know if removing this timeout would break hCaptcha or not.. but there's a good chance that it might.
There was a problem hiding this comment.
Of course.. these comments about hCaptcha don't apply to the plugin for the Cloudflare provider; the update to Cloudflare was just because I wanted the 2x plugins to keep as nearly identical as possible.. for symmetry.. so any differences in logic can be easily grok'd by diff'ing the two.
There was a problem hiding this comment.
update:
Regarding my earlier comment/assertion that hCaptcha requires the timeout.. and also..
Regarding my earlier comments about testing the ES5-transpiled release of the extension in older browsers..
I just ran a very interesting test that addresses both topics:
- added a polyfill library to the public directory and loaded it from both the background page and the popup
- using Chrome 30:
- packaged the extension to a crx2 and installed
- no errors or warnings.. everything looks good
- solved an hCaptcha and inspected network traffic
- request from normal window tab to solve captcha: 200
- request from background page to issue tokens: 403
{"success":false,"error-codes":["invalid-data"]}- which is the response that occurs when the request to issue tokens happens concurrent to the POST to solve
- packaged the extension to a crx2 and installed
- made a quick change to the timeout in the background page: increased from
0to2500milliseconds- repacked the crx2 and reinstalled
- solved another hCaptcha and... earned 5x tokens
- it works in Chrome 30! ..how cool is that?
- the Cloudflare provider doesn't work out-of-the-box
- haven't looked into it yet
- guessing it's because the TLS certificate isn't trusted by this browser
- which would make the necessary
window.cryptoAPI unavailable - I'll need to add its root authority.. and retest..
- which would make the necessary
- update:
- the problem with the TLS certificate isn't that the root CA is untrusted.. it validates OK
- the problem is actually that the generic top-level domain used by the CF website that issues tokens (captcha.website) isn't recognized by the browser as a public domain.. here is a related thread on this exact error message
- update:
- the TLS certificate might not actually be the cause for CF not working..
- the only
window.cryptoAPI needed is available from a non-secure context- I was originally thinking of
window.crypto.subtle.. which I was using recently in another project
- I was originally thinking of
- the network trace doesn't seem to show any requests that match the criteria used by the extension to trigger an issue request..
- I'll need to debug.. and see if the code to trigger that secondary request is actually firing
- confirmed:
- the PrivacyPass chrome extension is working
- the issue is that Cloudflare captchas don't work in this browser
- presumably, whatever version of Chrome/FF that is minimally supported by the website/service will also work with the extension (w/ ES6+ polyfills)
There was a problem hiding this comment.
I just added another commit to this branch.. it's tagged as v3.4.0 with a corresponding crx under releases
- adds hook functions that delegate to providers for:
- reverts the Cloudflare provider to remove the previously added timeout
- refactors the hCaptcha provider to move the code that sends issuing requests
- from a timeout in 'handleBeforeRequest'
- ie: the hook for
chrome.webRequest.onBeforeRequest
- ie: the hook for
- to 'handleOnCompleted' and 'handleOnErrorOccurred':
- ie: the new hooks for
chrome.webRequest.onCompletedandchrome.webRequest.onErrorOccurred, respectively
- ie: the new hooks for
- from a timeout in 'handleBeforeRequest'
- updates tests to adjust for revisions
- all pass
this design is:
- much cleaner
- much more reliable, since the timing of the secondary request depends upon completion of the 1st
the result is:
- stable and fully functional
There was a problem hiding this comment.
PS: this new release was built using the improved build scripts, which automates the insertion of the ES6+ polyfill library (that I mentioned previously) into the CRX2 extension.. so
- the CRX3 does not include polyfill
- since it only runs in newer browsers
- the CRX2 does include polyfill
- since it only runs in legacy browsers
- this backport adds support for some crazy old browsers..
- the primary bottleneck now is whether or not the websites operated by each provider (ie: where captchas are solved) will also support the older browsers
|
|
||
| const CHL_BYPASS_SUPPORT = 'cf-chl-bypass'; | ||
| const DEFAULT_ISSUING_HOSTNAME = 'captcha.website'; | ||
| const ALL_ISSUING_CRITERIA: { |
There was a problem hiding this comment.
Making the criteria programmable is more flexible than making it configurable. Please read https://blog.cloudflare.com/privacy-pass-v3/ for more detail. We had difficultly by making things configurable in the past because configuration is less flexible.
That is, if we want to change the criteria, it makes more sense to change the code in some function rather change some configuration object.
There was a problem hiding this comment.
I absolutely agree with you..
- that it would be bad to:
- only allow providers to supply data used to configure global behavior (within narrow limits)
- that it is much better to:
- require that each provider implement a set of hook function
However, since some tasks are repeated by all providers.. it makes sense (to me) to provide tooling in the form of shared static methods that providers have the option to utilize from within their custom hook functions.
Those tools will of course require formatted input that providers who choose to call these methods will need to supply; that's all that these particular data structures are for.
There was a problem hiding this comment.
to make my earlier comment more concrete..
handleBeforeRequest() needs to quickly determine whether the intercepted request:
- solves a captcha belonging to the particular provider
- is being solved on a domain that issues tokens
these determinations are always based on:
- the requested url
- POST data (if present)
I added static methods that allow providers to pass the data from the request as well as a configuration object.. and the result is a simple yes or no.. the request does or doesn't match the filter criteria as configured
If a provider wants to apply specialized logic.. then that's great; simply don't use these methods and roll your own.
|
regarding using long/descriptive file and directory names..
I'm sure that a similar observation/criticism could be made of my choice for using long/descriptive class and variable names..
In both cases, it's a matter of personal preference and whatever conventions the project chooses to use. I prefer descriptive names so I don't have to relearn the code when I look at it months or years later. But, it's not my project.. so do as you like. |
|
regarding whether this is still a work in progress.. without getting too philosophical.. :)
so.. nope; I'd consider it a release candidate. of course, we can discuss some of the decisions and make a few tweaks.. but this definitely moves the ball forward. |
|
One thing that I didn't touch.. but would take a good look at if it was my project.. is the version of the ecmascript in the generated bundle that gets packed into the extension. It's currently set to ES2020. Personally, I always like to support as many users as possible.. and would probably down level all the way to ES5 (if feasible). |
|
PS: regarding my earlier comment about needing to test the tests.. I made an additional commit a few hours ago that fixes the tests (mainly to adjust for how to confirm that the code in the new timeouts has run). Now everything passes. |
|
I just added another commit to this branch.. it's tagged as
If there's a reason that you've chosen to target ES2020.. feel free to exclude this commit. Annecodally, I tested the resulting crx and it works exactly the same. update:
possible cause:
update:
|
|
2x points to bring up (before I forget)..
|
fix: transpile ES6 node modules: 'buffer' and 'keccak'
change: * name of output directory from 'PrivacyPass' to 'dist/PrivacyPass' * location of crx and xpi build scripts from '.bin' to 'dist/.bin' * location of pem input and crx/xpi output files from '/' to 'dist'
* the methodology used in this commit does NOT work
- I fully expected that it would
- it does NOT
- this commit is being tagged so I can point to it later,
if asked: "why didn't you just...?"
* the methodology is to:
- detect requests that solve catchas made to issuing domains
* as was done previously
- rather than make the issuing request then and there,
wait until the corresponding call to "handleHeadersReceived"
* problem:
- there (apparently) is no corresponding call to "handleHeadersReceived"
* to do:
- add new hook for call to "chrome.webRequest.onCompleted"
- add plumbing for all providers to receive this new hook function
- ...hope it receives a corresponding call
- - - -
hCaptcha should send issuing requests from a different hook function
similar to the way redemption works (but in reverse),
detect solved captcha on issuing domains before requests are sent,
but delay the processing of hits until after the response is received
previously, a timeout was used to delay processing;
but this methodology was inexact and could lead to failed attempts.
note that this is in stark contrast to the Cloudflare provider,
which cancels the requests that it detects,
and processes those hits immediately without any delay.
|
Are you using Firefox? Earlier you mentioned using Chrome. My tests with Chrome show a 100% success rate. But.. maybe there are other variables (such as geography, network speed, hardware speed) that could result in a different outcome. Dunno.. |
|
No, Chrome. That was half joking although I definitely get frustrated with the captchas at times. |
|
Oh ok. Well.. personally, I couldn't care less about Firefox.. but this is just super frustating.. because the extension appears to be doing everything right. Without knowing what's going on at the server.. it's just banging my head against a wall. |
|
I understand. Personally, I was somewhat surprised to discover that such an extension/add-on could even have a chance to work. I figured that the captcha people wouldn't like the idea of it and would code defensively against such functionality, even though to me it doesn't seem like much of a threat. It's just that they're all about gatekeeping. |
|
From the first sentence of the README:
This project isn't about finding a clever way to work around the captcha providers. My understanding is that they want this solution to work, and are in collaboration to establish a standard. To do so doesn't make their services obsolete.. quite the opposite; it makes their services more transparent.. less invasive.. and more likely to be used ubiquitously across the interweb. |
|
Yes, I get that, but I was rather surprised when I first found out about it. I'm just saying that security types tend to be on the cautious side. |
|
@MikeGitAZ ..I just tested the 3x URLs that have been listed in this conversation from a VPN server in São Paulo, Brazil:
|
This comment has been minimized.
This comment has been minimized.
|
disregard my last comment..
I'm not even sure if cookies are used to authenticate the 2nd request..
|
summary of what does consistently work correctly (v3.6.6):
summary of unresolved issues (v3.6.6):
notes:
|
During some real-world testing, I noticed that Chrome doesn't always parse 'application/x-www-form-urlencoded' data in the POST body. This new helper method is available to providers to normalize the format of this data into a key/value hash object. Furthermore, it will also parse 'application/json' data.
previous methodology:
=====================
* popup window:
- dynamically creates an input file element
- click event triggers the element to open the file chooser dialog
- processes the list of selected files by:
* reading the content of each file
* parsing its JSON
* passing the resulting object in a message to the background page
* background page:
- merges backup data with the tokens already saved in local storage
- updates local storage with this new aggregate value
- updates the popup window
problem with previous methodology:
==================================
* only works if the browser doesn't close the popup window
before the data is passed to the background page
* many browsers do close the popup window
when the file input dialog is triggered for selection of input files
new methodology:
================
* popup window:
- sends a message to the background page
* background page:
- opens a new tab and loads a static html page
* static html page:
- dynamically creates an input file element
- adds a click event handler to the element,
which requires user interaction to trigger
- the click event handler processes the list of selected files by:
* reading the content of each file
* parsing its JSON
* passing the resulting object in a message to the background page
- the click event handler also tracks the count of files pending
* after the processing of all files is complete,
sends a final message to the background page
* background page:
- merges backup data with the tokens already saved in local storage
- updates local storage with this new aggregate value
- closes the tab containing the static html page
comparison between methodologies:
=================================
* previous methodology:
- pros:
* simpler implementation
* more elegant user experience
- cons:
* doesn't work in many browsers
* new methodology:
- pros:
* works in all supported browsers
- cons:
* much more complicated implementation
* less elegant user experience,
which requires interaction with a standalone page in a new tab
|
Hi @warren-bank, Since this PR is quite huge and it touches the code structure of the project which needs discussion, would you mind if I ask you to create a smaller PR first? (For example, the PR which only makes hCaptcha tokens work) Another one is could you make the commit message up-to-date? for example, "work in progess.." means that the work is not finished yet. There are many things that should be done by the maintainer, such as updating the version number. I'm sorry for the delayed response and thank you for the PR. |
Though I have noticed that a few of my more minor tweaks have already been cherry picked into standalone PRs that have already been merged.. the trouble with this request is that none of the more serious aspects of the code can live in a sandbox, isolated from the rest of the project. Using the hCaptcha provider plugin as an example:
These commit messages and version numbers were made during a period of very active development in a separate fork. I never assumed that the dev branch would be merged verbatim. If you were to perform a squashed merge, then:
My point is.. that none of this should matter. Do you think it would be easier to discuss and selectively undo the parts you don't like? |
What if we add the parts incrementally with small PRs. I can spot some features that are, by themselves, good additions for the extension:
@warren-bank would you like to create PRs addressing specific changes, or are you ok if we cherry pick changes from your fork? |
you can cherry pick code from the fork, if:
regarding your ability to incrementally add features, rather than lump sum..
I'm absolutely willing/eager to collaborate; |
probably not perfect, but hopefully a good starting point; PRs are welcome.
warren-bank
force-pushed
the
PR/v3-rc
branch
from
4a3a979 to
7057859
Compare
* each row of providers contains: name, token count
- previously, 2x floating divs were used for layout
* there were no css rules to properly display text overflow
* when only english strings were displayed,
this was acceptable
* when other language translations are displayed,
this goes wrong very quickly
- now, a simple 1x row by 2x column table element is used for layout
* though a flexbox layout would be perfectly suited,
it's only supported by newer browsers
* table elements have always existed,
and it works equally well
example:
========
* run Chrome using strings translated for German locale
- bash:
LANGUAGE='de' && chrome --lang=de
- cmd in Windows:
set "LANGUAGE=de" && chrome --lang=de
previously, request to issue tokens required querystring parameter: __cf_chl_captcha_tk__ now, the name of this parameter has been changed to: __cf_chl_f_tk
the detection criteria allows for any of several parameter names, but the backend endpoint to issue Cloudflare tokens only accepts a single parameter name.
both in terms of making the design simpler for adding new providers..
and in terms of adding features (such as re-adding support for the hCaptcha provider).
status:
to-dos:
although I updated the tests, I haven't actually run them yet.. so I'm sure they'll need an hour or two to get everything passing