OONI Probe CLI v3.17.0-beta.1

• 🚧 all: update dependencies and assets
• 🚧 dash: migrate to m-lab locate services v2
• 🐛 dependencies: upgrade pion/dtls/v2
• 🐛 oohelperd: use cached cert pool

Full Changelog: v3.17.0-beta...v3.17.0-beta.1

OONI Probe CLI v3.17.0-beta

This beta release fixes running tor on Android and the CNAME generated by getaddrinfo.

Full Changelog: v3.17.0-alpha.1...v3.17.0-beta

OONI Probe CLI v3.17.0-alpha.1

Overview

This release should be the solid starting point for the 3.17.x train of releases. We will test it next week. If everything is fine, we can then move forward with creating v3.17.0 and with updating all the apps.

What's Changed

✨✨✨ We rewrote the build procedure in Go. We did this for two reasons. The first reason is unit testing. We can now be confident that the build procedure invokes the correct commands. The second reason is that we wanted to compile tor and its dependencies for Android using hardening flags. We implemented these changes in these pull requests:

• feat: simplify writing tests for code using shellx by @bassosimone in #1042
• refactor: factor common code to check whether file exists by @bassosimone in #1045
• feat(shellx): support incremental command construction by @bassosimone in #1046
• feat(runtimex): add the TryN functions by @bassosimone in #1047
• feat(must): convenience package with "must" functions by @bassosimone in #1048
• feat: rewrite darwin build rules in Go by @bassosimone in #1049
• feat: rewrite windows build rules in Go by @bassosimone in #1050
• feat: rewrite generic build rules in Go by @bassosimone in #1051
• feat: rewrite linux build rules in Go by @bassosimone in #1044
• feat: mechanism compile and embed tor by @bassosimone in #1052
• feat: rewrite the android build in Go by @bassosimone in #1053
• feat: convert ios build to use Go by @bassosimone in #1054
• feat: build for Android using Go by @bassosimone in #1055
• cleanup: ghgen should be in internal/cmd by @bassosimone in #1056
• refactor(buildtool): unify C deps with other deps by @bassosimone in #1057
• refactor(buildtool): merge cdeps and android build envs by @bassosimone in #1058
• refactor(buildtool): use naming reminding of bash variables by @bassosimone in #1059
• refactor(buildtool): improve how we set environment variables by @bassosimone in #1060
• feat: cross-compile tor for android using the Go builder by @bassosimone in #1061
• feat(buildtool): add unit tests for android builds by @bassosimone in #1062
• feat: use internal/libtor for android builds by @bassosimone in #1063
• chore: update to ooni/go-libtor v1.1.7 by @bassosimone in #1064
• doc(buildtool): explain ANDROID_NDK_HOME for android gomobile by @bassosimone in #1066

✨✨✨ We continued to improve and modernize the code that communicates with the OONI backend. In particular, in this release, we added new code that allows us to perform A/B testing of the Web Connectivity implementation. This change will allow us to start testing the new Web Connectivity implementation. We will take advantage of this functionality to start implementing and testing throttling measurements inside Web Connectivity, which we will work on soon. In addition, the changes we implemented simplify providing more complex inputs to experiments. (The fact that we can tell the probe which version of Web Connectivity to run is indeed a special case of providing more complex input.) We did this work in:

• refactor: rename check-in nettests results for clarity by @bassosimone in #1036
• refactor(session): observe full check-in response by @bassosimone in #1037
• fix(httpapi): don't return nil, nil with null JSON input by @bassosimone in #1038
• feat: enable A/B testing using check-in API by @bassosimone in #1039

✨✨✨ We added support for exposing build information. This gives us extra confidence about the specific version of the measurement engine being used for collecting a measurement, which is great for data quality assessments. We did this work in:

• feat: expose build information by @bassosimone in #1030

🐛🐛🐛 We fixed a bug where Web Connectivity could be tricked to fetch from localhost:

• fix([email protected]): stop fetching from localhost by @bassosimone in #1027

🚧 🚧 🚧 We completed a long standing refactoring by moving internal/engine packages to internal. It's good to do this right before making a new release train (3.17.x) because it simplifies backporting patches if needed:

• refactor: prepare for renaming all experiments by @bassosimone in #1032
• refactor: continue moving packages outside of internal/engine by @bassosimone in #1031
• refactor: move all experiments into internal/experiment by @bassosimone in #1033
• cleanup: finish moving packages away from engine by @bassosimone in #1034
• chore: update readme files and perform minor cleanups by @bassosimone in #1035

🚧 🚧 🚧 We did the usual grunt work associated with preparing a release: updating dependencies and cleaning up stuff. We specifically did this work in the following pull requests:

• cleanup: use sync/atomic instead of atomicx by @bassosimone in #1028
• chore: use go1.19.5 by @bassosimone in #1029
• feat(engine): make netxlite.utlsConn public by @kelmenhorst in #1018
• chore: update dependencies by @bassosimone in #1041
• fix: attempt to make all workflows green by @bassosimone in #1065

Full Changelog: v3.17.0-alpha...v3.17.0-alpha.1

OONI Probe CLI v3.17.0-alpha

Changes since v3.16.0-alpha.3

• ✨ ooniprobe: add the --proxy command line flag which works like miniooni --proxy
• ✨ miniooni and ooniprobe: add support for the --proxy torsf:///, which uses snowflake over tor
• ✨ miniooni: add --snowflake-rendezvous CLI flag to specify the snowflake rendezvous policy
• ✨ miniooni: read OONI Run v2 descriptors form the filesystem with miniooni oonirun -f FILE
• ✨ oohelperd: add support for measuring HTTP/3
• ✨ portfiltering: new experiment for measuring port blocking
• ✨ echcheck: new experiment for measuring Encrypted Client Hello blocking
• ✨ tlsmiddlebox: new experiment implementing iterative network tracing for TLS
• ✨ [forwardport] webconnectivity: update LTE implementation to v0.5.19
• ✨ geoipx: use ASN+Country database generated using ooni/historical-geoip
• ✨ httpapi: new API allowing to fallback when multiple endpoints are available
• ✨ measurexlite: add refraction-networking/utls support
• ✨ probeservices: support compressed check-in API responses
• ❗ [API BREAK] oonimkall: drop deprecated APIs
• ❗ [API BREAK] oonimkall: rename CheckInConfigWebConnectivity's Add method to AddCategory
• ❗ [ABI BREAK] model: use udp rather than quic as the protocol (see #946 for details)
• 🐛 psiphon: make sure we include a config when building for Linux
• 🐛 [forwardport] signal: update the embedded signal CA
• 🐛 webconnectivity: increase robustness by trying to use all available TH before giving up
• 🐛 whatsapp: stop measuring http://web.whatsapp.com/ to avoid false positives
• 🐛 geolocate: make sure we use the session resolver for consistency
• 🐛 measurexlite: emit the resolve_start and resolve_done events
• 🐛 model: mark optional fields as optional and sync up with ooni/spec's data formats
• 🐛 [forwardport] model: improve measurements scrubbing
• 🐛 netxlite: make sure we wrap DNS decoding errors
• 🐛 scrubber: merge improvements from the snowflake codebase
• 🚧 [forwardport] telegram: stop measuring http://web.telegram.org/ to avoid potential false positives
• 🚧 .github/workflows: automatically generate several github workflows files
• 🚧 all: use go1.19.4 and update dependencies
• 🚧 QA: simplify quality assurance scripts and focus on web connectivity only
• 🚧 Makefile: add generic rules for building ooniprobe and miniooni for the current GOOS/GOARCH
• 🚧 MONOREPO: merge monorepo-like scripts to facilitate integration testing
• 🚧 ooniprobe: make database code abstract to facilitate subsequent refactoring
• 🚧 internal: change way in which we provide arguments to OONI experiments
• 🚧 probeservices: start using the httpapi package
• 🚧 probeservices: remove support for deprecated APIs
• 🚧 httpx: deprecate in favour of httpapi
• 🚧 model: reintroduce netxlite's underlying network functionality

Entries marked as [forwardport] lift patches from the latest stable releases in the v3.16.x series.

We don't need to bump the major version number because API changes in pkg/oonimkall do not influence the major version number.

New Contributors

• @arky made their first contribution in #967
• @ooninoob made their first contribution in #987
• @d1vyank made their first contribution in #970

Full Changelog: v3.16.0-alpha.3...v3.17.0-alpha

OONI Probe CLI v3.16.7

This patch release includes the following features and fixes:

• [backport] fix(whatsapp): stop measuring http://web.whatsapp.com/ (#998) 4329b89
• [backport] fix(telegram): stop measuring http://web.telegram.org/ (#999) 7a03da8
• [backport] doc(whatsapp): explain the tricky >= 2 check (#1000) e3bf526
• [backport] fix(whatsapp): allow registration server to HTTP-fail (#1001) 1c99427

OONI Probe CLI v3.16.6

This patch release includes the following features and fixes:

• [backport] refactor([email protected]): improve logging clarity (#964): ac423f6
• [backport] doc([email protected]): improve readme: d5b3ab9
• [backport] doc([email protected]): link to analysiscore.go: dc9ff91
• [backport] fix([email protected]): limit number of redirects (#965): 9bf6e38
• [backport] feat(webconnectivity): try all the available THs (#980): 25d8c86
• fix(QA): reckon with TH fallback policies: 1de058d

OONI Probe CLI v3.16.5

This release contains the following fixes:

• 🐛 Fix the signal root CA

Full Changelog: v3.16.4...v3.16.5

OONI Probe CLI v3.16.4

This release contains the following fixes:

• [backport] fix(go-build-alpine): honour OONI_PSIPHON_TAGS

• [backport] fix: stop using deprecated ams-pg.ooni.org domain name

• [backport] Add new root CA to the signal test

Full Changelog: v3.16.3...v3.16.4

OONI Probe CLI v3.16.3

This patch release contains minor fixes as well as improvements for the experimental [email protected].

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

Full Changelog: v3.16.2...v3.16.3

OONI Probe CLI v3.16.2

This release contains changes from the master branch that it felt okay to backport to release/3.16. Because the master branch does not include refactoring yet, it feels safe to backport fixes and improvements, while skipping entirely new features.

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

Full Changelog: v3.16.1...v3.16.2