Skip to content

Releases: ooni/probe-cli

OONI Probe CLI v3.19.0-alpha

10 Oct 16:36
@github-actions github-actions
v3.19.0-alpha
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
29e8c88
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OONI Probe CLI v3.19.0-alpha Pre-release
Pre-release

Summary

This release is an alpha release. It is not ready for general availability and should only be used for QA.

A stable release will follow suit hopefully in a few days from today.

On a high-level, these are the most important changes:

  • ✨ Better Support for Measuring Throttling
  • ✨ Introducing OONI Probe Bridges
  • ✨ Improving Measurements Scrubbing
  • ✨ Netemx: A Better Unit/Integration Testing Framework
  • ✨ OONI Run v2 Support
  • ✨ Unit Testing Improvements (go test -short ./... does not use the host network anymore)
  • 🐛 OONI Probe Bootstrap fixes
  • 🐛 Web Connectivity v0.4 Fixes
  • 🐛 Snowflake Fixes

Read on for more specific details!

✨ New Features

Better Support for Measuring Throttling

These changes implement a better support for measuring throttling where we periodically collect download speed samples during the lifetime of downloads performed using Web Connectivity (or any other network experiment using measurexlite):

Note that, because these changes are measurexlite specific, and because only Web Connectivity LTE uses measurexlite, we would only get the benefit of better throttling measurements once we're fine switching all users to use Web Connectivity LTE. (Currently, 99% of the users are still using Web Connectivity v0.4, since we still have to finish doing A/B testing to make sure that the new version of Web Connectivity is not introducing any regressions compared to version v0.4.)

Introducing OONI Probe Bridges

We added support for OONI Probe Bridges. (We initially named this feature "beacons", which is why many pull requests below mention "beacons".) A OONI Probe Bridge is a host that allows us to connect to the OONI backend or test helpers. The current implementation knows about a single bridge, but it is possible to add more bridges (and we will add more bridges). The important concepts are that the probe knows the bridge IP address in advance and that there's the guarantee we can use pretty much any SNI with the bridge itself, even though, obviously, we use the api.ooni.io verify hostname when verifying the certificate. By default OONI Probe tries using a bunch of predefined SNIs and remembers what works to use it consistently. This change currently only modifies how we create TLS connections with the OONI backend, but we plan on extending the scope of these changes further in subsequent releases.

There is also a mechanism to bypass the probe behavior and force it to use a specific bridge. We're documenting this mechanism here with the understanding that this functionality should be considered as a technical preview and may change in future releases without any need on our end to change the major version number. With that caveat out of the way, with this release of OONI Probe, if you create $HOME/.ooniprobe/engine/bridges.conf, it will be used to override how we communicate with the backend. The file format is JSON with comments, and here is an example:

{
  // The DomainEndpoints field maps a domain's endpoint to a list of bridges
  "DomainEndpoints": {
    "api.ooni.io:443": [

      // A bridge is characterized by an IP address, a Port, a SNI to send on the wire, the
      // hostname to use for TLS certificate verification and a delay.
      //
      // The delay can be useful to stagger dial attempts such that you do not try all
      // the bridges at the same time. The unit of InitialDelay is nanoseconds.
      {
        "Address": "130.192.91.211",
        "InitialDelay": 0,
        "Port": "443",
        "SNI": "www.example.com",
        "VerifyHostname": "api.ooni.io"
      }

    ]
  }
}

There is also a new file $HOME/.ooniprobe/engine/httpsdialer.state that keeps the on-disk state used by OONI Probe to remember which bridges worked. This file also includes information about some bridges that did not work, however we periodically prune this cache to avoid it growing too large in case there are many failures. (We're more interested on what works anyway.)

Support for HTTP/HTTPS proxies

As part of work to introduce OONI Probe bridges, we also removed limitations in the codebase that prevented to use HTTP or HTTPS proxy URLs with ooniprobe --proxy URL and miniooni --proxy URL. We will update mobile apps to allow using this kind of proxies as well. Before these changes, we only supported (a) SOCKS5 proxies and (b) creating proxies using Psiphon, vanilla tor, or tor along with snowflake (respectively, --proxy socks5://<addr>:<port>, --proxy psiphon:///, --proxy tor:///, --proxy torsf:///). Assuming you have, e.g., tor exposing a proxy on port 9080, now you will be able to use ooniprobe --proxy http://127.0.0.1:9080/.

  • ✨ chore: run go fmt ./... by @bassosimone in #1272
  • ✨ feat(testingx): introduce more comprehensive HTTP(S) proxy by @bassosimone in #1274
  • ✨ feat(testingproxy): test HTTP(S) proxies using netem by @bassosimone in #1275
  • ✨ feat(testingx): more tests for NewHTTPProxyHandler by @bassosimone in #1276
  • ✨ cleanup: use testingx.NewHTTPProxyHandler as proxy by @bassosimone in #1277
  • ✨ feat(UnderlyingNetwork): add support for ListenTC...
Read more

Contributors

bassosimone, imw, and 3 other contributors
Assets 35
Loading

OONI Probe CLI v3.18.1

18 Jul 16:07
@github-actions github-actions
v3.18.1
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
4eff308
Compare
Choose a tag to compare
OONI Probe CLI v3.18.1

This patch release includes the following changes since v3.18.0:

  • ✨ webconnectivity LTE: better throttling support e1a63d1 5c1b7d2
  • 🚧 oonimkall: experimental OONI Run v2 API 4eff308
  • 🚧 measurexlite: do not depend on tracex 527d115
  • 🚧 update to go1.20.6 394a273
  • 🚧 update to 2023-07 geoip databases ae93c56
  • 🚧 update all the other deps we can easily update 2d91295

Full Changelog: v3.18.0...v3.18.1

Assets 35
Loading

OONI Probe CLI v3.18.0

27 Jun 08:19
@github-actions github-actions
v3.18.0
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
952ab29
Compare
Choose a tag to compare
OONI Probe CLI v3.18.0

This release adds these minor changes on top of v3.18.0-alpha:

  • 🚧 [backport] chore: use go1.20.5 89d09b1
  • 🚧 [backport] fix(.github): avoid using deprecated ::set-output 29a1ba4
  • 🐛 fix: use correct Xcode and mingw version 8a85b63
  • 🐛 [backport] fix: avoid submitting when tor binary is missing 33e28aa

Full Changelog: v3.18.0-alpha...v3.18.0

Assets 35
Loading

OONI Probe CLI v3.17.5

08 Jun 15:54
@github-actions github-actions
v3.17.5
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
f40d753
Compare
Choose a tag to compare
OONI Probe CLI v3.17.5

Full Changelog: v3.17.4...v3.17.5

Assets 35
Loading

OONI Probe CLI v3.18.0-alpha

06 Jun 13:13
@github-actions github-actions
v3.18.0-alpha
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
78bba71
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OONI Probe CLI v3.18.0-alpha Pre-release
Pre-release

Overview

The main reason for preparing this release is to support go1.20 users.

Here are the main changes since v3.17.0-alpha.1:

New Features

  • ✨ dslx: we introduced an internal DSL for writing experiments #1115 #1116 #1117 #1118 #1119 #1120 #1127 #1128 #1141 #1133 #1142
  • ✨ gardener: a new, experimental tool for curating test lists #1114
  • ✨ netemx: start adding support for writing integration tests using ooni/netem #1121
  • ✨ optional: add the optional type #1131
  • ✨ data quality: handle the TLS unrecognized name alert and map it to FailureSSLInvalidHostname
  • ✨ data format: add a tags: [] field with tags to all data types #1142
  • ✨ data format: add test_helpers field to check-in response

Maintenance

  • 🚧 all: we're now using go1.20.4
  • 🚧 dash: refactor and modernize the implementation, upgrade to m-lab locate API v2 (the v1 is now deprecated)
  • 🚧 oohelperd: disabled QUIC by default but allow enabling it using the TH request
  • 🚧 mlablocate: remove because all code now uses mlablocatev2

Performance

  • 📈 all: use cached Mozilla cert pool most of the time
  • 📈 oohelperd: allow collecting CPU profiles

Bug Fixes

We applied the following fixes (which we also applied to the v3.17.x release train maintenance branch):

  • 🐛 oohelperd: fix test helper twitter.com HTTP measurements by using a cookie jar #1149
  • 🐛 signal: stop using decommissioned endpoint #1135
  • 🐛 webconnectivitylte: avoid data race #1124
  • 🐛 libtor: avoid data race #1132
  • 🐛 libtor: avoid file descriptor leak #1072

New Contributors

Full Changelog: v3.17.0-alpha.1...v3.18.0-alpha

Contributors

simoncpu, mstrahov, and dependabot
Assets 29
Loading

OONI Probe CLI v3.17.4

06 Jun 08:18
@github-actions github-actions
v3.17.4
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
a3af554
Compare
Choose a tag to compare
OONI Probe CLI v3.17.4
  • 🚧 feat: use 2023-06 geoip database 56438f2
  • 🐛 fix(oohelperd): use cookiejar for HTTP measurements a3af554
  • 🐛 fix: use openssl-1.1.1u 325a841

Full Changelog: v3.17.3...v3.17.4

Assets 35
Loading

OONI Probe CLI v3.17.3

31 May 11:43
@github-actions github-actions
v3.17.3
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
ef263a6
Compare
Choose a tag to compare
OONI Probe CLI v3.17.3

This patch release contains the following changes since v3.17.2:

  • 🐛 fix(signal): bump the version number 2915bb6
  • 🚧 chore: update to go1.19.9 bb77cd8
  • 🐛 fix: use 2023-05 geoip databases bc54bee
  • 🚧 fix(stuninput): sync with tor-browser-build df9677a
  • 🚧 chore: run go generate ./... 79bba6f

Full Changelog: v3.17.2...v3.17.3

Assets 35
Loading

OONI Probe CLI v3.17.2

11 May 06:02
@github-actions github-actions
v3.17.2
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
837a522
Compare
Choose a tag to compare
OONI Probe CLI v3.17.2

This patch release contains the following fixes since v3.17.1:

  • 🐛 signal: remove check for api.directory.signal.org by @simoncpu (thanks!) 837a522

All these patches have been backported from the main development branch.

Full Changelog: v3.17.1...v3.17.2

Contributors

simoncpu
Assets 35
Loading

OONI Probe CLI v3.17.1

16 Mar 17:08
@github-actions github-actions
v3.17.1
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
50054f5
Compare
Choose a tag to compare
OONI Probe CLI v3.17.1

This patch release contains the following major fixes since v3.17.0:

  • 🐛 [backport] fix(netxlite): handle "tls: unrecognized name" alert 9501668

  • 🐛 [backport] fix(mobile): honor check-in feature flags 0f9b235

  • 🐛 [backport] fix(webconnectivitylte): handle i18n domains 691e539

All these patches have been backported from the main development branch.

Full Changelog: v3.17.0...v3.17.1

Assets 35
Loading

OONI Probe CLI v3.17.0

20 Feb 06:54
@github-actions github-actions
v3.17.0
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
76c47ec
Compare
Choose a tag to compare
OONI Probe CLI v3.17.0

Highlights

New Features

✨ ✨ ✨ New Experiments

✨ ✨ ✨ Proxy improvements

  • ooniprobe: add the --proxy command line flag which works like miniooni --proxy
  • miniooni and ooniprobe: add support for the --proxy torsf:///, which uses snowflake over tor
  • miniooni: add --snowflake-rendezvous CLI flag to specify the snowflake rendezvous policy

✨ ✨ ✨ OONI Run v2 improvements

  • miniooni: read OONI Run v2 descriptors form the filesystem with miniooni oonirun -f FILE

✨ ✨ ✨ Misc improvements

  • geoipx: use ASN+Country database generated using ooni/historical-geoip
  • httpapi: new API allowing to fallback when multiple endpoints are available
  • measurexlite: add refraction-networking/utls support
  • probeservices: support compressed check-in API responses
  • buildtool: new Go based build tool with unit testing of the build flags passed to compilers
  • buildtool: rewrite Android build to compile tor, libevent, openssl, and zlib with hardened compiler flags
  • ooniprobe and miniooni: add support for A/B testing webconnectivity versus webconnectivity LTE
  • model: expose build information as annotations inside measurements

API changes

  • ❗ [API BREAK] oonimkall: drop deprecated APIs
  • ❗ [API BREAK] oonimkall: rename CheckInConfigWebConnectivity's Add method to AddCategory
  • ❗ [ABI BREAK] model: use udp rather than quic as the protocol (see #946 for details)

Bug Fixes

  • 🐛 psiphon: make sure we include a config when building for Linux
  • 🐛 signal: update the embedded signal CA
  • 🐛 webconnectivity: increase robustness by trying to use all available TH before giving up
  • 🐛 whatsapp: stop measuring http://web.whatsapp.com/ to avoid false positives
  • 🐛 geolocate: make sure we use the session resolver for consistency
  • 🐛 measurexlite: emit the resolve_start and resolve_done events
  • 🐛 model: mark optional fields as optional and sync up with ooni/spec's data formats
  • 🐛 model: improve measurements scrubbing
  • 🐛 netxlite: make sure we wrap DNS decoding errors
  • 🐛 scrubber: merge improvements from the snowflake codebase
  • 🐛 webconnectivity LTE: don't fetch webpages from 127.0.0.1 and ::1
  • 🐛 oohelperd and nextlite: use cached cert pools
  • 🐛 getaddrinfo: make sure we always emit a canonical CNAME

Maintenance

  • 🚧 telegram: stop measuring http://web.telegram.org/ to avoid potential false positives
  • 🚧 .github/workflows: automatically generate several github workflows files
  • 🚧 all: use go1.19.6 and update dependencies
  • 🚧 QA: simplify quality assurance scripts and focus on web connectivity only
  • 🚧 MONOREPO: merge monorepo-like scripts to facilitate integration testing
  • 🚧 ooniprobe: make database code abstract to facilitate subsequent refactoring
  • 🚧 internal: change way in which we provide arguments to OONI experiments
  • 🚧 probeservices: start using the httpapi package
  • 🚧 probeservices: remove support for deprecated APIs
  • 🚧 httpx: deprecate in favour of httpapi
  • 🚧 model: reintroduce netxlite's underlying network functionality
  • 🚧 internal: finish refactoring internal/engine subpackages by moving them into internal
  • 🚧 dash: migrate to m-lab locate services v2

What's Changed Since v3.17.0-beta.1

  • 🚧 all: update to go1.19.6
  • 🚧 oohelperd: introduce request flag to enable QUIC

Full Changelog: v3.17.0-beta.1...v3.17.0

What's Changed Since v3.16.7

Full Changelog: v3.16.7...v3.17.0

New Contributors

Contributors

arky, d1vyank, and ooninoob
Assets 35
Loading