feat: make sure LTE and v0.4 emit the same test keys #1392

bassosimone · 2023-11-22T16:02:07Z

This diff switches LTE to use the classic "engine", removes restrictions regarding comparing test keys, and ensures with the existing test suite that LTE and v0.4 produce the same top-level test keys under the same conditions.

While there, let's also bump LTE's version number, lest we forget.

See ooni/probe#2634

See ooni/probe#2628 See ooni/probe#2634

We're introducing failure modes that do not exist hence it seems this is not the correct way of moving forward.

I'm doing this mainly to explore whether we could have more robust webconnectivity v0.5 analysis code

Because I am dropping the requests again, we break again the tests with the redirects. I could possibly fix it by putting requests back again but I am not super happy about doing this because that would cause the DSL to do some strange work and I'd honestly rather not do this.

what remains to be done now is to make sure we make green all the tests that are currently skipped we also need to account for differences between the two

then next step is to sort out this mess :-)

internal/experiment/webconnectivityqa/badssl.go

internal/experiment/webconnectivityqa/control.go

internal/experiment/webconnectivityqa/dnsblocking.go

This diff introduces the `minipipeline` package and command. The general idea here is to analyze experiments results using an architecture that matches https://github.com/ooni/data as closely as possible. The advantage of using this architecture is that we can neatly divide the data analysis process into three phases: 1. ingestion converts an OONI measurement into observations, which are flat data structures; 2. analysis converts observations into an analysis structure, which is significantly simpler than observations; 3. scoring is the process (implemented by each experiment) to turn the analysis into top-level keys. That is: ```mermaid stateDiagram-v2 state "{ingestion}" as ingestion MEASUREMENT --> ingestion ingestion --> OBSERVATIONS state "{analysis}" as analysis OBSERVATIONS --> analysis analysis --> ANALYSIS state "{scoring}" as scoring ANALYSIS --> scoring scoring --> TOP_LEVEL_KEYS ``` This diff in particular tackles points 1 and 2. We'll implement 3 for Web Connectivity LTE in a separate diff. The diff itself commits several observations and analysis files along with the measurements from which they originate, which we all collectively use to write regression tests for the `minipipeline` package. The motivation for implementing these changes is that I need a more clearly defined model to address the differences between Web Connectivity LTE and v0.4 (see also ooni#1392). Because there is already an architectural model for implementing these changes in ooni/data, it seems logical to borrow liberally from ooni/data. Incidentally, this change has the benefit of making it easier, in the future, to align ooni/probe-cli and ooni/data. The reference issue is ooni/probe#2634. Implementing this change seems now a precondition for implementing extra changes that would address such an issue for good, by changing LTE's scoring. Unlike the ooni/data pipeline, this pipeline is minimal (hence the name): it only aims to process measurements collected by a recent version of ooniprobe, rather than all measurements from the beginning of time. While there, disable signal integration tests because of ooni/probe#2636.

This diff introduces the qatool, originally developed in ooni#1392. This tool allows to run webconnectivitylte netemx-based QA tests and produces the measurement, observations, and analysis files. We can use this tool to run individual tests and inspect their results when they're failing. Additionally, we can use this tool to regenerate the test suite used by the minipipeline. When running the tool, we can select which QA tests to run, whether to rerun netemx and to reprocess. This feature is handy because we don't need to rerun netemx every time and sometimes we may only want to rerun netemx. This work is part of ooni/probe#2634. It helps us to manage and test the minipipeline and webconnectivitylte in an easier way, and it enables further development and testing.

This diff adds more QA tests extracted from ooni#1392. Part of ooni/probe#2634.

) We're not saving "connect" network events when TCP connect fails. This diff fixes the codebase so that we always save these events. Fix extracted from ooni#1392. Reference issue ooni/probe#2634.

Extracted from ooni#1392. Part of ooni/probe#2634.

This diff switches LTE to use the classic "engine", removes restrictions regarding comparing test keys, and ensures with the existing test suite that LTE and v0.4 produce the same top-level test keys under the same conditions. While there, let's also bump LTE's version number, lest we forget. See ooni/probe#2634

bassosimone added 22 commits November 22, 2023 17:00

chore: start investigating LTE vs v0.4

766d4d0

See ooni/probe#2628 See ooni/probe#2634

document why some QA tests with redirects are broken

f3701a0

document more doubts about emmitting events

f1cc4bb

document more caveats

0b2203d

[ci skip] remember to update files in sync

8eb1ba1

doc: document more doubts that I have

92eb7b7

[ci skip] more documentation on what to do

ea0d3bf

feat: progress towards fixing some fundamental issues

fd06406

resolve one more test case

ad75714

more fixes

d6cbfd9

doc: explain issues caused by adding HTTP response

41fbd3f

We're introducing failure modes that do not exist hence it seems this is not the correct way of moving forward.

try to sketch out an ooni/data-inspired pipeline

e5e4c37

I'm doing this mainly to explore whether we could have more robust webconnectivity v0.5 analysis code

convert more of v0.5's analysis to the ooni/data-like style

6aff4f0

some more progress

94f9fd7

feat: rewrite the pipeline to match ooni/data more closely

5ad88d5

also implement the analysis

132ba4d

work

be18947

we're mostly done in terms of passing the existing QA tests

18da855

tests now green

e7764c8

what remains to be done now is to make sure we make green all the tests that are currently skipped we also need to account for differences between the two

make more test cases work with LTE

dac4170

we now pass all tests

258d7fb

then next step is to sort out this mess :-)