diff --git a/data-formats/df-007-errors.md b/data-formats/df-007-errors.md
index d3ba8c2e..caf90af5 100644
--- a/data-formats/df-007-errors.md
+++ b/data-formats/df-007-errors.md
@@ -46,6 +46,12 @@ is not flagged, it means that it's used by all the three products.
- `"http_request_failed"` (MK, PE): HTTP request did not return a successful response code
+- `"http_unexpected_redirect_url"` (PE): we expected a specific redirect URL and
+instead we saw either no redirect URL or a different redirect URL
+
+- `"http_unexpected_status_code"` (PE): we did not expected to see this status
+code (e.g. we expected a redirection and saw something else)
+
- `"interrupted"` (PE): the user interrupted us by cancelling the context
- `"response_never_received"` (PL): [t.w._newclient.ResponseNeverReceived](https://twistedmatrix.com/documents/15.4.0/api/twisted.web._newclient.ResponseNeverReceived.html)
diff --git a/nettests/ts-018-whatsapp.md b/nettests/ts-018-whatsapp.md
index 727a11d2..69429d19 100644
--- a/nettests/ts-018-whatsapp.md
+++ b/nettests/ts-018-whatsapp.md
@@ -1,6 +1,6 @@
# Specification version number
-2020-02-21-001
+2020-07-09-001
# Specification name
@@ -12,10 +12,9 @@ WhatsApp
# Expected impact
-Ability to detect if the WhatsApp instant messaging platform is working, by
-checking if the DNS resolutions are consistent and if it's possible to
-establish a TCP connection with the IPs of the endpoints.
-
+Ability to detect whether specific services used by WhatsApp are
+accessible and working as intended. Even if this test does not detect
+any issue, WhatsApp may still be blocked in more complex ways.
# Expected inputs
@@ -25,17 +24,16 @@ None
This test will check if 3 services are working as they should:
-1. The whatsapp endpoints (the addresses used by the WhatsApp mobile client to work)
+1. The whatsapp endpoints used by the WhatsApp mobile app;
-2. The registration service
+2. The registration service, i.e. the service used to register a new account;
-3. The whatsapp web interface
+3. The WhatsApp web interface.
## WhatsApp endpoints check
The WhatsApp endpoints are those used by the WhatsApp app to send and receive
-messages.
-These endpoints have the following hostnames:
+messages. These endpoints have the following hostnames:
* e1.whatsapp.net
* e2.whatsapp.net
@@ -54,57 +52,65 @@ These endpoints have the following hostnames:
* e15.whatsapp.net
* e16.whatsapp.net
-When the `--all-endpoints` option is **not** specified we will be testing one
-of this endpoints picked at random, while when that option is specified we
-will be testing all of the ones listed above.
+The behaviour of the WhatsApp app is to attempt to connect to each of the
+following hostnames on ports 443 and 5222 until one succeeds. A previous
+version of this specification checked only just one of the above endpoints,
+but since 2020-07-09 we check all the endpoints on ports 443 and 5222.
-To check if they work properly we will first do a DNS A lookup to the
-endpoint in question. We then check if any of the returned IPs (we ignore
-anything that is not an IPv4 address, like a CNAME) are part of the address
-space used by whatsapp.
+We lookup every domain name and record the ASN of each returned IP, deferring to
+OONI data processing pipeline to determine whether the IP is legitimate.
-As a reference point to know if a certain IP is part of the WhatsApp network
-we use the list of IP blocks published by WhatsApp here:
-https://www.whatsapp.com/cidr.txt.
-The test contains a harcoded version of the cirdr list with a timestamp of when
-it was last retrieved from the URL.
+If we can connect to at least one endpoint, we set:
-If any of the returned IPs are part of any network block listed in `cidr.txt`
-we consider the DNS response to be consistent. If that is not the case we
-consider the endpoint to be blocked and write in the report:
+```JSON
+{
+ "whatsapp_endpoints_status": "ok",
+}
+```
-```json
+If all endpoints fail, we set:
+
+```JSON
{
"whatsapp_endpoints_status": "blocked",
- "whatsapp_endpoints_dns_inconsistent": ["eN.whatsapp.net"]
}
```
-Since 2020-02-17, this consistency check has started failing consistently as
-documented in the bugs section at the end of this document.
+Therefore, this specific key tells us whether the WhatsApp would work by
+connecting to one of the hardcoded endpoints.
+
+As of 2020-07-09, all endpoints work with ports 443 and 5222. If an
+endpoint fails for both ports, then we consider it blocked and we add
+it to the list of blocked endpoints as follows:
-For every IP, both consistent and inconsistent, we then try to establish a TCP session to port `443`
-and `5222`.
+```JSON
+{
+ "whatsapp_endpoints_blocked": [
+ "e1.whatsapp.net"
+ ]
+}
+```
-For a given IP address we consider the connection successful if either connecting to `IP:443` or `IP:5222` succeeds.
+If we don't record any failure, such list shall be empty:
-If ANY of the connections to consistent endpoints (i.e. within the net blocks
-assigned to WhatsApp) succeed then we consider the endpoint to not be
-blocked.
+```JSON
+{
+ "whatsapp_endpoints_blocked": []
+}
+```
-Conversely if all attempts fail then we consider the endpoint to be blocked
-and mark is as such in the report:
+We always additionally write into the report:
-```json
+```JSON
{
- "whatsapp_endpoints_status": "blocked",
- "whatsapp_endpoints_blocked": ["eN.whatsapp.net"]
+ "whatsapp_endpoints_dns_inconsistent": []
}
```
-Note: Not every WhatsApp endpoint actually listens on both port `5222` and
-`443` and the behavior of the whatsapp client it to attempt to connect to
-both `5222` and `443` on each endpoint until it manages to connect.
+The `whatsapp_endpoints_dns_inconsistent` key is a legacy key that we
+used to compute whether an endpoint was part of the WhatsApp address
+space, but this check has been broken since at least 2020-02-17, so we
+first disabled the check and then updated the spec to not mention it.
## Registration service check
@@ -116,8 +122,13 @@ The registration service is a `HTTPS` service at the following URL:
https://v.whatsapp.net/v2/register.
To check if it is working properly we do a HTTP GET request to
-`https://v.whatsapp.net/v2/register` and if it succeeds we consider the
-endpoint to be working and write in the report:
+`https://v.whatsapp.net/v2/register`. We conside this request to
+be successful if we don't see any DNS, TCP connect, TLS, or I/O
+error. We don't rely on the HTTP status code to determine whether
+this request succeeded, because we assume it is enough to determine
+whether we could have an HTTPS conversation with this service.
+
+If there is no failure, we write in the report:
```json
{
@@ -138,25 +149,31 @@ When it fails we write:
## WhatApp web check
WhatsApp web is the service by which users are able to use WhatsApp from a
-web browser on their computer.
-When using WhatsApp web users scan a QR code displayed in the browser from
-their phone to authenticate the web app.
+web browser on their computer. When using WhatsApp web users scan a QR code
+displayed in the browser from their phone to authenticate the web app.
For the service to work a user needs to have whatsapp be working properly
from their phone (it needs to be unblocked there) and if the "Keep me signed
in" option is unticked their phone needs to be connected to the internet and
be able to reach the whatsapp endpoints for the duration of the session.
-We check to see if WhatsApp web is working properly by doing a HTTPS GET request to the following URLs:
+We check to see if WhatsApp web is working properly by doing a HTTPS GET request
+to the following URLs:
* https://web.whatsapp.com/
+
* http://web.whatsapp.com/
-If the HTTP(s) requests fail or the HTML `` tag text is not "WhatsApp
-Web" we consider the endpoint to be blocked.
+We consider the GET for the HTTPS URL successful if we do not see any
+DNS, TCP connect, TLS, or I/O errors when fetching the URL. WhatsApp may
+return an 400 Bad Request response if the User-Agent header we use does
+not seem to be consistent with our ClientHello.
+
+The GET for the HTTP URL should not follow redirects. We consider it
+successful if the response redirects us to the HTTPS URL.
-If either one of the HTTP or HTTPS endpoints are blocked then we write in the
-report:
+If either one of the HTTP or HTTPS endpoints are blocked then we write
+in the report:
```json
{
@@ -182,39 +199,68 @@ If none of the endpoints are blocked then we write:
* df-002-dnst
-## Semantics
+* df-005-tcpconnect
-```json
-{
- "whatsapp_endpoints_status": "blocked" | "ok" | null,
- "whatsapp_endpoints_dns_inconsistent": ["e[1-16].whatsapp.net"],
- "whatsapp_endpoints_blocked": ["e[1-16].whatsapp.net"],
+* df-006-tlshandshake
- "whatsapp_web_status": "blocked" | "ok" | null,
- "whatsapp_web_failure": "FAILURE STRING",
+* df-008-netevents
- "registration_server_status": "blocked" | "ok" | null,
+* df-009-tunnel
- "registration_server_failure": "FAILURE STRING (since 0.5.0)",
- "registratison_server_failure": "FAILURE STRING (note: up until 0.4.0 name of the key was mispelled)",
+## Semantics
- "tcp_connect": [
- {
- "ip": "xxx.xxx.xxx.xxx",
- "port": 5222 | 443,
- "status": {
- "success": true | false,
- "failure": "FAILURE STRING"
- }
- }
- ],
+The `test_keys` emitted by this nettest include the following keys:
+
+```JSON
+{
+ // keys from parent data formats
+ "network_events": [],
+ "queries": [],
+ "requests": [],
+ "tcp_connect": [],
+ "tls_handshakes": [],
+
+ // keys specific of WhatsApp data format
+ "registration_server_failure": null,
+ "registration_server_status": "ok",
+ "whatsapp_endpoints_blocked": [],
+ "whatsapp_endpoints_dns_inconsistent": [],
+ "whatsapp_endpoints_status": "ok",
+ "whatsapp_web_failure": null,
+ "whatsapp_web_status": "ok"
}
```
-The meaning of the various keys is described in the above section.
+where:
-## Possible conclusions
+- `network_events` (`[]NetworkEvent`; nullable): see `df-008-netevents`;
+- `queries` (`[]Query`; nullable): see `df-002-dnst`;
+
+- `requests` (`[]Transaction`; nullable): see `df-001-httpt`;
+
+- `tcp_connect` (`[]TCPConnect`; nullable): see `df-005-tcpconnect`;
+
+- `tls_handshakes` (`[]Handshake`; nullable): see `df-006-tlshandshake`;
+
+- `registration_server_failure` (`string`; nullable): the failure when
+accessing the registration server (see `df-007-errors.md`);
+
+- `registration_server_status` (`string`): one of `"ok"` and `"blocked"`;
+
+- `whatsapp_endpoints_blocked` (`[]string`): list of blocked endpoints;
+
+- `whatsapp_endpoints_dns_inconsistent` (`[]string`): legacy key
+that is always empty;
+
+- `whatsapp_endpoints_status` (`string`): one of `"ok"` and `"blocked"`;
+
+- `whatsapp_web_failure` (`string`; nullable): the failure when
+accessing the WhatsApp web interface (see `df-007-errors.md`);
+
+- `whatsapp_web_status` (`string`): one of `"ok"` and `"blocked"`.
+
+## Possible conclusions
* If it is possible for users to create new accounts via WhatsApp
@@ -226,332 +272,900 @@ The meaning of the various keys is described in the above section.
```json
{
- "annotations": {
- "platform": "macos"
- },
- "data_format_version": "0.2.0",
- "id": "3efd4e05-be03-46b7-a21d-034cb03bdde9",
- "input": null,
- "input_hashes": [],
- "measurement_start_time": "2016-11-25 12:33:30",
- "options": [],
- "probe_asn": "AS30722",
- "probe_cc": "IT",
- "probe_city": null,
- "probe_ip": "127.0.0.1",
- "report_id": "m2fUNCj20WEaAHZaiNBf1vIUFkytNqCLQboP0XJrSFPpDpy5me7tDfEaAQktPHDe",
- "software_name": "ooniprobe",
- "software_version": "2.0.3.dev0",
- "test_helpers": {},
- "test_keys": {
- "agent": "redirect",
- "queries": [
- {
- "answers": [
- {
- "answer_type": "A",
- "ipv4": "169.47.5.199"
- },
- {
- "answer_type": "A",
- "ipv4": "174.36.208.132"
- },
- {
- "answer_type": "A",
- "ipv4": "173.192.231.41"
- },
- {
- "answer_type": "A",
- "ipv4": "174.37.217.84"
- },
- {
- "answer_type": "A",
- "ipv4": "169.45.214.245"
- },
- {
- "answer_type": "A",
- "ipv4": "169.45.219.230"
- },
- {
- "answer_type": "A",
- "ipv4": "169.45.248.175"
- },
- {
- "answer_type": "A",
- "ipv4": "169.45.248.107"
- }
- ],
- "failure": null,
- "hostname": "e14.whatsapp.net",
- "query_type": "A",
- "resolver_hostname": null,
- "resolver_port": null
- }
+ "annotations": {
+ "assets_version": "20200619115947",
+ "engine_name": "ooniprobe-engine",
+ "engine_version": "0.14.0",
+ "platform": "macos"
+ },
+ "data_format_version": "0.2.0",
+ "extensions": {
+ "dnst": 0,
+ "httpt": 0,
+ "netevents": 0,
+ "tlshandshake": 0,
+ "tunnel": 0
+ },
+ "input": null,
+ "measurement_start_time": "2020-07-09 11:02:41",
+ "probe_asn": "AS30722",
+ "probe_cc": "IT",
+ "probe_ip": "127.0.0.1",
+ "probe_network_name": "Vodafone Italia S.p.A.",
+ "report_id": "20200709T110241Z_AS30722_CIaQmE3CBdzrf1oIh7F3DwALQtjl4PrTaQxXlsNP7C011453ea",
+ "resolver_asn": "AS30722",
+ "resolver_ip": "91.80.36.84",
+ "resolver_network_name": "Vodafone Italia S.p.A.",
+ "software_name": "miniooni",
+ "software_version": "0.14.0",
+ "test_keys": {
+ "agent": "redirect",
+ "failed_operation": null,
+ "failure": null,
+ "network_events": [
+ {
+ "failure": null,
+ "operation": "http_transaction_start",
+ "t": 0.001153697
+ },
+ {
+ "failure": null,
+ "operation": "http_request_metadata",
+ "t": 0.001157765
+ },
+ {
+ "failure": null,
+ "operation": "resolve_start",
+ "t": 0.001227361
+ },
+ {
+ "failure": null,
+ "operation": "resolve_done",
+ "t": 0.055199778
+ },
+ {
+ "address": "69.171.250.60:443",
+ "failure": null,
+ "operation": "connect",
+ "proto": "tcp",
+ "t": 0.079603245
+ },
+ {
+ "failure": null,
+ "operation": "tls_handshake_start",
+ "t": 0.079654268
+ },
+ {
+ "failure": null,
+ "num_bytes": 286,
+ "operation": "write",
+ "t": 0.079957156
+ },
+ {
+ "failure": null,
+ "num_bytes": 517,
+ "operation": "read",
+ "t": 0.104859831
+ },
+ {
+ "failure": null,
+ "num_bytes": 1800,
+ "operation": "read",
+ "t": 0.105431746
+ },
+ {
+ "failure": null,
+ "num_bytes": 747,
+ "operation": "read",
+ "t": 0.105475141
+ },
+ {
+ "failure": null,
+ "num_bytes": 64,
+ "operation": "write",
+ "t": 0.107882362
+ },
+ {
+ "failure": null,
+ "operation": "tls_handshake_done",
+ "t": 0.107907192
+ },
+ {
+ "failure": null,
+ "num_bytes": 86,
+ "operation": "write",
+ "t": 0.108139396
+ },
+ {
+ "failure": null,
+ "num_bytes": 206,
+ "operation": "write",
+ "t": 0.109137636
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_headers",
+ "t": 0.109141745
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_request",
+ "t": 0.109142604
+ },
+ {
+ "failure": null,
+ "num_bytes": 74,
+ "operation": "read",
+ "t": 0.131422583
+ },
+ {
+ "failure": null,
+ "num_bytes": 31,
+ "operation": "write",
+ "t": 0.131530094
+ },
+ {
+ "failure": null,
+ "num_bytes": 31,
+ "operation": "read",
+ "t": 0.132357428
+ },
+ {
+ "failure": null,
+ "num_bytes": 35,
+ "operation": "read",
+ "t": 0.134856016
+ },
+ {
+ "failure": null,
+ "num_bytes": 1388,
+ "operation": "read",
+ "t": 0.178145709
+ },
+ {
+ "failure": null,
+ "num_bytes": 383,
+ "operation": "read",
+ "t": 0.178193064
+ },
+ {
+ "failure": null,
+ "operation": "http_first_response_byte",
+ "t": 0.178348521
+ },
+ {
+ "failure": null,
+ "operation": "http_response_metadata",
+ "t": 0.178471921
+ },
+ {
+ "failure": null,
+ "operation": "http_response_body_snapshot",
+ "t": 0.178485775
+ },
+ {
+ "failure": null,
+ "operation": "http_transaction_done",
+ "t": 0.178486692
+ },
+ {
+ "failure": null,
+ "num_bytes": 24,
+ "operation": "write",
+ "t": 0.178593354
+ },
+ {
+ "failure": null,
+ "operation": "resolve_start",
+ "t": 0.001173242
+ },
+ {
+ "failure": null,
+ "operation": "resolve_done",
+ "t": 0.093193011
+ },
+ {
+ "address": "34.194.71.217:443",
+ "failure": null,
+ "operation": "connect",
+ "proto": "tcp",
+ "t": 0.207018418
+ },
+ {
+ "failure": null,
+ "operation": "http_transaction_start",
+ "t": 0.179558692
+ },
+ {
+ "failure": null,
+ "operation": "http_request_metadata",
+ "t": 0.179570108
+ },
+ {
+ "failure": null,
+ "operation": "resolve_start",
+ "t": 0.180708632
+ },
+ {
+ "failure": null,
+ "operation": "resolve_done",
+ "t": 0.182215501
+ },
+ {
+ "address": "69.171.250.60:80",
+ "failure": null,
+ "operation": "connect",
+ "proto": "tcp",
+ "t": 0.207763367
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_headers",
+ "t": 0.208031246
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_request",
+ "t": 0.208033326
+ },
+ {
+ "failure": null,
+ "num_bytes": 282,
+ "operation": "write",
+ "t": 0.208117192
+ },
+ {
+ "failure": null,
+ "num_bytes": 242,
+ "operation": "read",
+ "t": 0.232560865
+ },
+ {
+ "failure": null,
+ "operation": "http_first_response_byte",
+ "t": 0.232572517
+ },
+ {
+ "failure": null,
+ "operation": "http_response_metadata",
+ "t": 0.232644304
+ },
+ {
+ "failure": null,
+ "operation": "http_response_body_snapshot",
+ "t": 0.23266589
+ },
+ {
+ "failure": null,
+ "operation": "http_transaction_done",
+ "t": 0.232666803
+ },
+ {
+ "failure": null,
+ "operation": "http_transaction_start",
+ "t": 0.001150339
+ },
+ {
+ "failure": null,
+ "operation": "http_request_metadata",
+ "t": 0.00115765
+ },
+ {
+ "failure": null,
+ "operation": "resolve_start",
+ "t": 0.001227368
+ },
+ {
+ "failure": null,
+ "operation": "resolve_done",
+ "t": 0.071238074
+ },
+ {
+ "address": "69.171.250.60:443",
+ "failure": null,
+ "operation": "connect",
+ "proto": "tcp",
+ "t": 0.096787583
+ },
+ {
+ "failure": null,
+ "operation": "tls_handshake_start",
+ "t": 0.096826301
+ },
+ {
+ "failure": null,
+ "num_bytes": 284,
+ "operation": "write",
+ "t": 0.097124463
+ },
+ {
+ "failure": null,
+ "num_bytes": 517,
+ "operation": "read",
+ "t": 0.122612789
+ },
+ {
+ "failure": null,
+ "num_bytes": 1800,
+ "operation": "read",
+ "t": 0.122967996
+ },
+ {
+ "failure": null,
+ "num_bytes": 747,
+ "operation": "read",
+ "t": 0.12299006
+ },
+ {
+ "failure": null,
+ "num_bytes": 64,
+ "operation": "write",
+ "t": 0.124585528
+ },
+ {
+ "failure": null,
+ "operation": "tls_handshake_done",
+ "t": 0.124645709
+ },
+ {
+ "failure": null,
+ "num_bytes": 86,
+ "operation": "write",
+ "t": 0.124800198
+ },
+ {
+ "failure": null,
+ "num_bytes": 214,
+ "operation": "write",
+ "t": 0.125082793
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_headers",
+ "t": 0.125085502
+ },
+ {
+ "failure": null,
+ "operation": "http_wrote_request",
+ "t": 0.125086283
+ },
+ {
+ "failure": null,
+ "num_bytes": 74,
+ "operation": "read",
+ "t": 0.14885878
+ },
+ {
+ "failure": null,
+ "num_bytes": 31,
+ "operation": "write",
+ "t": 0.148993236
+ },
+ {
+ "failure": null,
+ "num_bytes": 44,
+ "operation": "read",
+ "t": 0.1490235
+ },
+ {
+ "failure": null,
+ "num_bytes": 161,
+ "operation": "read",
+ "t": 0.630222125
+ },
+ {
+ "failure": null,
+ "operation": "http_first_response_byte",
+ "t": 0.630348977
+ },
+ {
+ "failure": null,
+ "operation": "http_response_metadata",
+ "t": 0.63050614
+ },
+ {
+ "failure": null,
+ "operation": "http_response_body_snapshot",
+ "t": 0.630518627
+ },
+ {
+ "failure": null,
+ "operation": "http_transaction_done",
+ "t": 0.630519479
+ },
+ {
+ "failure": null,
+ "num_bytes": 24,
+ "operation": "write",
+ "t": 0.630606143
+ }
+ ],
+ "queries": [
+ {
+ "answers": [
+ {
+ "asn": 32934,
+ "as_org_name": "Facebook, Inc.",
+ "answer_type": "A",
+ "ipv4": "69.171.250.60",
+ "ttl": null
+ }
+ ],
+ "engine": "system",
+ "failure": null,
+ "hostname": "web.whatsapp.com",
+ "query_type": "A",
+ "resolver_hostname": null,
+ "resolver_port": null,
+ "resolver_address": "",
+ "t": 0.055199778
+ },
+ {
+ "answers": [
+ {
+ "asn": 32934,
+ "as_org_name": "Facebook, Inc.",
+ "answer_type": "AAAA",
+ "ipv6": "2a03:2880:f2ff:c0:face:b00c:0:167",
+ "ttl": null
+ }
],
- "registration_server_failure": null,
- "registration_server_status": "ok",
- "requests": [
- {
- "failure": null,
- "request": {
- "body": null,
- "headers": {},
- "method": "GET",
- "tor": {
- "exit_ip": null,
- "exit_name": null,
- "is_tor": false
- },
- "url": "https://web.whatsapp.com/"
- },
- "response": {
- "body": "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js lt-ie9 lt-ie8 lt-ie7\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js lt-ie9 lt-ie8\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js lt-ie9\"> <![endif]-->\n<!--[if gt IE 8]><!-->\n<html class=\"no-js\" dir=\"ltr\">\n<!--<![endif]-->\n\n<head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\n <title>WhatsApp Web\n \n \n\n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n
\n
\n
\n
\n
\n
WhatsApp Web
\n
Send and receive WhatsApp messages right from your computer.
\n
We recommend using WhatsApp with one of the following browsers: