HTTP Tracker client: Print unrecognized responses in JSON

[josecelano]

Parent issue: #669
Depends on: #673

Not all tracker responses are always the same for the same request. Some trackers can include only mandatory fields. The example below is a wrong response because the scrape request should include the downloaded field. However, there could be other cases when the tracker response is valid but is not exactly the same response as in the Torrust Tracker. If we want to support other trackers we should be flexible with valid responses that don't match our resposnses.

Announce Request

When you run the HTTP tracker client with a Torrust Tracker:

cargo run --bin http_tracker_client announce http://127.0.0.1:7070 9c38422213e30bff212b30c360d26f9a02136422

you receive a response like this:

{
  "complete": 1,
  "incomplete": 0,
  "interval": 1800,
  "min interval": 900,
  "peers": [
    {
      "ip": "111.222.111.222",
      "peer id": [
        45,
        113,
        66,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        48,
        49
      ],
      "port": 17548
    }
  ]
}

Scrape Request

When you run the HTTP tracker client with a Torrust Tracker:

cargo run --bin http_tracker_client scrape http://127.0.0.1:7070 9c38422213e30bff212b30c360d26f9a02136422

you receive a response like this:

{
  "9c38422213e30bff212b30c360d26f9a02136422": {
    "complete": 1,
    "downloaded": 4,
    "incomplete": 0
  }
}

Compatibility Problem

$ cargo run --bin http_tracker_client scrape http://open.acgnxtracker.com:80 9c3842
2213e30bff212b30c360d26f9a02136422 | jq
    Finished dev [optimized + debuginfo] target(s) in 0.08s
     Running `target/debug/http_tracker_client scrape 'http://open.acgnxtracker.com:80' 9c38422213e30bff212b30c360d26f9a02136422`
thread 'main' panicked at src/shared/bit_torrent/tracker/http/client/responses/scrape.rs:143:60:
called `Result::unwrap()` on an `Err` value: MissingFileField { field_name: "downloaded" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Alternative Tracker request:

http://open.acgnxtracker.com/scrape?info_hash=%44%3C%76%02%B4%FD%E8%3D%11%54%D6%D9%DA%48%80%84%18%B1%81%B6

Alternative Tracker response:

scrape.zip

{
   "files": {
      "<hex>44 3C 76 02 B4 FD E8 3D 11 54 D6 D9 DA 48 80 84 18 B1 81 B6</hex>": {
         "incomplete": 0,
         "complete": 32
      }
   }
}

Torrust Tracker request:

https://tracker.torrust-demo.com/scrape?info_hash=%44%3C%76%02%B4%FD%E8%3D%11%54%D6%D9%DA%48%80%84%18%B1%81%B6

Torrust Tracker response:

scrape-torrust.zip

{
   "files": {
      "<hex>44 3C 76 02 B4 FD E8 3D 11 54 D6 D9 DA 48 80 84 18 B1 81 B6</hex>": {
         "complete": 1,
         "downloaded": 0,
         "incomplete": 0
      }
   }
}

NOTICE: the alternative tracker does not include the downloaded attribute.

In these cases, the client should print the JSON response anyway. But we need a generic serialisation from Becnode to JSON.

Implementation

The implementation requires changing these functions:

async fn announce_command(tracker_url: String, info_hash: String) -> anyhow::Result<()> {
    let base_url = Url::parse(&tracker_url).context("failed to parse HTTP tracker base URL")?;
    let info_hash =
        InfoHash::from_str(&info_hash).expect("Invalid infohash. Example infohash: `9c38422213e30bff212b30c360d26f9a02136422`");

    let response = Client::new(base_url)
        .announce(&QueryBuilder::with_default_values().with_info_hash(&info_hash).query())
        .await;

    let body = response.bytes().await.unwrap();

    let announce_response: Announce = serde_bencode::from_bytes(&body)
        .unwrap_or_else(|_| panic!("response body should be a valid announce response, got: \"{:#?}\"", &body));

    let json = serde_json::to_string(&announce_response).context("failed to serialize scrape response into JSON")?;

    println!("{json}");

    Ok(())
}

async fn scrape_command(tracker_url: &str, info_hashes: &[String]) -> anyhow::Result<()> {
    let base_url = Url::parse(tracker_url).context("failed to parse HTTP tracker base URL")?;

    let query = requests::scrape::Query::try_from(info_hashes).context("failed to parse infohashes")?;

    let response = Client::new(base_url).scrape(&query).await;

    let body = response.bytes().await.unwrap();

    let scrape_response = scrape::Response::try_from_bencoded(&body)
        .unwrap_or_else(|_| panic!("response body should be a valid scrape response, got: \"{:#?}\"", &body));

    let json = serde_json::to_string(&scrape_response).context("failed to serialize scrape response into JSON")?;

    println!("{json}");

    Ok(())
}

We first try to deserialize the encoded response into our data structures:

    let announce_response: Announce = serde_bencode::from_bytes(&body)
        .unwrap_or_else(|_| panic!("response body should be a valid announce response, got: \"{:#?}\"", &body));

If that operation fails, then we try to serialize to a generic json.

You can get a list of HTTP trackers from https://newtrackon.com/. The client should print all the responses from all those trackers.

[qmi03]

hello i think i can take this on, can you assign this to me?

[josecelano]

hello i think i can take this on, can you assign this to me?

Hi @qmi03, that's great! For your information, I'm working on a small crate to convert Bencoded values to JSON. It's already working but not published yet because I'm still cleaning it up a bit, adding more tests, and thinking about improving the API.

Could you let me know if you are planning to use another crate for that or implement the conversion yourself? If you were planning to implement the conversion, you could start when I publish the new crate and use that one. I think I will publish version 0.1.0 in two weeks.

If you were planning to use another crate, you can go with it, and we can consider changing to mine later if it makes sense.

[josecelano]

And @qmi03, this is how we would like to show the final JSON:

torrust/teps#15

[josecelano]

Hi @qmi03, I've published the repo I'm working on to convert from Bencode to JSON:

https://github.com/torrust/torrust-bencode2json

That crate is the one I was planning to use for this issue.

The crate has yet to be published. I would like other people to review the crate's public interface.

[josecelano]

Hi @qmi03, I've published the repo I'm working on to convert from Bencode to JSON:

https://github.com/torrust/torrust-bencode2json

That crate is the one I was planning to use for this issue.

The crate has yet to be published. I would like other people to review the crate's public interface.

It was renamed: https://github.com/torrust/bencode2json

[josecelano]

Clients were extracted into a new package, bittorrent-tracker-client