getblock using verbose=3 to fetch prevout from vin

[DenisCarriere]

Issue

Attempting to extract additional information from vin (inputs) which is present in prevout.

• spent_block_height
• spent_tx_id
• spent_output_number
• value
• address
• type

"prevout": {
  "generated": false,
  "height": 799988,
  "value": 0.00615523,
  "scriptPubKey": {
    "asm": "0 436d0f301e99932e46dadb712b2bf99cdb3a7704",
    "desc": "addr(bc1qgdks7vq7nxfju3k6mdcjk2lenndn5acy5f2c4c)#g95pzkc8",
    "hex": "0014436d0f301e99932e46dadb712b2bf99cdb3a7704",
    "address": "bc1qgdks7vq7nxfju3k6mdcjk2lenndn5acy5f2c4c",
    "type": "witness_v0_keyhash"
  }
}

Bitcoin RPC poller

Currently uses GetBlockVerbose, perhaps GetBlockVerboseTx or raw getblock request with verbose=3

firehose-bitcoin/fetch/poller.go

Line 129 in 8bde96e

bestBlock, err := p.rpcClient.GetBlockVerbose(bestBlockHash)

Context

When using the bitcoin-cli with verbosity=3 you can get full detailed transactions of inputs

including prevout information for inputs (only for unpruned blocks in the current best chain).

getblock "blockhash" ( verbosity )

If verbosity is 0, returns a string that is serialized, hex-encoded data for block 'hash'.
If verbosity is 1, returns an Object with information about block <hash>.
If verbosity is 2, returns an Object with information about block <hash> and information about each transaction.
If verbosity is 3, returns an Object with information about block <hash> and information about each transaction, including prevout information for inputs (only for unpruned blocks in the current best chain).

getblock 800007 using verbose=3

$ bitcoin-cli getblock 0000000000000000000221ce7e43028c690783a79d19c16f16482a3dad2f741b 3

{
  "hash": "00000000000000000003b23539814444975c630a65094821113801d71953bdba",
  "confirmations": 53950,
  "height": 800007,
  "version": 536887296,
  "versionHex": "20004000",
  "merkleroot": "f58ff91b6b7159de5f785826168dd081145d7f1cb26460d6e6f8f725aea9ded5",
  "time": 1690174725,
  "mediantime": 1690169223,
  "nonce": 2967617187,
  "bits": "17053894",
  "difficulty": 53911173001054.59,
  "chainwork": "00000000000000000000000000000000000000004fc9b1edc67742ebc6c524c8",
  "nTx": 3438,
  "previousblockhash": "00000000000000000000886edafb8a1ab4f837b88d4e8323f6b376f21652ead9",
  "nextblockhash": "00000000000000000004d4b0199777457b09afec3c1cc4a327b6ce4047d25b45",
  "strippedsize": 809065,
  "size": 1565895,
  "weight": 3993090,
  "tx": [
   ...
    {
      "txid": "a0cd5283286c1d0adb28e4c6a69efd006dfbbcbd1c3205dd570d660f90af0796",
      "hash": "3ba929be1a95fb67c530f493a9cddf87a0b9ff6edea3b1e7d73ae1639f1e1c35",
      "version": 2,
      "size": 223,
      "vsize": 141,
      "weight": 562,
      "locktime": 0,
      "vin": [
        {
          "txid": "396b8e37dea0afb5756c7218118021159e711fb32682efff018b32fb16c243a8",
          "vout": 79,
          "scriptSig": {
            "asm": "",
            "hex": ""
          },
          "txinwitness": [
            "3045022100dd9d9158523a66c7d37e7230ffe7c0962c18705026d25f549cdaf0fe077be79e022022574566877a8e748a79065989b7bf38455606b89b95d3f5f87b5b57c7772e2e01",
            "0201994ac2dc75445b663e2b02196c441e8180908d467e4e40f562de1b4c059d8c"
          ],
          "prevout": {
            "generated": false,
            "height": 799988,
            "value": 0.00615523,
            "scriptPubKey": {
              "asm": "0 436d0f301e99932e46dadb712b2bf99cdb3a7704",
              "desc": "addr(bc1qgdks7vq7nxfju3k6mdcjk2lenndn5acy5f2c4c)#g95pzkc8",
              "hex": "0014436d0f301e99932e46dadb712b2bf99cdb3a7704",
              "address": "bc1qgdks7vq7nxfju3k6mdcjk2lenndn5acy5f2c4c",
              "type": "witness_v0_keyhash"
            }
          },
          "sequence": 4294967295
        }
      ],
      "vout": [
        {
          "value": 0.00167836,
          "n": 0,
          "scriptPubKey": {
            "asm": "0 9f882f5ca22e8746f14c7b6966d45ec90a5c09d8",
            "desc": "addr(bc1qn7yz7h9z96r5du2v0d5kd4z7ey99czwcue9xkc)#mpnhfd6q",
            "hex": "00149f882f5ca22e8746f14c7b6966d45ec90a5c09d8",
            "address": "bc1qn7yz7h9z96r5du2v0d5kd4z7ey99czwcue9xkc",
            "type": "witness_v0_keyhash"
          }
        },
        {
          "value": 0.00445870,
          "n": 1,
          "scriptPubKey": {
            "asm": "0 effe6ea9da6bf705a4a37e8318ffc4fc5f73870e",
            "desc": "addr(bc1qallxa2w6d0mstf9r06p33l7yl30h8pcwn0vfwf)#xnxsrrnr",
            "hex": "0014effe6ea9da6bf705a4a37e8318ffc4fc5f73870e",
            "address": "bc1qallxa2w6d0mstf9r06p33l7yl30h8pcwn0vfwf",
            "type": "witness_v0_keyhash"
          }
        }
      ],
      "fee": 0.00001817,
      "hex": "02000000000101a843c216fb328b01ffef8226b31f719e1521801118726c75b5afa0de378e6b394f00000000ffffffff029c8f0200000000001600149f882f5ca22e8746f14c7b6966d45ec90a5c09d8aecd060000000000160014effe6ea9da6bf705a4a37e8318ffc4fc5f73870e02483045022100dd9d9158523a66c7d37e7230ffe7c0962c18705026d25f549cdaf0fe077be79e022022574566877a8e748a79065989b7bf38455606b89b95d3f5f87b5b57c7772e2e01210201994ac2dc75445b663e2b02196c441e8180908d467e4e40f562de1b4c059d8c00000000"
    }
  ]
}

[DenisCarriere]

Compared the extra data requirements for that additional prevout field using verbose=3 when fetching getblock, we're only talking about **+5% size increase **

15MB (verbose 2) vs. 16MB (verbose 3)

$ bitcoin-cli getblock 0000000000000000000221ce7e43028c690783a79d19c16f16482a3dad2f741b 2
$ bitcoin-cli getblock 0000000000000000000221ce7e43028c690783a79d19c16f16482a3dad2f741b 3

$ du -h getblock/bitcoin-800007-block*
 15M    getblock/bitcoin-800007-block-verbose=2.json
 16M    getblock/bitcoin-800007-block-verbose=3.json

Using Substreams file Parquet sink (with snappy compression) (6.4MB)
uncompressed was ~11MB

$ du -h ./out/**/0000800007-0000800008.parquet
8.0K    ./out/blocks/0000800007-0000800008.parquet
2.3M    ./out/inputs/0000800007-0000800008.parquet
1.3M    ./out/outputs/0000800007-0000800008.parquet
2.8M    ./out/transactions/0000800007-0000800008.parquet