[Transaction Stream Service] [Data Service] attempt to subtract with overflow

[lukasz-layerzerolabs]

Description

TSS is able to pull data from PFN (public full archival node). But then, when rust processors pull data from TSS, data service runs into an issue. Restarting TSS and the processors help for a while, but run into same issue eventually.

Repro

% cat data-service-config.yaml 

health_check_port: 8084

server_config:
  data_service_grpc_non_tls_config:
      data_service_grpc_listen_address: 0.0.0.0:50052
  disable_auth_check: true
  whitelisted_auth_tokens: []
  file_store_config:
    file_store_type: LocalFileStore
    local_file_store_path: /opt/aptos/data/file-store
  redis_read_replica_address: redis://localhost:6379

{"timestamp":"2025-01-17T06:36:33.949667Z","level":"INFO","message":"[Data Service] Transactions decoded.","start_version":405765000,"end_version":405765999,"start_txn_timestamp_iso":"2024-01-15T01:57:27.092361000Z","end_txn_timestamp_iso":"2024-01-15T01:58:36.807990000Z","num_transactions":1000,"duration_in_secs":0.049244331,"size_in_bytes":7334895,"processor_name":"user_transaction_processor","request_identifier_type":"unspecified","request_identifier":"unspecified","request_email":"unspecified","request_application_name":"unspecified","connection_id":"c208464f-7fa3-4234-815e-aff324561ed4","service_type":"data_service","step":"3.1","filename":"ecosystem/indexer-grpc/indexer-grpc-utils/src/counters.rs","line_number":278,"threadName":"tokio-runtime-worker","threadId":"ThreadId(9)"}

details = """
panicked at ecosystem/indexer-grpc/indexer-grpc-data-service/src/service.rs:655:29:
attempt to subtract with overflow"""
backtrace = """
   0:     0x5626a66f9a83 - backtrace::backtrace::libunwind::trace::h080984fce8cc0d2e
                               at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/libunwind.rs:93:5
                           backtrace::backtrace::trace_unsynchronized::h5ad767334a9176ec
                               at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/mod.rs:66:5
                           backtrace::backtrace::trace::h80b35d796876c071
                               at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/backtrace/mod.rs:53:14
                           backtrace::capture::Backtrace::create::h432de5c9b6d787e4
                               at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/capture.rs:176:9
                           backtrace::capture::Backtrace::new::h145127d7ab18bed0
                               at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.69/src/capture.rs:140:22
   1:     0x5626a63aa04f - aptos_indexer_grpc_server_framework::handle_panic::h7ee4abe32678cbcf
                               at /aptos/ecosystem/indexer-grpc/indexer-grpc-server-framework/src/lib.rs:147:38
                           aptos_indexer_grpc_server_framework::setup_panic_handler::{{closure}}::hc52788d2d6fa2b71
                               at /aptos/ecosystem/indexer-grpc/indexer-grpc-server-framework/src/lib.rs:139:9
   2:     0x5626a6813750 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h022ca2c0d8c21c9e
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9
                           std::panicking::rust_panic_with_hook::h0ad14d90dcf5224f
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13
   3:     0x5626a6813459 - std::panicking::begin_panic_handler::{{closure}}::h4a1838a06f542647
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13
   4:     0x5626a6811f86 - std::sys_common::backtrace::__rust_end_short_backtrace::h77cc4dc3567ca904
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18
   5:     0x5626a68131c4 - rust_begin_unwind
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   6:     0x5626a683bae5 - core::panicking::panic_fmt::h940d4fd01a4b4fd1
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   7:     0x5626a683bba3 - core::panicking::panic::h8ddd58dc57c2dc00
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:145:5
   8:     0x5626a5e64bb3 - aptos_indexer_grpc_data_service::service::ensure_sequential_transactions::h9150464e9b6315b9
                               at /aptos/ecosystem/indexer-grpc/indexer-grpc-data-service/src/service.rs:655:29
   9:     0x5626a5f4a5cf - aptos_indexer_grpc_data_service::service::data_fetcher_task::{{closure}}::ha0b109e9fa0a07aa
                               at /aptos/ecosystem/indexer-grpc/indexer-grpc-data-service/src/service.rs:503:36
  10:     0x5626a5f4e3a5 - <aptos_indexer_grpc_data_service::service::RawDataServerWrapper as aptos_protos::pb::aptos::indexer::v1::raw_data_server::RawData>::get_transactions::{{closure}}::{{closure}}::h35a0e7474c6ee725
                               at /aptos/ecosystem/indexer-grpc/indexer-grpc-data-service/src/service.rs:212:18
  11:     0x5626a5f3c27d - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h990c96b9af5786ff