Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure seq_join is called with an appropriate value for active_work #1499

Open
andyleiserson opened this issue Dec 14, 2024 · 1 comment
Open

Ensure seq_join is called with an appropriate value for active_work #1499

andyleiserson opened this issue Dec 14, 2024 · 1 comment

Comments

@andyleiserson
Copy link
Collaborator

Historically, active_work was configured globally and had the same value everywhere.

Now, active_work is adjusted when creating a DZKP validator, to have an appropriate value relative to the batch size.

Unfortunately, seq_join is often invoked as a free function, like this: seq_join(ctx.active_work(), ...). If there are multiple contexts in scope (e.g. a non-upgraded ctx and a DZKP-upgraded m_ctx), the active work used for seq_join may not match the validator-adjusted value.

Assuming we can avoid the dreaded 100013-related errors, it might be better to invoke seq_join as a helper on the context, so that the appropriate active work can be used automatically. Although, that may not improve things that much, since one could still do ctx.seq_join(/* futures referencing m_ctx */).

In any case, filing this as a reminder to revisit this, possibly doing some refactoring, or maybe just auditing seq_join calls.
@akoshelev
Copy link
Collaborator

I didn't capture all the logs wrt to memory, but one issue I see that is related to seq_join that it takes 10% of the overall protocol execution time (10mins vs 58mis overall time)

2024-12-23T04:22:06.898143Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=90.6µs time.idle=2.71ms
2024-12-23T04:22:27.364097Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 23 is ready for validation
2024-12-23T04:22:27.364141Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 23
2024-12-23T04:22:27.364158Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:22:27.366729Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=79.8µs time.idle=2.49ms
2024-12-23T04:22:27.454832Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::telemetry::memory::jemalloc: i=786432: 3594 MiB allocated
2024-12-23T04:22:47.224847Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 24 is ready for validation
2024-12-23T04:22:47.224893Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 24
2024-12-23T04:22:47.224924Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:22:47.227597Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=77.9µs time.idle=2.59ms
2024-12-23T04:23:08.679074Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 25 is ready for validation
2024-12-23T04:23:08.679116Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 25
2024-12-23T04:23:08.679127Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:23:08.683727Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=94.5µs time.idle=4.50ms
2024-12-23T04:23:29.110581Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 26 is ready for validation
2024-12-23T04:23:29.110629Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 26
2024-12-23T04:23:29.110651Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:23:29.114146Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=75.3µs time.idle=3.42ms
2024-12-23T04:23:49.565109Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 27 is ready for validation
2024-12-23T04:23:49.565149Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 27
2024-12-23T04:23:49.565160Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:23:49.568696Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=74.4µs time.idle=3.46ms
2024-12-23T04:24:09.930296Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 28 is ready for validation
2024-12-23T04:24:09.930342Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 28
2024-12-23T04:24:09.930353Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:24:09.934108Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=76.5µs time.idle=3.68ms
2024-12-23T04:24:30.307269Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 29 is ready for validation
2024-12-23T04:24:30.307313Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 29
2024-12-23T04:24:30.307323Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:24:30.310152Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=79.1µs time.idle=2.75ms
2024-12-23T04:24:50.148570Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 30 is ready for validation
2024-12-23T04:24:50.148615Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 30
2024-12-23T04:24:50.148646Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:24:50.151240Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=72.3µs time.idle=2.52ms
2024-12-23T04:25:10.118020Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 31 is ready for validation
2024-12-23T04:25:10.118063Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 31
2024-12-23T04:25:10.118074Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:25:10.120398Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=93.6µs time.idle=2.23ms
2024-12-23T04:25:10.206685Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::telemetry::memory::jemalloc: i=1048576: 3646 MiB allocated
2024-12-23T04:25:32.110522Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 32 is ready for validation
2024-12-23T04:25:32.110567Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 32
2024-12-23T04:25:32.110578Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:25:32.115323Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=80.7µs time.idle=4.66ms
2024-12-23T04:25:53.982577Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 33 is ready for validation
2024-12-23T04:25:53.982617Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 33
2024-12-23T04:25:53.982627Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:25:53.987450Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=91.0µs time.idle=4.73ms
2024-12-23T04:26:15.895585Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 34 is ready for validation
2024-12-23T04:26:15.895626Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 34
2024-12-23T04:26:15.895637Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:26:15.900273Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=97.1µs time.idle=4.54ms
2024-12-23T04:26:38.848777Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 35 is ready for validation
2024-12-23T04:26:38.848825Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 35
2024-12-23T04:26:38.848836Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:26:38.851662Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=84.2µs time.idle=2.74ms
2024-12-23T04:27:01.840517Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 36 is ready for validation
2024-12-23T04:27:01.840564Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 36
2024-12-23T04:27:01.840575Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:27:01.843536Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=81.6µs time.idle=2.88ms
2024-12-23T04:27:22.275974Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 37 is ready for validation
2024-12-23T04:27:22.276018Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 37
2024-12-23T04:27:22.276028Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:27:22.278650Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=87.6µs time.idle=2.53ms
2024-12-23T04:27:25.753962Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: batch 38 is ready for validation
2024-12-23T04:27:25.754020Z DEBUG hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::context::batcher: validating batch 38
2024-12-23T04:27:25.754039Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: new
2024-12-23T04:27:25.757010Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs:validate{gate=/hybrid/eval_prf/validate}: ipa_core::protocol::context::validator: close time.busy=70.8µs time.idle=2.90ms
2024-12-23T04:30:38.910499Z  INFO hybrid_query{sz=1000000000}:compute_prf_for_inputs: ipa_core::protocol::hybrid::oprf: close time.busy=1105s time.idle=697s
2024-12-23T04:30:50.465674Z DEBUG hybrid_query{sz=1000000000}: ipa_core::protocol::context::dzkp_malicious: Changed active_work from 32768 to 4194304 to match batch size
2024-12-23T04:30:50.466724Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=0: 1030 MiB allocated
2024-12-23T04:30:50.467810Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=256: 1030 MiB allocated
2024-12-23T04:30:50.468672Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=512: 1030 MiB allocated
2024-12-23T04:30:50.469825Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=768: 1031 MiB allocated
2024-12-23T04:30:50.471724Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=1024: 1031 MiB allocated
2024-12-23T04:30:50.473400Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=1536: 1032 MiB allocated
2024-12-23T04:30:50.475056Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=2048: 1033 MiB allocated
2024-12-23T04:30:50.479634Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=3072: 1034 MiB allocated
2024-12-23T04:30:50.483377Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=4096: 1035 MiB allocated
2024-12-23T04:30:50.487862Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=6144: 1038 MiB allocated
2024-12-23T04:30:50.491619Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=8192: 1041 MiB allocated
2024-12-23T04:30:50.498494Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=12288: 1046 MiB allocated
2024-12-23T04:30:50.506376Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=16384: 1051 MiB allocated
2024-12-23T04:30:50.520543Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=24576: 1062 MiB allocated
2024-12-23T04:30:50.533951Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=32768: 1072 MiB allocated
2024-12-23T04:30:50.562928Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=49152: 1093 MiB allocated
2024-12-23T04:30:50.588309Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=65536: 1114 MiB allocated
2024-12-23T04:30:50.638926Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=98304: 1154 MiB allocated
2024-12-23T04:30:50.683828Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=131072: 1195 MiB allocated
2024-12-23T04:30:50.794084Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=196608: 1278 MiB allocated
2024-12-23T04:30:50.962882Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=262144: 1371 MiB allocated
2024-12-23T04:30:51.308981Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=393216: 1556 MiB allocated
2024-12-23T04:30:51.646152Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=524288: 1739 MiB allocated
2024-12-23T04:30:52.278125Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=786432: 2103 MiB allocated
2024-12-23T04:30:52.955654Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=1048576: 2470 MiB allocated
2024-12-23T04:30:54.338063Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=1572864: 3230 MiB allocated
2024-12-23T04:30:55.690192Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=2097152: 3970 MiB allocated
2024-12-23T04:30:58.348091Z DEBUG hybrid_query{sz=1000000000}: ipa_core::telemetry::memory::jemalloc: i=3145728: 5477 MiB allocated
2024-12-23T04:41:56.186137Z  INFO hybrid_query{sz=1000000000}: ipa_core::protocol::context::batcher: batch 0 is ready for validation
2024-12-23T04:41:56.186187Z DEBUG hybrid_query{sz=1000000000}: ipa_core::protocol::context::batcher: validating batch 0
2024-12-23T04:41:58.311716Z  INFO hybrid_query{sz=1000000000}: ipa_core::protocol::context::dzkp_validator: validating 36700928 multiplications
2024-12-23T04:42:14.647177Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}: ipa_core::protocol::hybrid::breakdown_reveal: new
2024-12-23T04:42:14.647228Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:apply_dp_padding: ipa_core::protocol::ipa_prf::oprf_padding: new
2024-12-23T04:42:16.805445Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:apply_dp_padding: ipa_core::protocol::ipa_prf::oprf_padding: Total number of padding records added: 9216. Padding Parameters: PaddingParameters { aggregation_padding: Parameters { aggregation_epsilon: 5.0, aggregation_delta: 1e-6, aggregation_padding_sensitivity: 10 }, oprf_padding: Parameters { oprf_epsilon: 5.0, oprf_delta: 1e-6, matchkey_cardinality_cap: 10, oprf_padding_sensitivity: 2 } }
2024-12-23T04:42:16.805501Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:apply_dp_padding: ipa_core::protocol::ipa_prf::oprf_padding: close time.busy=728µs time.idle=2.16s
2024-12-23T04:42:16.805517Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::protocol::hybrid::breakdown_reveal: new
2024-12-23T04:42:16.806518Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=0: 1123 MiB allocated
2024-12-23T04:42:17.112889Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=256: 1124 MiB allocated
2024-12-23T04:42:17.125244Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=512: 1124 MiB allocated
2024-12-23T04:42:17.126245Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=768: 1125 MiB allocated
2024-12-23T04:42:17.127467Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=1024: 1126 MiB allocated
2024-12-23T04:42:17.128956Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=1536: 1127 MiB allocated
2024-12-23T04:42:17.130834Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=2048: 1129 MiB allocated
2024-12-23T04:42:17.136073Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=3072: 1132 MiB allocated
2024-12-23T04:42:17.139863Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=4096: 1135 MiB allocated
2024-12-23T04:42:17.152101Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=6144: 1141 MiB allocated
2024-12-23T04:42:17.155919Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=8192: 1147 MiB allocated
2024-12-23T04:42:17.172467Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=12288: 1159 MiB allocated
2024-12-23T04:42:17.179912Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=16384: 1170 MiB allocated
2024-12-23T04:42:17.200745Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=24576: 1193 MiB allocated
2024-12-23T04:42:17.214171Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=32768: 1215 MiB allocated
2024-12-23T04:42:17.942922Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=49152: 1244 MiB allocated
2024-12-23T04:42:18.526222Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=65536: 1256 MiB allocated
2024-12-23T04:42:19.705729Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::telemetry::memory::jemalloc: i=98304: 1278 MiB allocated
2024-12-23T04:42:40.114072Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:shuffle_attribution_outputs: ipa_core::protocol::hybrid::breakdown_reveal: close time.busy=16.9s time.idle=6.38s
2024-12-23T04:42:40.114139Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::protocol::hybrid::breakdown_reveal: new
2024-12-23T04:42:40.115156Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=0: 1124 MiB allocated
2024-12-23T04:42:40.116182Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=256: 1124 MiB allocated
2024-12-23T04:42:40.142053Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=512: 1124 MiB allocated
2024-12-23T04:42:40.143009Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=768: 1125 MiB allocated
2024-12-23T04:42:40.143856Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=1024: 1125 MiB allocated
2024-12-23T04:42:40.144966Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=1536: 1125 MiB allocated
2024-12-23T04:42:40.146321Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=2048: 1126 MiB allocated
2024-12-23T04:42:40.148156Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=3072: 1127 MiB allocated
2024-12-23T04:42:40.150065Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=4096: 1128 MiB allocated
2024-12-23T04:42:40.155012Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=6144: 1131 MiB allocated
2024-12-23T04:42:40.159670Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=8192: 1133 MiB allocated
2024-12-23T04:42:40.168860Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=12288: 1137 MiB allocated
2024-12-23T04:42:40.177446Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=16384: 1142 MiB allocated
2024-12-23T04:42:40.196277Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=24576: 1151 MiB allocated
2024-12-23T04:42:40.215117Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=32768: 1159 MiB allocated
2024-12-23T04:42:40.415061Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=49152: 1162 MiB allocated
2024-12-23T04:42:40.622044Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=65536: 1165 MiB allocated
2024-12-23T04:42:40.926762Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=98304: 1171 MiB allocated
2024-12-23T04:42:41.236993Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=131072: 1177 MiB allocated
2024-12-23T04:42:41.846045Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=196608: 1189 MiB allocated
2024-12-23T04:42:42.482010Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=262144: 1202 MiB allocated
2024-12-23T04:42:43.728703Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=393216: 1226 MiB allocated
2024-12-23T04:42:44.968792Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=524288: 1252 MiB allocated
2024-12-23T04:42:47.446127Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=786432: 1301 MiB allocated
2024-12-23T04:42:50.015278Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=1048576: 1353 MiB allocated
2024-12-23T04:42:55.243006Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=1572864: 1450 MiB allocated
2024-12-23T04:43:00.474803Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=2097152: 1555 MiB allocated
2024-12-23T04:43:10.761407Z DEBUG hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::telemetry::memory::jemalloc: i=3145728: 1747 MiB allocated
2024-12-23T04:43:13.334759Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:reveal_breakdowns{total=3345652}: ipa_core::protocol::hybrid::breakdown_reveal: close time.busy=16.2s time.idle=17.0s
2024-12-23T04:43:13.359507Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=32768}: ipa_core::protocol::ipa_prf::aggregation: new
2024-12-23T04:44:35.717971Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=32768}: ipa_core::protocol::ipa_prf::aggregation: close time.busy=1.52s time.idle=80.8s
2024-12-23T04:44:36.778954Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}: ipa_core::protocol::context::dzkp_validator: validating 33549568 multiplications
2024-12-23T04:44:37.148069Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=128}: ipa_core::protocol::ipa_prf::aggregation: new
2024-12-23T04:44:37.502821Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=128}: ipa_core::protocol::ipa_prf::aggregation: close time.busy=8.20ms time.idle=347ms
2024-12-23T04:44:37.508184Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}: ipa_core::protocol::context::dzkp_validator: validating 128256 multiplications
2024-12-23T04:44:37.513354Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=2}: ipa_core::protocol::ipa_prf::aggregation: new
2024-12-23T04:44:37.547123Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}:aggregate_values{num_rows=2}: ipa_core::protocol::ipa_prf::aggregation: close time.busy=7.68ms time.idle=26.1ms
2024-12-23T04:44:37.549432Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}: ipa_core::protocol::context::dzkp_validator: validating 16384 multiplications
2024-12-23T04:44:37.553359Z  INFO hybrid_query{sz=1000000000}:breakdown_reveal_aggregation{total=3336436}: ipa_core::protocol::hybrid::breakdown_reveal: close time.busy=37.6s time.idle=105s
2024-12-23T04:46:28.403317Z  INFO hybrid_query{sz=1000000000}: ipa_core::protocol::context::dzkp_validator: validating 802816 multiplications
2024-12-23T04:46:28.456617Z  INFO hybrid_query{sz=1000000000}: ipa_core::query::runner::hybrid: close time.busy=680s time.idle=2532s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@akoshelev @andyleiserson