feat: Vector Search #952
Triggered via pull request
November 25, 2024 21:25
hinthornw
Status
Success
Total duration
48m 16s
Artifacts
–
Annotations
2 notices
|
Benchmark results:
libs/checkpoint-postgres/Makefile#L1
.........................................
fanout_to_subgraph_10x: Mean +- std dev: 61.0 ms +- 1.3 ms
.........................................
fanout_to_subgraph_10x_sync: Mean +- std dev: 51.4 ms +- 0.6 ms
.........................................
WARNING: the benchmark result may be unstable
* the standard deviation (11.1 ms) is 12% of the mean (94.0 ms)
Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.
fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 94.0 ms +- 11.1 ms
.........................................
fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 94.7 ms +- 2.5 ms
.........................................
fanout_to_subgraph_100x: Mean +- std dev: 601 ms +- 33 ms
.........................................
fanout_to_subgraph_100x_sync: Mean +- std dev: 501 ms +- 7 ms
.........................................
fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 1.04 sec +- 0.07 sec
.........................................
fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 941 ms +- 35 ms
.........................................
react_agent_10x: Mean +- std dev: 30.9 ms +- 0.7 ms
.........................................
react_agent_10x_sync: Mean +- std dev: 22.5 ms +- 0.4 ms
.........................................
react_agent_10x_checkpoint: Mean +- std dev: 47.1 ms +- 0.7 ms
.........................................
react_agent_10x_checkpoint_sync: Mean +- std dev: 38.1 ms +- 1.5 ms
.........................................
react_agent_100x: Mean +- std dev: 342 ms +- 7 ms
.........................................
react_agent_100x_sync: Mean +- std dev: 271 ms +- 3 ms
.........................................
react_agent_100x_checkpoint: Mean +- std dev: 955 ms +- 24 ms
.........................................
react_agent_100x_checkpoint_sync: Mean +- std dev: 864 ms +- 15 ms
.........................................
wide_state_25x300: Mean +- std dev: 24.0 ms +- 0.4 ms
.........................................
wide_state_25x300_sync: Mean +- std dev: 15.4 ms +- 0.1 ms
.........................................
wide_state_25x300_checkpoint: Mean +- std dev: 279 ms +- 4 ms
.........................................
wide_state_25x300_checkpoint_sync: Mean +- std dev: 268 ms +- 6 ms
.........................................
wide_state_15x600: Mean +- std dev: 28.1 ms +- 0.5 ms
.........................................
wide_state_15x600_sync: Mean +- std dev: 17.9 ms +- 0.2 ms
.........................................
wide_state_15x600_checkpoint: Mean +- std dev: 485 ms +- 5 ms
.........................................
wide_state_15x600_checkpoint_sync: Mean +- std dev: 471 ms +- 7 ms
.........................................
wide_state_9x1200: Mean +- std dev: 28.1 ms +- 0.5 ms
.........................................
wide_state_9x1200_sync: Mean +- std dev: 17.8 ms +- 0.1 ms
.........................................
wide_state_9x1200_checkpoint: Mean +- std dev: 315 ms +- 4 ms
.........................................
wide_state_9x1200_checkpoint_sync: Mean +- std dev: 301 ms +- 5 ms
|
|
Comparison against main:
libs/checkpoint-postgres/Makefile#L1
+-----------------------------------------+---------+------------------------+
| Benchmark | main | changes |
+=========================================+=========+========================+
| fanout_to_subgraph_100x | 609 ms | 601 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_9x1200_sync | 18.0 ms | 17.8 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_25x300_checkpoint | 282 ms | 279 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_9x1200_checkpoint | 318 ms | 315 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_25x300_sync | 15.5 ms | 15.4 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| fanout_to_subgraph_10x | 61.4 ms | 61.0 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| react_agent_10x_sync | 22.6 ms | 22.5 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| react_agent_100x_sync | 272 ms | 271 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_25x300_checkpoint_sync | 269 ms | 268 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_15x600_sync | 18.0 ms | 17.9 ms: 1.01x faster |
+-----------------------------------------+---------+------------------------+
| react_agent_100x | 344 ms | 342 ms: 1.00x faster |
+-----------------------------------------+---------+------------------------+
| fanout_to_subgraph_10x_sync | 51.6 ms | 51.4 ms: 1.00x faster |
+-----------------------------------------+---------+------------------------+
| wide_state_9x1200_checkpoint_sync | 302 ms | 301 ms: 1.00x faster |
+-----------------------------------------+---------+------------------------+
| react_agent_10x_checkpoint | 46.8 ms | 47.1 ms: 1.01x slower |
+-----------------------------------------+---------+------------------------+
| react_agent_100x_checkpoint | 947 ms | 955 ms: 1.01x slower |
+-----------------------------------------+---------+------------------------+
| fanout_to_subgraph_100x_checkpoint_sync | 932 ms | 941 ms: 1.01x slower |
+-----------------------------------------+---------+------------------------+
| react_agent_100x_checkpoint_sync | 846 ms | 864 ms: 1.02x slower |
+-----------------------------------------+---------+------------------------+
| react_agent_10x_checkpoint_sync | 36.5 ms | 38.1 ms: 1.04x slower |
+-----------------------------------------+---------+------------------------+
| fanout_to_subgraph_100x_checkpoint | 944 ms | 1.04 sec: 1.10x slower |
+-----------------------------------------+---------+------------------------+
| Geometric mean | (ref) | 1.00x slower |
+-----------------------------------------+---------+------------------------+
Benchmark hidden because not significant (9): wide_state_25x300, fanout_to_subgraph_10x_checkpoint_sync, fanout_to_subgraph_100x_sync, wide_state_15x600, react_agent_10x, wide_state_9x1200, wide_state_15x600_checkpoint, wide_state_15x600_checkpoint_sync, fanout_to_subgraph_10x_checkpoint
|