What are the best practices for numaflow in production environments? And performance tuning?

[qianbeibuzui]

I'm sorry to disturb you, but I'm planning to perform performance tests on numaflow, but I have some doubts:
What is the relationship between verticals.sections and scale.min?

Is it similar to the relationship between Kafka partitions and consumers? Can a consumer group only read one partition?
Or is it true that vertices. partitions only affect the reading efficiency of inter step buffer by vertices?
2. The document states: To achieve higher throughput (>10K but<30K tps), users can create multi-partitioned edges. Multi-partitioned edges are only supported for pipelines with JetStream as ISB. Please ensure that the JetStream is provisioned with more nodes to support higher throughput.
Does it mean that the limit tps of a single link inter stop buffer is 30K? What are the hardware resources of JetStream?
If we want to break through TPS>30K, currently, can we only do load balancing from the source?
3. Lastly, I would like to provide feedback on the issue I encountered:
Does vertices.sink not support partitions?
When I am using Vertices. UDF, setting partitions works fi

partitions = 1

partitions = 3

[vigith]

1. What is the relationship between verticals.sections and scale.min?
   Is it similar to the relationship between Kafka partitions and consumers? Can a consumer group only read one partition?
   Or is it true that vertices. partitions only affect the reading efficiency of inter step buffer by vertices?

partitions is the number of streams between the vertices, larger the partition number, higher the TPS that vertex can take. scale is for the autoscaling, min is minimum number of pods in the vertex, it there can also be scale.max. partition and scale.min are independent, you can increase the min value without increasing the partitions.

2. The document states: To achieve higher throughput (>10K but<30K tps), users can create multi-partitioned edges. Multi-partitioned edges are only supported for pipelines with JetStream as ISB. Please ensure that the JetStream is provisioned with more nodes to support higher throughput.
   Does it mean that the limit tps of a single link inter stop buffer is 30K? What are the hardware resources of JetStream? If we want to break through TPS>30K, currently, can we only do load balancing from the source?

We have only benchmarked for 50K TPS (because most of our use case is well within the limit), you can increase the TPS by adding more partitions and bigger ISBs (more CPU and MEM). Please make sure the volume attached to the ISB can support high i/o.

3. Lastly, I would like to provide feedback on the issue I encountered:
   Does vertices.sink not support partitions?

partition is to increase the TPS between two vertices, for Sink you do not need partitions because there is no Vertex after the Sink.

NOTE: partition and scale are independent, to increase the TPS increase the partition; even with one partition you can have many pods in a vertex. Increasing the partition increased the number of streams in the ISB and this is completely handled by the platform, you do not have to worry about it.