[core] Fix HashBucketAssigner load index too large error with refactor exception #3796
Conversation
| try { | ||
| map = new Int2ShortHashMap(keyList.size()); | ||
| } catch (IllegalArgumentException e) { | ||
| map = new Int2ShortHashMap(); |
There was a problem hiding this comment.
Is there a way to solve it without downgrading?
|
@izhangzhihao Can you try the pr first to comfirm it can be work? Expect for your feedback ~ |
|
I will try this tomorrow |
thanks~ |
|
Just created a simple unit test: public class HashBucketAssignerTest {
@Test
public void testFunction() {
IntStream stream = IntStream.rangeClosed(1, 1_100_000_000);
final Int2ShortHashMap.Builder builder = Int2ShortHashMap.builder();
stream.forEach(
i -> builder.put(i, (short)0)
);
builder.build();
}
}Error: |
Hi， do you had apply the patch，I had test the ut it in my local，not report the exception only oom error. @izhangzhihao |
|
For your oom error, you need to add |
Yes，my compute memory not enough to so large，so you had tested in your flink job in the issue:#3776? @izhangzhihao |
|
Can you find a 32G machine and run |
OK，but from your error strack，I can not find the source position in HashCommon which is litter strange. |
|
Error happens here: public static int arraySize(final int expected, final float f) {
final long s = Math.max(2, nextPowerOfTwo((long)Math.ceil(expected / f)));
if (s > (1 << 30)) throw new IllegalArgumentException("Too large (" + expected + " expected elements with load factor " + f + ")");
return (int)s;
}BTW, I'm using: <dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil</artifactId>
<version>8.5.13</version>
</dependency>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil-core</artifactId>
<version>8.5.13</version>
</dependency> |
It seems not consistent with master paimon dependecies，would you change it the same with master？From the patch it should not run to the logic as your error stack during Int2ShortOpenHashMap build. |
|
the master branch is using |
OK，apply the patch to the code，Thanks~ @izhangzhihao |
|
exactly the same error: And it took 10 mins: 
|
|
OK，Int2ShortOpenHashMap insert operation need rehash array which would occur the error. Thanks for your support. @izhangzhihao |
xuzifu666
changed the title
There was a problem hiding this comment.
So is this not a fix but an exception msg improvement?
If true, pls remove close #3776 from your PR desc
Yes, currently we can give more advise to user about it. @izhangzhihao |
| map.put(keyList.getInt(i), valueList.getShort(i)); | ||
| } | ||
| } catch (IllegalArgumentException e) { | ||
| throw new PaimonUtilsException( |
There was a problem hiding this comment.
just throw runtime exception is OK.
There was a problem hiding this comment.
OK，had change exception to RumtimeException. @JingsongLi
|
+1 |
There was a problem hiding this comment.
The error message looks good to me; adding parallelism is a workaround. Ideally, the resource allocation for the task should match the data flow, rather than matching the total amount of data at rest. The current situation does not meet this ideal. Is there any plan for optimization?
Purpose
HashCommon in Int2ShortHashMap would cause IllegalArgumentException when expected > 1073741824L，need make regression to default constructor to continue in this condition.
relate to #3776
Tests
API and Format
Documentation