Postgres destination accepts only 10k rows from a table containing json

[D1111111]

Howdy ✋

I have a Postgres -> Postgres connection that fails to load more than 10k rows. It is stuck in some kind of loop. After several hours or days of loading, I need to cancel it manually. Job history shows a larger number of rows extracted (around 30k), but only 10k inserted.

The problem is a json column inside the source table, which is not too big, let's say a couple of kB per row. When I remove that json column from the pipeline rows are loaded as expected.

I tried all versions of Postgres destination connector - same problem.

This is the example of the log, these lines are repeated over and over...

Namespace: raw Stream: prs__public__xxx -- remaining records: **68020**

97475

2024-11-07 19:03:56 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.

97476

2024-11-07 19:03:56 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 0 bytes, after penalty b: 240.43 MB

97477

2024-11-07 19:03:56 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 25 MB, after penalty b: 215.43 MB

97478

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (acbe7) -- Worker picked up work.

97479

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (acbe7) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97480

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (54150) -- Worker picked up work.

97481

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (acbe7) -- Batch contains: 0 records, 0 bytes bytes.

97482

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (54150) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97483

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97484

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (54150) -- Batch contains: 0 records, 0 bytes bytes.

97485

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97486

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97487

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97488

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97489

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97490

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97491

2024-11-07 19:03:56 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (acbe7) -- Worker finished flushing. Current queue size: 68020

97492

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97493

2024-11-07 19:03:56 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (54150) -- Worker finished flushing. Current queue size: 68020

97494

2024-11-07 19:03:57 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO

97495

  Namespace: raw Stream: prs__public_xxx -- remaining records: 68020

97496

2024-11-07 19:03:57 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.

97497

2024-11-07 19:03:57 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 0 bytes, after penalty b: 240.43 MB

97498

2024-11-07 19:03:57 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):109 flushing: trigger info: raw - prs__public__xxx, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 240.43 MB, penalty b: 25 MB, after penalty b: 215.43 MB

97499

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (f8a3c) -- Worker picked up work.

97500

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (f8a3c) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97501

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):135 Flush Worker (dbe81) -- Worker picked up work.

97502

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (f8a3c) -- Batch contains: 0 records, 0 bytes bytes.

97503

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):141 Flush Worker (dbe81) -- Attempting to read from queue namespace: raw, stream: prs__public__xxx.

97504

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97505

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):159 Flush Worker (dbe81) -- Batch contains: 0 records, 0 bytes bytes.

97506

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97507

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states

97508

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97509

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete

97510

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97511

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97512

2024-11-07 19:03:57 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (f8a3c) -- Worker finished flushing. Current queue size: 68020

97513

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..

97514

2024-11-07 19:03:57 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (dbe81) -- Worker finished flushing. Current queue size: 68020

97515

2024-11-07 19:03:58 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO

97516

  Namespace: raw Stream: prs__public__xxx -- remaining records: **68020**

97517

[D1111111]

The issue is that there are some json values with more than 25 MB in size. When such records are removed loader works as expected.

Can the loader be configured to handle bigger payloads?