Stop sync from an API when a certain number of pages is reached

[pai911]

Hi! We're trying Airbyte to download data from some API.

This API support cursor-based pagination

• the records are not sorted by time, so it's hard to check record creation time and decide when to stop.
• it doesn't support any time related param to limit the records, so the only way to limit the number of records to download is through number of pages.

So we perform sync every day, and each time we want to only paginate through the first N pages.

However, I can't get my head around doing this by using the connector builder.

Is it possible to make sync stop when a certain number of pages is reached with connector builder or even low-code builder?
Is there a way for me to define a custom variable and keep increment it by myself so I can use it in the stop condition?

Thank you!

[szemek]

https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination#cursor
You need to figure out how to set cursor_value and stop_condition for your API.

[pai911]

@szemek thank you for the response!

but actually, i do know how to use cursor and stop condition. The problem is that in the stop condition, I can't check the current page num so I can't make the sync stop when a certain number of pages has been reached.

Could you shade some lights on this?

[szemek]

Okay, so I guess it is not possible to do this in that way.
An idea came to my mind, DatetimeBasedCursor https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor
with start_datetime: "{{ day_delta(-N) }}", end_datetime: {{ now_utc() }} and step: P1D it should be possible to iterate N times

Another idea, https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/CursorPagination
https://docs.airbyte.com/connector-development/config-based/advanced-topics#custom-components

[pai911]

@szemek thanks again for sharing some thought.

Could I check with you are these solutions work for "Connector Builder"? or do I have to use Low-Code CDK instead?

[szemek]

First idea should work for Connector Builder (No Code), second one requires Low-Code CDK

[pai911]

@szemek could you elaborate more on how could the first idea help with my need?

I am trying to make the sync stop when a certain number of pages has been reached, and the records are not sorted by time.

Thank you!

[szemek]

I was thinking about DatetimeBasedCursor as an iterator. Let's use an example -- you set start date 2024-11-19 (day_delta(-N)) and end date 2024-11-21 (now_utc())
2024-11-19
2024-11-20
2024-11-21

By using some formatting %d: Day of the month (zero-padded) - 01, 02, ..., 31 you will get
19
20
21

You might inject it anywhere as long as it doesn't interfere with API response.
The main point in this is to execute requests N times.
I don't know whether it is going to work, it's just an idea for you to explore.

Personally, I'd use Python CDK.