Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change blast start_date and remove ignore_parent_replication_key #5

Merged
merged 2 commits into from
Sep 18, 2024
Merged

change blast start_date and remove ignore_parent_replication_key #5

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
949e5a5
change blast start_date and remove ignore_parent_replication_key
yujoy Sep 18, 2024
d2453e2
add condition if self.stream_state.get(starting_replication_value) is…
yujoy Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions tap_sailthru/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,10 +158,17 @@ def prepare_request_payload(
:param context: Stream partition or context dictionary.
:returns: dict, A dictionary containing the request payload.
"""

# set the start date to 7 days before the last replication state
starting_replication_time = self.stream_state.get("starting_replication_value") or self.config.get('start_date')
starting_replication_date = pendulum.parse(starting_replication_time)
starting_replication_date_minus_7 = starting_replication_date.subtract(days=7).start_of('day')
start_date = starting_replication_date_minus_7.to_date_string()

return {
"status": "sent",
"limit": 0,
"start_date": self.stream_state.get("starting_replication_value"),
"start_date": start_date,
}

def get_child_context(self, record: dict, context: Optional[dict]) -> dict:
Expand Down Expand Up @@ -205,8 +212,6 @@ class BlastStatsStream(SailthruStream):
schema_filepath = SCHEMAS_DIR / "blast_stats.json"
parent_stream_type = BlastStream
rest_method = "GET"
# we set ignore_parent_replication_key = True here since we'd want the latest stats for each blast. the tradeoff is that the ingestion takes longer.
ignore_parent_replication_key = True

def get_url(self, context: Optional[dict]) -> str:
"""Construct url for api request.
Expand Down
Loading