Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] reduce port system load #1318

Open
wants to merge 51 commits into
base: main
Choose a base branch
from

Conversation

yaelibarg
Copy link
Contributor

@yaelibarg yaelibarg commented Jan 12, 2025

Description

What - reduce the amount of upserts we send to port api

Why - many of the upserts does not contain an actual change, reduce load from port api

How - check if the entity from the third party has a change from the entity in port, and only if there is an actual change, upsert the entity

Type of change

Please leave one option from the following and delete the rest:

  • Bug fix (non-breaking change which fixes an issue)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

  • Integration able to create all default resources from scratch
  • Resync finishes successfully
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Scheduled resync able to abort existing resync and start a new one
  • Tested with at least 2 integrations from scratch
  • Tested with Kafka and Polling event listeners
  • Tested deletion of entities that don't pass the selector

@yaelibarg yaelibarg requested a review from a team as a code owner January 12, 2025 14:44
@yaelibarg yaelibarg changed the title Port 12078 reduce port system load [Core] Port 12078 reduce port system load Jan 12, 2025
@github-actions github-actions bot added size/L and removed size/M labels Jan 12, 2025
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
port_ocean/core/utils/utils.py Outdated Show resolved Hide resolved
port_ocean/core/utils/utils.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
@Tankilevitch Tankilevitch changed the title [Core] Port 12078 reduce port system load [Core] reduce port system load Jan 13, 2025
@github-actions github-actions bot added size/XL and removed size/L labels Jan 15, 2025
integrations/jira/.port/resources/port-app-config.yaml Outdated Show resolved Hide resolved
port_ocean/clients/port/mixins/entities.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
Comment on lines +142 to +157
return {
"combinator": "and",
"rules": [
{
"combinator": "or",
"rules": [
{
"property": "$identifier",
"operator": "=",
"value": entity.identifier,
}
for entity in entities
]
}
]
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the first layer is redundant as eventually we are constructing a query for finding the entity identifiers therefor only the or part is of interest for us.

Suggested change
return {
"combinator": "and",
"rules": [
{
"combinator": "or",
"rules": [
{
"property": "$identifier",
"operator": "=",
"value": entity.identifier,
}
for entity in entities
]
}
]
}
return
{
"combinator": "or",
"rules": [
{
"property": "$identifier",
"operator": "=",
"value": entity.identifier,
}
for entity in entities
]
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we add on this query after , in the search_entities function, so you have that the first layer will be 'and' layer

port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
port_ocean/core/utils/utils.py Outdated Show resolved Hide resolved
port_ocean/core/utils/utils.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
port_ocean/core/integrations/mixins/sync_raw.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants