Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create DomainMetrics SQL table #35415

Merged
merged 8 commits into from
Dec 9, 2024
Merged

Create DomainMetrics SQL table #35415

merged 8 commits into from
Dec 9, 2024

Conversation

gherceg
Copy link
Contributor

@gherceg gherceg commented Nov 21, 2024

Product Description

Technical Summary

https://dimagi.atlassian.net/browse/SAAS-16282

@nospame identified an issue in how we store calculated properties for domains in elasticsearch. How ElasticProcessor in pillows is setup to always index documents to elasticsearch. We could make changes to the processor so that it updates instead indexes (merges new changes into the existing doc instead of replacing any existing doc). However there felt like a lot of unknowns associated with that solution, and that it would be easier to just create a new SQL table to store calculated properties for domains into.

That is where this PR comes in. This is the first in a series of PRs to switch our calculated properties from ES to SQL. We went with the name DomainStats since that felt concise and appropriate, but are open to suggestions. We also chose to put it in the data_analytics app since it aligns with that idea and is similar in function to reports like the MALT. This means other calculated properties code will be moved to this app as we work through this.

Feature Flag

Safety Assurance

Safety story

Adds a new, unused table.

Automated test coverage

QA Plan

No

Migrations

  • The migrations in this code can be safely applied first independently of the code

Rollback instructions

  • This PR can be reverted after deploy with no further considerations

Labels & Review

  • Risk label is set correctly
  • The set of people pinged as reviewers is appropriate for the level of risk of the change

@dimagimon dimagimon added the reindex/migration Reindex or migration will be required during or before deploy label Nov 21, 2024
@gherceg gherceg marked this pull request as ready for review November 22, 2024 13:57
@gherceg gherceg changed the title Create DomainStats SQL table Create DomainMetrics SQL table Dec 3, 2024
@gherceg
Copy link
Contributor Author

gherceg commented Dec 3, 2024

Still viewing this as a WIP. I wanted to have a conversation around possible defaults for these fields, but also think it is the case that when we first create a DomainMetrics row for a domain, we should be populating every column. Once it exists, we only need to update columns that have changed. That is my logic for not providing any default values, nullable fields, or blank fields.

corehq/apps/data_analytics/models.py Outdated Show resolved Hide resolved
corehq/apps/data_analytics/models.py Outdated Show resolved Hide resolved
corehq/apps/data_analytics/models.py Outdated Show resolved Hide resolved
corehq/apps/data_analytics/models.py Outdated Show resolved Hide resolved
Naming and nullable DateTimeFields
Copy link
Contributor

@nospame nospame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have thoughts about the best way(s) to map the model fields to the DomainMetadataResource calculated properties? I do like the naming here better overall. It also means the new version of the task that populates these properties will have to use the new names, but that's mostly a note for myself since I expect to do that work.

@gherceg
Copy link
Contributor Author

gherceg commented Dec 9, 2024

I imagined a helper method on this class that basically returns a dictionary of key/values analogous to what is currently returned from the ES doc.

@gherceg gherceg merged commit 6df8883 into master Dec 9, 2024
13 checks passed
@gherceg gherceg deleted the gh/calc-props/sql-table branch December 9, 2024 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
reindex/migration Reindex or migration will be required during or before deploy
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants