Skip to content

Commit

Permalink
Merge pull request #1276 from BLSQ/TRIA-7-small-fixes
Browse files Browse the repository at this point in the history
TRIA-7 Small fixes
  • Loading branch information
bramj authored May 6, 2024
2 parents e396d99 + 4722cf3 commit 65037fd
Show file tree
Hide file tree
Showing 6 changed files with 32 additions and 36 deletions.
25 changes: 9 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -843,29 +843,22 @@ Alternatively this can be done outside of docker by running:

# Background tasks & worker

Iaso queue certain functions (task) for later execution, so they can run
outside an HTTP request. This is used for functions that take a long time to execute,
so they don't canceled in the middle by a timeout of a connection closed.
e.g: bulk import, modifications or export of OrgUnits. Theses are the functions
marked by the decorator @task_decorator, when called they get added to a Queue
and get executed by a worker.
Iaso queues certain functions (tasks) for later execution, so they can run outside an HTTP request. This is used for functions that take a long time to execute, so they're not canceled in the middle by a timeout of a connection closed.
Examples include: bulk import, modifications or export of OrgUnits.
These functions are marked by the decorator `@task_decorator`. When called, they get added to a Queue and get executed by a worker.

In local development, you can run a worker by using the command:

```
docker-compose run iaso manage tasks_worker
```

Alternatively, you can call the url `tasks/run_all` which will run all the pending tasks in queue.

If you want to develop a new background task, the endpoint `/api/copy_version/`
is a good example of how to create a task and to plug it to the api.
If you want to develop a new background task, the endpoint `/api/copy_version/` is a good example of how to create a task and to plug it to the api.

To call a function with the @task decorator, you need to pass it a User objects, in addition to
the other function's arguments, this arg represent which user is launching
the task. At execution time the task will receive a iaso.models.Task
instance in argument that should be used to report progress. It's
mandatory for the function, at the end of a successful execution to call
task.report_success() to mark its proper completion.
To call a function with the `@task_decorator`, you need to pass it a `User` object, in addition to the other function's arguments. The user argument represents the user that is launching the task. At execution time the task will receive a `iaso.models.Task` instance as an argument that can be used to report progress. It's
mandatory for the function to call `task.report_success()` at the end of a successful execution to mark its proper completion.

We have two background workers mechanisms: a postgres backed one, and a SQS backed one. You can choose which one to use with the `BACKGROUND_TASK_SERVICE` environment variable, use either `SQS` or `POSTGRES` (it defaults to `SQS` in production).

Expand All @@ -877,6 +870,6 @@ In production on AWS, we use Elastic Beanstalk workers which use a SQS queue. Th

## Postgres

This is also the one that you get when running locally with `docker-compose run iaso manage tasks_worker`, instead of enqueuing the tasks to SQS, we now enqueue them to our postgres server.
This is the one you get when running locally with `docker-compose run iaso manage tasks_worker`. Instead of enqueuing the tasks to SQS, we enqueue them to our postgres server.

Our tasks_worker process (which runs indefinitely) will listen for new tasks and run them when it gets notified (using PostgreSQL NOTIFY/LISTEN features)
Our `tasks_worker` process (which runs indefinitely) will listen for new tasks and run them when it gets notified (using PostgreSQL NOTIFY/LISTEN features).
2 changes: 1 addition & 1 deletion hat/assets/js/apps/Iaso/domains/app/components/messages.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ const MESSAGES = defineMessages({
id: 'iaso.tooltip.viewUserManual',
},
iasoVersion: {
defautMessage: 'App version',
defaultMessage: 'App version',
id: 'iaso.label.iasoVersion',
},
});
Expand Down
8 changes: 4 additions & 4 deletions hat/assets/js/apps/Iaso/domains/dataSources/messages.js
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ const MESSAGES = defineMessages({
id: 'iaso.snackBar.importFromDhis2Success',
},
importFromDhis2Error: {
defaulteEssage: 'An error occurred while importing from DHIS2',
defaultMessage: 'An error occurred while importing from DHIS2',
id: 'iaso.snackBar.importFromDhis2Error',
},
exportToDhis2Success: {
Expand All @@ -323,15 +323,15 @@ const MESSAGES = defineMessages({
id: 'iaso.snackBar.importGpkgSuccess',
},
importGpkgError: {
defaultMEssage: 'An error occurred while importing the gpkg file',
defaultMessage: 'An error occurred while importing the gpkg file',
id: 'iaso.snackBar.importGpkgError',
},
checkDhis2Success: {
defaultMEssage: 'Connection to server ok',
defaultMessage: 'Connection to server ok',
id: 'iaso.dataSources.checkDHIS2.success',
},
checkDhis2Error: {
defaultMEssage: 'Connection Error check settings',
defaultMessage: 'Connection Error check settings',
id: 'iaso.dataSources.checkDHIS2.error',
},
newEmptyVersionSavedSuccess: {
Expand Down
14 changes: 5 additions & 9 deletions iaso/api/completeness_stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -390,12 +390,10 @@ def to_dict(row_ou: OrgUnitWithFormStat):
return {
"name": row_ou.name,
"id": row_ou.id,
"org_unit": row_ou.as_dict_for_completeness_stats(),
"org_unit": row_ou.as_minimal_dict(),
"form_stats": row_ou.form_stats,
"org_unit_type": row_ou.org_unit_type.as_dict_for_completeness_stats() if row_ou.org_unit_type else {},
"parent_org_unit": (
row_ou.parent.as_dict_for_completeness_stats_with_parent() if row_ou.parent else None
),
"org_unit_type": row_ou.org_unit_type.as_minimal_dict() if row_ou.org_unit_type else {},
"parent_org_unit": row_ou.parent.as_minimal_dict_with_parent() if row_ou.parent else None,
"has_children": has_children(row_ou),
}

Expand All @@ -409,10 +407,8 @@ def to_map(row_ou: OrgUnitWithFormStat):
"latitude": row_ou.location.y if row_ou.location else None,
"longitude": row_ou.location.x if row_ou.location else None,
"altitude": row_ou.location.z if row_ou.location else None,
"org_unit_type": row_ou.org_unit_type.as_dict_for_completeness_stats() if row_ou.org_unit_type else {},
"parent_org_unit": (
row_ou.parent.as_dict_for_completeness_stats_with_parent() if row_ou.parent else None
),
"org_unit_type": row_ou.org_unit_type.as_minimal_dict() if row_ou.org_unit_type else {},
"parent_org_unit": row_ou.parent.as_minimal_dict_with_parent() if row_ou.parent else None,
"has_children": has_children(row_ou),
}
if temp_org_unit["has_geo_json"] == True:
Expand Down
2 changes: 1 addition & 1 deletion iaso/models/forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def as_dict(self, additional_fields=None, show_version=True):

return res

def as_dict_for_completeness_stats(self):
def as_minimal_dict(self):
return {
"name": self.name,
"id": self.id,
Expand Down
17 changes: 12 additions & 5 deletions iaso/models/org_unit.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,14 @@ def filter_for_user_and_app_id(
if user and user.is_anonymous and app_id is None:
return self.none()

queryset = self.all()
queryset = self.prefetch_related(
"projects",
"projects__account",
"projects__feature_flags",
"allow_creating_sub_unit_types",
"reference_forms",
"sub_unit_types",
)

if user and user.is_authenticated:
queryset = queryset.filter(projects__account=user.iaso_profile.account)
Expand Down Expand Up @@ -146,7 +153,7 @@ def as_dict(self, sub_units=True, app_id=None):
res["sub_unit_types"] = sub_unit_types
return res

def as_dict_for_completeness_stats(self):
def as_minimal_dict(self):
return {
"name": self.name,
"id": self.id,
Expand Down Expand Up @@ -501,14 +508,14 @@ def as_dict_for_csv(self):
"org_unit_type": self.org_unit_type.name,
}

def as_dict_for_completeness_stats_with_parent(self):
def as_minimal_dict_with_parent(self):
return {
"name": self.name,
"id": self.id,
"parent": self.parent.as_dict_for_completeness_stats() if self.parent else None,
"parent": self.parent.as_minimal_dict() if self.parent else None,
}

def as_dict_for_completeness_stats(self):
def as_minimal_dict(self):
return {
"name": self.name,
"id": self.id,
Expand Down

0 comments on commit 65037fd

Please sign in to comment.