Background sync for datasets from CommCare HQ

README.md

@@ -6,9 +6,14 @@ This is a Python package that integrates Superset and CommCare HQ.
 Local Development
 -----------------
 
-Follow below instructions.
+### Preparing CommCare HQ
 
-### Setup env
+The 'User configurable reports UI' feature flag must be enabled for the
+domain in CommCare HQ, even if the data sources to be imported were
+created by Report Builder, not a UCR.
+
+
+### Setting up a dev environment
 
 While doing development on top of this integration, it's useful to
 install this via `pip -e` option so that any changes made get reflected
@@ -51,11 +56,12 @@ directly without another `pip install`.
 Read through the initialization instructions at
 https://superset.apache.org/docs/installation/installing-superset-from-scratch/#installing-and-initializing-superset.
 
-Create the database. These instructions assume that PostgreSQL is
-running on localhost, and that its user is "commcarehq". Adapt
-accordingly:
+Create a database for Superset, and a database for storing data from
+CommCare HQ. Adapt the username and database names to suit your
+environment.
 ```bash
-$ createdb -h localhost -p 5432 -U commcarehq superset_meta
+$ createdb -h localhost -p 5432 -U postgres superset
+$ createdb -h localhost -p 5432 -U postgres superset_hq_data
 ```
 
 Set the following environment variables:
@@ -64,10 +70,17 @@ $ export FLASK_APP=superset
 $ export SUPERSET_CONFIG_PATH=/path/to/superset_config.py
 ```
 
-Initialize the database. Create an administrator. Create default roles
+Set this environment variable to allow OAuth 2.0 authentication with
+CommCare HQ over insecure HTTP. (DO NOT USE THIS IN PRODUCTION.)
+```bash
+$ export AUTHLIB_INSECURE_TRANSPORT=1
+```
+
+Initialize the databases. Create an administrator. Create default roles
 and permissions:
 ```bash
 $ superset db upgrade
+$ superset db upgrade --directory hq_superset/migrations/
 $ superset fab create-admin
 $ superset load_examples  # (Optional)
 $ superset init
@@ -78,31 +91,34 @@ You should now be able to run superset using the `superset run` command:
 ```bash
 $ superset run -p 8088 --with-threads --reload --debugger
 ```
-However, OAuth login does not work yet as hq-superset needs a Postgres
-database created to store CommCare HQ data.
+
+You can now log in as a CommCare HQ web user.
+
+In order for CommCare HQ to sync data source changes, you will need to
+allow OAuth 2.0 authentication over insecure HTTP. (DO NOT USE THIS IN
+PRODUCTION.) Set this environment variable in your CommCare HQ Django
+server. (Yes, it's "OAUTHLIB" this time, not "AUTHLIB" as before.)
+```bash
+$ export OAUTHLIB_INSECURE_TRANSPORT=1
+```
 
 
-### Create a Postgres Database Connection for storing HQ data
+### Logging in as a local admin user
 
-- Create a Postgres database. e.g.
-  ```bash
-  $ createdb -h localhost -p 5432 -U commcarehq hq_data
-  ```
-- Log into Superset as the admin user created in the Superset
-  installation and initialization. Note that you will need to update
-  `AUTH_TYPE = AUTH_DB` to log in as admin user. `AUTH_TYPE` should be
-  otherwise set to `AUTH_OAUTH`.
-- Go to 'Data' -> 'Databases' or http://127.0.0.1:8088/databaseview/list/
-- Create a database connection by clicking '+ DATABASE' button at the top.
-- The name of the DISPLAY NAME should be 'HQ Data' exactly, as this is
-  the name by which this codebase refers to the Postgres DB.
+There might be situations where you need to log into Superset as a local
+admin user, for example, to add a database connection. To enable local
+user authentication, in `superset_config.py`, set
+`AUTH_TYPE = AUTH_DB`.
 
-OAuth integration should now be working. You can log in as a CommCare
-HQ web user.
+Doing this will prevent CommCare HQ users from logging in, so it should
+only be done in production environments when CommCare Analytics is not
+in use.
 
+To return to allowing CommCare HQ users to log in, set it back to
+`AUTH_TYPE = AUTH_OAUTH`.
 
-### Importing UCRs using Redis and Celery
 
+### Importing UCRs using Redis and Celery
 
 Celery is used to import UCRs that are larger than
 `hq_superset.views.ASYNC_DATASOURCE_IMPORT_LIMIT_IN_BYTES`. If you need
@@ -137,6 +153,41 @@ code you want to test will need to be in a module whose dependencies
 don't include Superset.
 
 
+### Creating a migration
+
+You will need to create an Alembic migration for any new SQLAlchemy
+models that you add. The Superset CLI should allow you to do this:
+
+```shell
+$ superset db revision --autogenerate -m "Add table for Foo model"
+```
+
+However, problems with this approach have occurred in the past. You
+might have more success by using Alembic directly. You will need to
+modify the configuration a little to do this:
+
+1. Copy the "HQ_DATA" database URI from `superset_config.py`.
+
+2. Paste it as the value of `sqlalchemy.url` in
+   `hq_superset/migrations/alembic.ini`.
+
+3. Edit `env.py` and comment out the following lines:
+   ```
+   hq_data_uri = current_app.config['SQLALCHEMY_BINDS'][HQ_DATA]
+   decoded_uri = urllib.parse.unquote(hq_data_uri)
+   config.set_main_option('sqlalchemy.url', decoded_uri)
+   ```
+
+Those changes will allow Alembic to connect to the "HD Data" database
+without the need to instantiate Superset's Flask app. You can now
+autogenerate your new table with:
+
+```shell
+$ cd hq_superset/migrations/
+$ alembic revision --autogenerate -m "Add table for Foo model"
+```
+
+
 Upgrading Superset
 ------------------
 

hq_superset/__init__.py

@@ -9,10 +9,14 @@ def flask_app_mutator(app):
     # return
     from superset.extensions import appbuilder
 
-    from . import hq_domain, views
+    from . import api, hq_domain, oauth2_server, views
 
     appbuilder.add_view(views.HQDatasourceView, 'Update HQ Datasource', menu_cond=lambda *_: False)
     appbuilder.add_view(views.SelectDomainView, 'Select a Domain', menu_cond=lambda *_: False)
+    appbuilder.add_api(api.OAuth)
+    appbuilder.add_api(api.DataSetChangeAPI)
+    oauth2_server.config_oauth2(app)
+
     app.before_request_funcs.setdefault(None, []).append(
         hq_domain.before_request_hook
     )

hq_superset/api.py

@@ -0,0 +1,69 @@
+import json
+from http import HTTPStatus
+
+from flask import jsonify, request
+from flask_appbuilder.api import BaseApi, expose
+from sqlalchemy.orm.exc import NoResultFound
+from superset.superset_typing import FlaskResponse
+from superset.views.base import (
+    handle_api_exception,
+    json_error_response,
+    json_success,
+)
+
+from .models import DataSetChange
+from .oauth2_server import authorization, require_oauth
+
+
+class OAuth(BaseApi):
+
+    def __init__(self):
+        super().__init__()
+        self.route_base = "/oauth"
+
+    @expose("/token", methods=('POST',))
+    def issue_access_token(self):
+        try:
+            response = authorization.create_token_response()
+        except NoResultFound:
+            return jsonify({"error": "Invalid client"}), 401
+
+        if response.status_code >= 400:
+            return response
+
+        data = json.loads(response.data.decode("utf-8"))
+        return jsonify(data)
+
+
+class DataSetChangeAPI(BaseApi):
+    """
+    Accepts changes to datasets from CommCare HQ data forwarding
+    """
+
+    MAX_REQUEST_LENGTH = 10 * 1024 * 1024  # reject JSON requests > 10MB
+
+    def __init__(self):
+        self.route_base = '/commcarehq_dataset'
+        self.default_view = 'post_dataset_change'
+        super().__init__()
+
+    @expose('/change/', methods=('POST',))
+    @handle_api_exception
+    @require_oauth()
+    def post_dataset_change(self) -> FlaskResponse:
+        if request.content_length > self.MAX_REQUEST_LENGTH:
+            return json_error_response(
+                HTTPStatus.REQUEST_ENTITY_TOO_LARGE.description,
+                status=HTTPStatus.REQUEST_ENTITY_TOO_LARGE.value,
+            )
+
+        try:
+            request_json = json.loads(request.get_data(as_text=True))
+            change = DataSetChange(**request_json)
+            change.update_dataset()
+            return json_success('Dataset updated')
+        except json.JSONDecodeError:
+            return json_error_response(
+                'Invalid JSON syntax',
+                status=HTTPStatus.BAD_REQUEST.value,
+            )

hq_superset/const.py

@@ -0,0 +1,4 @@
+# The name of the database for storing data related to CommCare HQ
+HQ_DATABASE_NAME = "HQ Data"
+
+OAUTH2_DATABASE_NAME = "oauth2-server-data"

hq_superset/exceptions.py

@@ -1,3 +1,7 @@
+class DatabaseMissing(Exception):
+    pass
+
+
 class HQAPIException(Exception):
     pass
 
@@ -6,5 +10,5 @@ class OAuthSessionExpired(Exception):
     pass
 
 
-class UnitTestingOnly(Exception):
-  pass
+class TableMissing(Exception):
+    pass

hq_superset/hq_domain.py

@@ -19,15 +19,17 @@ def after_request_hook(response):
 
 
 DOMAIN_EXCLUDED_VIEWS = [
-    "AuthOAuthView.login",
-    "AuthOAuthView.logout",
-    "AuthOAuthView.oauth_authorized",
-    "AuthDBView.logout",
-    "AuthDBView.login",
-    "SelectDomainView.list",
-    "SelectDomainView.select",
-    "appbuilder.static",
-    "static",
+    'AuthDBView.login',
+    'AuthDBView.logout',
+    'AuthOAuthView.login',
+    'AuthOAuthView.logout',
+    'AuthOAuthView.oauth_authorized',
+    'DataSetChangeAPI.post_dataset_change',
+    'OAuth.issue_access_token',
+    'SelectDomainView.list',
+    'SelectDomainView.select',
+    'appbuilder.static',
+    'static',
 ]
 
 
@@ -39,7 +41,10 @@ def is_user_admin():
 def ensure_domain_selected():
     # Check if a hq_domain cookie is set
     #   Ensure necessary roles, permissions and DB schemas are created for the domain
-    if is_user_admin() or (request.url_rule and request.url_rule.endpoint in DOMAIN_EXCLUDED_VIEWS):
+    if is_user_admin() or (
+        request.url_rule
+        and request.url_rule.endpoint in DOMAIN_EXCLUDED_VIEWS
+    ):
         return
     hq_domain = request.cookies.get('hq_domain')
     valid_domains = user_domains()

hq_superset/hq_requests.py

@@ -0,0 +1,30 @@
+import superset
+from hq_superset.oauth import get_valid_cchq_oauth_token
+
+
+class HQRequest:
+
+    def __init__(self, url):
+        self.url = url
+
+    @property
+    def oauth_token(self):
+        return get_valid_cchq_oauth_token()
+
+    @property
+    def commcare_provider(self):
+        return superset.appbuilder.sm.oauth_remotes["commcare"]
+
+    @property
+    def api_base_url(self):
+        return self.commcare_provider.api_base_url
+
+    @property
+    def absolute_url(self):
+        return f"{self.api_base_url}{self.url}"
+
+    def get(self):
+        return self.commcare_provider.get(self.url, token=self.oauth_token)
+
+    def post(self, data):
+        return self.commcare_provider.post(self.url, data=data, token=self.oauth_token)

hq_superset/hq_url.py

@@ -0,0 +1,25 @@
+"""
+Functions that return URLs on CommCare HQ
+"""
+
+
+def datasource_details(domain, datasource_id):
+    return f"a/{domain}/api/v0.5/ucr_data_source/{datasource_id}/"
+
+
+def datasource_list(domain):
+    return f"a/{domain}/api/v0.5/ucr_data_source/"
+
+
+def datasource_export(domain, datasource_id):
+    return (
+        f"a/{domain}/configurable_reports/data_sources/export/{datasource_id}/"
+        "?format=csv"
+    )
+
+
+def datasource_subscribe(domain, datasource_id):
+    return (
+        f"a/{domain}/configurable_reports/data_sources/subscribe/"
+        f"{datasource_id}/"
+    )