Chore/plugin scaffolding (#1)

* chore: run cookiecutter-template

Signed-off-by: F.N. Claessen <[email protected]>

* chore: update to CI pipeline to Python 3.11

Signed-off-by: F.N. Claessen <[email protected]>

* chore: add classifier for Python 3.12

Signed-off-by: F.N. Claessen <[email protected]>

* chore: add s2-python dependency

Signed-off-by: F.N. Claessen <[email protected]>

* chore: run pre-commit hooks

Signed-off-by: F.N. Claessen <[email protected]>

* feat: add S2Scheduler class

Signed-off-by: F.N. Claessen <[email protected]>

* feat: add importable variable S2_SCHEDULER_SPECS to set a custom scheduler

Signed-off-by: F.N. Claessen <[email protected]>

---------

Signed-off-by: F.N. Claessen <[email protected]>

.flake8

@@ -0,0 +1,7 @@
+[flake8]
+exclude = .git,__pycache__,documentation
+max-line-length = 160
+max-complexity = 13
+select = B,C,E,F,W,B9
+ignore = E501, W503, E203
+

.github/issue-branch.yml

@@ -0,0 +1,2 @@
+openDraftPR: true
+autoCloseIssue: true

.github/workflows/lint-and-test.yml

@@ -0,0 +1,33 @@
+name: lint-and-test
+
+on: push
+
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    name: "Check (on Python3.11)"
+    steps:
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.11
+      - uses: actions/checkout@v3
+      - uses: pre-commit/[email protected]
+
+  test:
+    needs: check
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+    name: "Test (on Python 3.11)"
+    steps:
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.11
+      - name: Check out src from Git
+        uses: actions/checkout@v3
+      - name: Get history and tags for SCM versioning to work
+        run: |
+          git fetch --prune --unshallow
+          git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+      - run: make test

.gitignore

@@ -20,6 +20,7 @@ parts/
 sdist/
 var/
 wheels/
+pip-wheel-metadata/
 share/python-wheels/
 *.egg-info/
 .installed.cfg
@@ -77,6 +78,7 @@ target/
 
 # Jupyter Notebook
 .ipynb_checkpoints
+notebooks/.ipynb_checkpoints/
 
 # IPython
 profile_default/
@@ -169,3 +171,8 @@ cython_debug/
 
 # PyPI configuration file
 .pypirc
+
+# custom project files
+.vscode
+*.pickle
+flexmeasures.log

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+repos:
+-   repo: https://github.com/pycqa/flake8
+    rev: 6.0.0  # New version tags can be found here: https://github.com/pycqa/flake8/tags
+    hooks:
+    - id: flake8
+      name: flake8 (code linting)
+-   repo: https://github.com/psf/black
+    rev: 22.10.0  # New version tags can be found here: https://github.com/psf/black/tags
+    hooks:
+    - id: black
+      name: black (code formatting)
+-   repo: local
+    hooks:
+    - id: mypy
+      name: mypy (static typing)
+      pass_filenames: false
+      language: script
+      entry: run_mypy.sh
+      verbose: true

Dockerfile

@@ -0,0 +1,9 @@
+FROM lfenergy/flexmeasures
+
+# Install requirements, e.g. like this
+#COPY requirements/app.in /app/requirements/flexmeasures_s2.txt
+#RUN pip3 install --no-cache-dir -r requirements/flexmeasures_s2.txt
+
+COPY flexmeasures_s2/ /app/flexmeasures_s2
+# Make sure FlexMeasures recognizes this plugin (requires FlexMeasures v0.14)
+ENV FLEXMEASURES_PLUGINS="/app/flexmeasures_s2"

Makefile

@@ -0,0 +1,45 @@
+# Note: use tabs
+# actions which are virtual, i.e. not a script
+.PHONY: install install-for-dev install-deps install-flexmeasures-s2 test freeze-deps upgrade-deps
+
+
+# ---- Development ---
+
+test:
+	make install-for-dev
+	pytest
+
+# ---- Installation ---
+
+install: install-deps install-flexmeasures-s2
+
+install-for-dev:
+	make freeze-deps
+	pip-sync requirements/app.txt requirements/dev.txt requirements/test.txt
+	make install-flexmeasures-s2
+	pre-commit install
+
+install-deps:
+	make install-pip-tools
+	make freeze-deps
+	pip-sync requirements/app.txt
+
+install-flexmeasures-s2:
+	python setup.py develop
+
+install-pip-tools:
+	pip3 install -q "pip-tools>=6.2"
+
+freeze-deps:
+	make install-pip-tools
+	pip-compile -o requirements/app.txt requirements/app.in
+	pip-compile -o requirements/dev.txt requirements/dev.in
+	pip-compile -o requirements/test.txt requirements/test.in
+
+upgrade-deps:
+	make install-pip-tools
+	pip-compile --upgrade -o requirements/app.txt requirements/app.in
+	pip-compile --upgrade -o requirements/test.txt requirements/test.in
+	pip-compile --upgrade -o requirements/dev.txt requirements/dev.in
+	make test
+

README.md

@@ -1 +1,31 @@
-# flexmeasures-s2
+# flexmeasures-s2 - a plugin for FlexMeasures
+
+
+## Usage
+
+
+## Installation
+
+1. Add "/path/to/flexmeasures-s2/flexmeasures_s2" to your FlexMeasures (>v0.7.0dev8) config file,
+   using the FLEXMEASURES_PLUGINS setting (a list).
+   Alternatively, if you installed this plugin as a package (e.g. via `python setup.py install`, `pip install -e` or `pip install flexmeasures_s2` should this project be on Pypi), then "flexmeasures_s2" suffices.
+
+2.  
+
+
+## Development
+
+We use pre-commit to keep code quality up.
+
+Install necessary tools with:
+
+    pip install pre-commit black flake8 mypy
+    pre-commit install
+
+or:
+
+    make install-for-dev
+
+Try it:
+
+    pre-commit run --all-files --show-diff-on-failure

flexmeasures_s2/__init__.py

@@ -0,0 +1,45 @@
+__version__ = "Unknown version"
+
+
+"""
+The __init__ for the flexmeasures-s2 FlexMeasures plugin.
+
+FlexMeasures registers the BluePrint objects it finds in here.
+"""
+
+
+from importlib_metadata import version, PackageNotFoundError
+
+from flask import Blueprint
+
+from .utils import ensure_bp_routes_are_loaded_fresh
+
+# Overwriting version (if possible) from the package metadata
+# ― if this plugin has been installed as a package.
+# This uses importlib.metadata behaviour added in Python 3.8.
+# Note that we rely on git tags (via setuptools_scm) to define that version.
+try:
+    __version__ = version("flexmeasures_s2")
+except PackageNotFoundError:
+    # package is not installed
+    pass
+
+# API
+flexmeasures_s2_api_bp: Blueprint = Blueprint(
+    "flexmeasures-s2 API", __name__, url_prefix="/flexmeasures-s2/api"
+)
+ensure_bp_routes_are_loaded_fresh("api.somedata")
+from flexmeasures_s2.api import somedata  # noqa: E402,F401
+
+
+# Use as follows:
+# from flexmeasures import Sensor
+# from flexmeasures_s2 import S2_SCHEDULER_SPECS
+# my_sensor = Sensor.query.filter(
+#     Sensor.name == "My power sensor on a flexible asset"
+# ).one_or_none()
+# my_sensor.attributes["custom-scheduler"] = S2_SCHEDULER_SPECS
+S2_SCHEDULER_SPECS = {
+    "module": "flexmeasures_s2.scheduler.schedulers",
+    "class": "S2Scheduler",
+}

flexmeasures_s2/api/somedata.py

@@ -0,0 +1,11 @@
+from flask_security import auth_token_required
+from flask_json import as_json
+
+from .. import flexmeasures_s2_api_bp
+
+
+@flexmeasures_s2_api_bp.route("/somedata")
+@auth_token_required
+@as_json
+def somedata():
+    return dict(a=1, b=2)

flexmeasures_s2/api/tests/test_api.py

@@ -0,0 +1,16 @@
+from flask import url_for
+
+
+def test_get_somedata_needs_authtoken(client):
+    response = client.get(
+        url_for("flexmeasures-s2 API.somedata"),
+        headers={"content-type": "application/json"},
+        follow_redirects=True,
+    )
+    assert response.status_code == 401  # HTTP error code 401 Unauthorized.
+    assert "application/json" in response.content_type
+    assert "not be properly authenticated" in response.json["message"]
+
+
+# TODO: The somedata endpoint requires authentication to be testes successfully.
+#       We'll need to add a user in conftest, which also requires us to add a db to testing

flexmeasures_s2/conftest.py

@@ -0,0 +1,27 @@
+import pytest
+
+from flexmeasures.app import create as create_flexmeasures_app
+from flexmeasures.conftest import (  # noqa F401
+    db,
+    fresh_db,
+)  # Use these fixtures to rely on the FlexMeasures database.
+
+# There might be others in flexmeasures/conftest you want to also re-use
+
+
+@pytest.fixture(scope="session")
+def app():
+    print("APP FIXTURE")
+
+    # Adding this plugin, making sure the name is known (as last part of plugin path)
+    test_app = create_flexmeasures_app(env="testing", plugins=["../flexmeasures_s2"])
+
+    # Establish an application context before running the tests.
+    ctx = test_app.app_context()
+    ctx.push()
+
+    yield test_app
+
+    ctx.pop()
+
+    print("DONE WITH APP FIXTURE")

flexmeasures_s2/scheduler/schedulers.py

@@ -0,0 +1,24 @@
+import pandas as pd
+from flexmeasures import Scheduler
+
+
+class S2Scheduler(Scheduler):
+
+    __author__ = "TNO"
+    __version__ = "1"
+
+    def compute(self, *args, **kwargs):
+        """
+        Just a dummy scheduler that always plans to consume at maximum capacity.
+        (Schedulers return positive values for consumption, and negative values for production)
+        """
+        return pd.Series(
+            self.sensor.get_attribute("capacity_in_mw"),
+            index=pd.date_range(
+                self.start, self.end, freq=self.resolution, inclusive="left"
+            ),
+        )
+
+    def deserialize_config(self):
+        """Do not care about any flex config sent in."""
+        self.config_deserialized = True

flexmeasures_s2/scheduler/schemas.py

@@ -0,0 +1,5 @@
+from marshmallow import Schema
+
+
+class S2FlexModelSchema(Schema):
+    ...

flexmeasures_s2/utils.py

@@ -0,0 +1,22 @@
+import sys
+import importlib
+
+
+def ensure_bp_routes_are_loaded_fresh(module_name):
+    """
+    Reload a module if it has been loaded before.
+    It's useful for situations in which some other process has read
+    the module before, but you need some action to happen which only
+    happens during module import ― decorators are a good example.
+
+    One use case is pytest, which reads all python code when it collects tests.
+    In our case, that happens before FlexMeasures' import mechanism
+    has had a chance to know which blueprints a plugin has. And
+    Seemingly, the importing code (plugin's __init__) can be imported later
+    than the imported module (containing @route decorators).
+    Re-importing helps to get this order right when FlexMeasures reads the
+    plugin's __init__.
+    """
+    m_name = "flexmeasures_s2." + module_name
+    if m_name in sys.modules:
+        importlib.reload(sys.modules[m_name])

requirements/Readme.md

@@ -0,0 +1,21 @@
+# Requirements
+
+All requirements for flexmeasures-flexmeasures-s2 are specified in this directory.
+We separate by use case:
+
+- app: All requirements for running the plugin
+- test: Additional requirements used for running automated tests 
+- dev: Additional requirements used for developers (this includes testing)
+
+Also note the following distinction:
+
+
+## .in files
+
+Here, we describe the requirements. We give the name of a requirement or even a range (e.g. `>=1.0.`).
+
+## .txt files
+
+These files are not to be edited by hand. They are created by `pip-compile` (or `make freeze-deps`).
+
+Each requirement is pinned to a specific version in these files. The great benefit is reproducibility across environments (local dev as well as staging or production).

requirements/app.in

@@ -0,0 +1,2 @@
+flexmeasures>=0.24.0
+s2-python