diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml index feed2895e..1b78388ea 100644 --- a/.github/workflows/lint-and-test.yml +++ b/.github/workflows/lint-and-test.yml @@ -23,7 +23,7 @@ jobs: strategy: fail-fast: false matrix: - py_version: [ "3.8", "3.9", "3.10", "3.11" ] + py_version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ] include: - python-version: "3.9" coverage: yes diff --git a/.gitignore b/.gitignore index b2168edb8..68ab1c2c9 100644 --- a/.gitignore +++ b/.gitignore @@ -47,4 +47,7 @@ poetry.lock .DS_Store -.gitconfig.* \ No newline at end of file +.gitconfig.* + +/postgres-data +coverage.lcov \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4c1644a4c..042fe6c09 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,11 +1,11 @@ repos: - repo: https://github.com/pycqa/flake8 - rev: 6.0.0 # New version tags can be found here: https://github.com/pycqa/flake8/tags + rev: 7.1.1 # New version tags can be found here: https://github.com/pycqa/flake8/tags hooks: - id: flake8 name: flake8 (code linting) - repo: https://github.com/psf/black - rev: 22.10.0 # New version tags can be found here: https://github.com/psf/black/tags + rev: 24.8.0 # New version tags can be found here: https://github.com/psf/black/tags hooks: - id: black name: black (code formatting) diff --git a/Makefile b/Makefile index 78f35bdfe..070096b71 100644 --- a/Makefile +++ b/Makefile @@ -43,6 +43,10 @@ install-for-dev: make ensure-deps-folder pip-sync requirements/${PYV}/app.txt requirements/${PYV}/dev.txt requirements/${PYV}/test.txt make install-flexmeasures +# Locally install HiGS on macOS + if [ "$(shell uname)" = "Darwin" ]; then \ + make install-highs-macos; \ + fi install-for-test: make install-pip-tools @@ -50,10 +54,7 @@ install-for-test: ifneq ($(pinned), no) pip-sync requirements/${PYV}/app.txt requirements/${PYV}/test.txt else - # cutting off the -c inter-layer dependency (that's pip-tools specific) - tail -n +3 requirements/test.in >> temp-test.in - pip install --upgrade -r requirements/app.in -r temp-test.in - rm temp-test.in + pip install --upgrade -r requirements/app.in -r requirements/test.in endif make install-flexmeasures # Locally install HiGS on macOS diff --git a/ci/update-packages.sh b/ci/update-packages.sh index 9ee226d40..0c72da98c 100755 --- a/ci/update-packages.sh +++ b/ci/update-packages.sh @@ -12,7 +12,7 @@ set -e set -x -PYTHON_VERSIONS=(3.8 3.9 3.10 3.11) +PYTHON_VERSIONS=(3.8 3.9 3.10 3.11 3.12) # check if we will upgrade or just freeze UPDATE_CMD=freeze-deps diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index 3f32d6313..7c67d6a6a 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -5,18 +5,81 @@ API change log .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace. +v3.0-20 | 2024-09-18 +"""""""""""""""""""" + +- Introduce (optional) pagination to the endpoint `/assets` (GET), also adding the `all_accessible` option to allow querying all accessible accounts in one go. + + +v3.0-19 | 2024-08-13 +"""""""""""""""""""" + +- Allow passing a fixed price in the ``flex-context`` using the new fields ``consumption-price`` and ``production-price``, which are meant to replace the ``consumption-price-sensor`` and ``production-price-sensor`` fields, respectively. +- Allow posting a single instantaneous belief as a list of one element to `/sensors/data` (POST). +- Allow setting a SoC unit directly in some fields (formerly ``Float`` fields, and now ``Quantity`` fields), while still falling back on the contents of the ``soc-unit`` field, for backwards compatibility: + + - ``soc-at-start`` + - ``soc-min`` + - ``soc-max`` + +- Allow setting a unit directly in fields that already supported passing a time series: + + - ``soc-maxima`` + - ``soc-minima`` + - ``soc-targets`` + +- Allow passing a time series in fields that formerly only accepted passing a fixed quantity or a sensor reference: + + - ``power-capacity`` + - ``consumption-capacity`` + - ``production-capacity`` + - ``charging-efficiency`` + - ``discharging-efficiency`` + - ``storage-efficiency`` + - ``soc-gain`` + - ``soc-usage`` + +- Added API notation section on variable quantities. +- Updated section on scheduling; specifically, most flex-context and flex-model fields are now variable quantity fields, so a footnote now explains the few fields that aren't (yet) a variable quantity field. +- Removed section on singular vs plural keys, which is no longer valid for crucial endpoints. + +v3.0-19 | 2024-08-09 +"""""""""""""""""""" + +- Allow setting a SoC unit directly in some fields (formerly ``Float`` fields, and now ``Quantity`` fields), while still falling back on the contents of the ``soc-unit`` field, for backwards compatibility: + + - ``soc-at-start`` + - ``soc-min`` + - ``soc-max`` + +- Allow setting a unit directly in fields that already supported passing a time series: + + - ``soc-maxima`` + - ``soc-minima`` + - ``soc-targets`` + +- Allow passing a time series in fields that formerly only accepted passing a fixed quantity or a sensor reference: + + - ``power-capacity`` + - ``consumption-capacity`` + - ``production-capacity`` + - ``charging-efficiency`` + - ``discharging-efficiency`` + - ``storage-efficiency`` + - ``soc-gain`` + - ``soc-usage`` + v3.0-18 | 2024-03-07 """""""""""""""""""" -- Add support for providing a sensor definition to the ``soc-minima``, ``soc-maxima`` and ``soc-targets`` flex-model fields for `/sensors//schedules/trigger` (POST). +- Add support for providing a sensor definition to the ``soc-minima``, ``soc-maxima`` and ``soc-targets`` flex-model fields for `/sensors//schedules/trigger` (POST). v3.0-17 | 2024-02-26 """""""""""""""""""" - Add support for providing a sensor definition to the ``site-power-capacity``, ``site-consumption-capacity`` and ``site-production-capacity`` flex-context fields for `/sensors//schedules/trigger` (POST). - v3.0-16 | 2024-02-26 """""""""""""""""""" @@ -55,7 +118,12 @@ v3.0-11 | 2023-08-02 v3.0-10 | 2023-06-12 """""""""""""""""""" -- Introduced the ``storage-efficiency`` field to the ``flex-model``field for `/sensors//schedules/trigger` (POST). +- Introduced new ``flex-model`` fields for `/sensors//schedules/trigger` (POST): + + - ``storage-efficiency`` + - ``soc-minima`` + - ``soc-maxima`` + - Introduced the ``database_redis`` optional field to the response of the endpoint `/health/ready` (GET). v3.0-9 | 2023-04-26 @@ -126,7 +194,6 @@ v3.0-1 | 2022-05-08 - Added REST endpoint for checking application health (readiness to accept requests): `/health/ready` (GET). - v3.0-0 | 2022-03-25 """"""""""""""""""" diff --git a/documentation/api/notation.rst b/documentation/api/notation.rst index b2c4ad41d..13537509a 100644 --- a/documentation/api/notation.rst +++ b/documentation/api/notation.rst @@ -6,20 +6,58 @@ Notation This page helps you to construct messages to the FlexMeasures API. Please consult the endpoint documentation first. Here we dive into topics useful across endpoints. -Singular vs plural keys +.. _variable_quantities: + +Variable quantities ^^^^^^^^^^^^^^^^^^^^^^^ -Throughout this document, keys are written in singular if a single value is listed, and written in plural if multiple values are listed, for example: +Many API fields deal with variable quantities, for example, :ref:`flex-model ` and :ref:`flex-context ` fields. +Unless stated otherwise, values of such fields can take one of the following forms: -.. code-block:: json +- A fixed quantity, to describe steady constraints such as a physical power capacity. + For example: - { - "keyToValue": "this is a single value", - "keyToValues": ["this is a value", "and this is a second value"] - } + .. code-block:: json + + { + "power-capacity": "15 kW" + } + +- A variable quantity defined at specific moments in time, to describe dynamic constraints/preferences such as target states of charge. + + .. code-block:: json + + { + "soc-targets": [ + {"datetime": "2024-02-05T08:00:00+01:00", "value": "8.2 kWh"}, + ... + {"datetime": "2024-02-05T13:00:00+01:00", "value": "2.2 kWh"} + ] + } + +- A variable quantity defined for specific time ranges, to describe dynamic constraints/preferences such as usage forecasts. + + .. code-block:: json + + { + "soc-usage": [ + {"start": "2024-02-05T08:00:00+01:00", "duration": "PT2H", "value": "10.1 kW"}, + ... + {"start": "2024-02-05T13:00:00+01:00", "end": "2024-02-05T13:15:00+01:00", "value": "10.3 kW"} + ] + } + + Note the two distinct ways of specifying a time period (``"end"`` + ``"duration"`` also works). + +- A reference to a sensor that records a variable quantity, which allows cross-referencing to dynamic contexts that are already recorded as sensor data in FlexMeasures. For instance, a site's contracted consumption capacity that changes over time. + + .. code-block:: json -The API, however, does not distinguish between singular and plural key notation. + { + "site-consumption-capacity": {"sensor": 55} + } + The unit of the data is specified on the sensor. Sensors and entity addresses ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 7c02afa9c..ed2a90901 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -3,9 +3,94 @@ FlexMeasures Changelog ********************** + +v0.23.2 | December 16, 2024 +============================ + +Bugfixes +----------- +* Correct source filters on reporter input, and support piping additional transformations in the ``PandasReporter`` [see `PR #1284 `_] + + +v0.23.1 | November 12, 2024 +============================ + +New features +------------- +* The data chart on the asset page splits up its color-coded sensor legend when showing more than 7 sensors, becoming a legend per subplot [see `PR #1176 `_ and `PR #1193 `_] +* Speed up loading the users page, by making the pagination backend-based and adding support for that in the API [see `PR #1160 `] +* X-axis labels in CLI plots show datetime values in a readable and informative format [see `PR #1172 `_] +* Enhanced API for listing sensors: Added filtering and pagination on sensor index endpoint and created new endpoint to get all sensors under an asset [see `PR #1191 `_ and `PR #1219 `_] +* Speed up loading the accounts page,by making the pagination backend-based and adding support for that in the API [see `PR #1196 `_] +* Speed up loading the account detail page by by switching to server-side pagination for assets, replacing client-side pagination [see `PR #1202 `_] +* Simplify and Globalize toasts in the flexmeasures project [see `PR #1207 _`] + +Infrastructure / Support +---------------------- +* Speed up status page by choosing for a faster query (only latest belief needed) [see `PR #1142 `_] +* For MacOS developers, install HiGHS solver automatically [see `PR #1187 `_] +* Add support for installing FlexMeasures under Python 3.12 [see `PR #1233 `_] + +Bugfixes +----------- +* The UI footer now stays at the bottom even on pages with little content [see `PR #1204 `_] +* Correct stroke dash (based on source type) for forecasts made by forecasters included in FlexMeasures [see `PR #1211 `_] +* Show the correct UTC offset for the data's time span as shown under sensor stats in the UI [see `PR #1213 `_] +* Fixed issue where audit log buttons are visible to users without the necessary permissions. [see `PR #1228 `_] +* Fix issue with displaying ``deactivate user`` and ``reset password`` buttons for non admin users [see `PR #1220 `_] + + +v0.23.0 | September 18, 2024 +============================ + +.. note:: Read more on these features on `the FlexMeasures blog `_. + +.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``). + +New features +------------- +* New chart type on sensor page: histogram [see `PR #1143 `_] +* Add basic sensor info to sensor page [see `PR #1115 `_] +* Add `Statistics` table on the sensor page and also add `api/v3_0/sensors//stats` endpoint to get sensor statistics [see `PR #1116 `_] +* Support adding custom titles to the graphs on the asset page, by extending the ``sensors_to_show`` format [see `PR #1125 `_ and `PR #1177 `_] +* Support zoom-in action on the asset and sensor charts [see `PR #1130 `_] +* Speed up loading the assets page, by making the pagination backend-based and adding support for that in the API, and by enabling to query all accounts one can see in a single call (for admins and consultants) [see `PR #988 `_] +* Added Primary and Secondary colors to account for white-labelled UI themes [see `PR #1137 `_] +* Added Logo URL to account for white-labelled UI themes [see `PR #1145 `_] +* Added PopUp form to edit account details [see `PR #1152 `_] +* When listing past jobs on the `Tasks` page, show the most recent jobs first [see `PR #1163 `_] +* Introduce the ``VariableQuantityField`` to allow three ways of passing a variable quantity in most of the ``flex-model`` and ``flex-context`` fields [see `PR #1127 `_ and `PR #1138 `_] +* Support directly passing a fixed price in the ``flex-context`` using the new fields ``consumption-price`` and ``production-price``, which are meant to replace the ``consumption-price-sensor`` and ``production-price-sensor`` fields, respectively [see `PR #1028 `_] + +Infrastructure / Support +---------------------- +* Save beliefs faster by bulk saving [see `PR #1159 `_] +* Support new single-belief fast track (looking up only one belief) [see `PR #1067 `_] +* Add new annotation types: ``"error"`` and ``"warning"`` [see `PR #1131 `_ and `PR #1150 `_] +* When deleting a sensor, asset or account, delete any annotations that belong to them [see `PR #1151 `_] +* Removed deprecated ``app.schedulers`` and ``app.forecasters`` (use ``app.data_generators["scheduler"]`` and ``app.data_generators["forecaster"]`` instead) [see `PR #1098 `_] +* Save beliefs faster by bulk saving [see `PR #1159 `_] +* Introduced dynamic, JavaScript-generated toast notifications. [see `PR #1152 `_] + +Bugfixes +----------- +* Fix string length exceeding the 255-character limit in the `event` field of `AssetAuditLog` by truncating long updates and logging each field or attribute change individually. [see `PR #1162 `_] +* Fix image carousel on the login page [see `PR #1154 `_] +* Fix styling for User and Documentation menu items [see `PR #1140 `_] +* Fix styling of sensor page, especially the graph chart dropdown [see `PR #1148 `_] +* Fix posting a single instantaneous belief [see `PR #1129 `_] +* Allow reassigning a public asset to private ownership using the ``flexmeasures edit transfer-ownership`` CLI command [see `PR #1123 `_] +* Fix missing value on spring :abbr:`DST (Daylight Saving Time)` transition for ``PandasReporter`` using daily sensor as input [see `PR #1122 `_] +* Fix date range persistence on session across different pages [see `PR #1165 `_] +* Fix issue with account creation failing when the --logo-url flag is omitted. [see related PRs `PR #1167 `_ and `PR #1145 `_] +* Fix ordering of audit logs (asset, account) and job list on status page [see PR `PR #1179 _` and `PR #1183 `_] + + v0.22.0 | June 29, 2024 ============================ +.. note:: Read more on these features on `the FlexMeasures blog `_. + New features ------------- * Add `asset//auditlog` to view asset related actions [see `PR #1067 `_] @@ -25,14 +110,11 @@ Infrastructure / Support * ``flexmeasures show beliefs`` uses the entity path (`/../`) in case of duplicated sensors [see `PR #1026 `_] * Add ``--resolution`` option to ``flexmeasures show chart`` to produce charts in different time resolutions [see `PR #1007 `_] * Add ``FLEXMEASURES_JSON_COMPACT`` config setting and deprecate ``JSONIFY_PRETTYPRINT_REGULAR`` setting [see `PR #1090 `_] -* Removed deprecated ``app.schedulers`` and ``app.forecasters`` (use ``app.data_generators["scheduler"]`` and ``app.data_generators["forecaster"]`` instead) [see `PR #1098 `_] Bugfixes ----------- * Fix ordering of jobs on the asset status page [see `PR #1106 `_] * Relax max staleness for status page using 2 * event_resolution as default instead of immediate staleness [see `PR #1108 `_] -* Fix missing value on spring :abbr:`DST (Daylight Saving Time)` transition for ``PandasReporter`` using daily sensor as input [see `PR #1122 `_] -* Allow reassigning a public asset to private ownership using the ``flexmeasures edit transfer-ownership`` CLI command [see `PR #1123 `_] v0.21.0 | May 16, 2024 diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index 3a1b58fa5..72598db5c 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -4,14 +4,17 @@ FlexMeasures CLI Changelog ********************** -since v.0.22.0 | June 29, 2024 +since v0.24.0 | October XX, 2024 ================================= +* ``flexmeasures show beliefs`` shows datetime values on x-axis labels. -* Add ``--resolution`` option to ``flexmeasures show chart`` to produce charts in different time resolutions. +since v0.22.0 | June 29, 2024 +================================= +* Add ``--resolution`` option to ``flexmeasures show chart`` to produce charts in different time resolutions. -since v.0.21.0 | April 16, 2024 +since v0.21.0 | April 16, 2024 ================================= * Include started, deferred and scheduled jobs in the overview printed by the CLI command ``flexmeasures jobs show-queues``. diff --git a/documentation/dev/setup-and-guidelines.rst b/documentation/dev/setup-and-guidelines.rst index 5dc2358df..8f37a0da9 100644 --- a/documentation/dev/setup-and-guidelines.rst +++ b/documentation/dev/setup-and-guidelines.rst @@ -12,6 +12,7 @@ Furthermore, we discuss several guidelines and best practices. :local: :depth: 1 +| .. note:: Are you implementing code based on FlexMeasures, you're probably interested in :ref:`datamodel`. @@ -149,6 +150,8 @@ Otherwise, you need to add some other user first. Here is how we add an admin: (The `account` you need in the 2nd command is printed by the 1st) +.. include:: ../notes/macOS-port-note.rst + .. note:: If you are on Windows, then running & developing FlexMeasures will not work 100%. For instance, the queueing only works if you install rq-win (https://github.com/michaelbrooks/rq-win) manually and the make tooling is difficult to get to work as well. diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 6d5410b13..a3ed9a934 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -21,19 +21,24 @@ Describing flexibility To compute a schedule, FlexMeasures first needs to assess the flexibility state of the system. This is described by: -- the ``flex-context`` ― information about the system as a whole, in order to assess the value of activating flexibility. -- the ``flex model`` ― information about the state and possible actions of the flexible device. We will discuss these per scheduled device type. +- :ref:`The flex-context ` ― information about the system as a whole, in order to assess the value of activating flexibility. +- :ref:`Flex-models ` ― information about the state and possible actions of the flexible device. We will discuss these per scheduled device type. This information goes beyond the usual time series recorded by an asset's sensors. It's being sent through the API when triggering schedule computation. -Some parts of it can be persisted on the asset & sensor model as attributes (that's design work in progress). +Some parts of it can be persisted on the asset & sensor model as attributes (that's design work in progress). Let's dive into the details ― what can you tell FlexMeasures about your optimization problem? + +.. _flex_context: + The flex-context ----------------- The ``flex-context`` is independent of the type of flexible device that is optimized. -With the flexibility context, we aim to describe the system in which the flexible assets operate: +With the flexibility context, we aim to describe the system in which the flexible assets operate. + +The full list of flex-context fields is as follows: .. list-table:: @@ -45,34 +50,53 @@ With the flexibility context, we aim to describe the system in which the flexibl - Description * - ``inflexible-device-sensors`` - ``[3,4]`` - - Power sensors that are relevant, but not flexible, such as a sensor recording rooftop solar power connected behind the main meter, whose production falls under the same contract as the flexible device(s) being scheduled. - * - ``consumption-price-sensor`` - - ``5`` - - The sensor that defines the price of consuming energy. This sensor can be recording market prices, but also CO₂ - whatever fits your optimization problem. - * - ``production-price-sensor`` - - ``6`` - - The sensor that defines the price of producing energy. + - Power sensors that are relevant, but not flexible, such as a sensor recording rooftop solar power connected behind the main meter, whose production falls under the same contract as the flexible device(s) being scheduled. Their power demand cannot be adjusted but still matters for finding the best schedule for other devices. Must be a list of integers. + * - ``consumption-price`` + - ``{"sensor": 5}`` + or + ``"0.29 EUR/kWh"`` + - The price of consuming energy. Can be (a sensor recording) market prices, but also CO₂ intensity - whatever fits your optimization problem. (This field replaced the ``consumption-price-sensor`` field. [#old_sensor_field]_) + * - ``production-price`` + - ``{"sensor": 6}`` + or + ``"0.12 EUR/kWh"`` + - The price of producing energy. Can be (a sensor recording) market prices, but also CO₂ intensity - whatever fits your optimization problem. (This field replaced the ``production-price-sensor`` field. [#old_sensor_field]_) * - ``site-power-capacity`` - ``"45kW"`` - - Maximum/minimum achievable power at the grid connection point [#asymmetric]_ (defaults to the Asset attribute ``capacity_in_mw``). A constant limit, or see [#sensor_field]_. + - Maximum achievable power at the grid connection point, in either direction [#asymmetric]_ (defaults to the Asset attribute ``capacity_in_mw``). * - ``site-consumption-capacity`` - ``"45kW"`` - - Maximum consumption power at the grid connection point [#consumption]_ (defaults to the Asset attribute ``consumption_capacity_in_mw``). A constant limit, or see [#sensor_field]_. If ``site-power-capacity`` is defined, the minimum between the ``site-power-capacity`` and ``site-consumption-capacity`` will be used. + - Maximum consumption power at the grid connection point [#consumption]_ (defaults to the Asset attribute ``consumption_capacity_in_mw``). If ``site-power-capacity`` is defined, the minimum between the ``site-power-capacity`` and ``site-consumption-capacity`` will be used. * - ``site-production-capacity`` - ``"0kW"`` - - Maximum production power at the grid connection point [#production]_ (defaults to the Asset attribute ``production_capacity_in_mw``). A constant limit, or see [#sensor_field]_. If ``site-power-capacity`` is defined, the minimum between the ``site-power-capacity`` and ``site-production-capacity`` will be used. + - Maximum production power at the grid connection point [#production]_ (defaults to the Asset attribute ``production_capacity_in_mw``). If ``site-power-capacity`` is defined, the minimum between the ``site-power-capacity`` and ``site-production-capacity`` will be used. +.. [#old_sensor_field] The old field only accepted an integer (sensor ID). .. [#asymmetric] ``site-consumption-capacity`` and ``site-production-capacity`` allow defining asymmetric contracted transport capacities for each direction (i.e. production and consumption). -.. [#production] Example: with a connection capacity (``site-power-capacity``) of 1 MVA (apparent power) and a production capacity (``site-production-capacity``) of 400 kW (active power), the scheduler will make sure that the grid inflow doesn't exceed 400 kW. + .. [#consumption] Example: with a connection capacity (``site-power-capacity``) of 1 MVA (apparent power) and a consumption capacity (``site-consumption-capacity``) of 800 kW (active power), the scheduler will make sure that the grid outflow doesn't exceed 800 kW. -.. note:: If no (symmetric, consumption and production) site capacity is defined (also not as defaults), the scheduler will not enforce any bound on the site power. The flexible device can still has its own power limit defined in its flex-model. +.. [#production] Example: with a connection capacity (``site-power-capacity``) of 1 MVA (apparent power) and a production capacity (``site-production-capacity``) of 400 kW (active power), the scheduler will make sure that the grid inflow doesn't exceed 400 kW. + +.. note:: If no (symmetric, consumption and production) site capacity is defined (also not as defaults), the scheduler will not enforce any bound on the site power. + The flexible device can still have its own power limit defined in its flex-model. + +For more details on the possible formats for field values, see :ref:`variable_quantities`. + +.. _flex_models_and_schedulers: The flex-models & corresponding schedulers ------------------------------------------- +FlexMeasures comes with a storage scheduler and a process scheduler, which work with flex models for storages and loads, respectively. + +The storage scheduler is suitable for batteries and :abbr:`EV (electric vehicle)` chargers, and is automatically selected when scheduling an asset with one of the following asset types: ``"battery"``, ``"one-way_evse"`` and ``"two-way_evse"``. + +The process scheduler is suitable for shiftable, breakable and inflexible loads, and is automatically selected for asset types ``"process"`` and ``"load"``. + + Storage ^^^^^^^^ @@ -87,72 +111,78 @@ You can do a lot with this ― examples for storage devices are: - buffers of energy-intensive chemicals that are needed in other industry processes -The ``flex-model`` for storage describes to the scheduler what the flexible asset's state is, +The ``flex-model`` for storage devices describes to the scheduler what the flexible asset's state is, and what constraints or preferences should be taken into account. +The full list of flex-model fields for the storage scheduler is as follows: + .. list-table:: :header-rows: 1 - :widths: 20 25 90 + :widths: 20 40 80 * - Field - Example value - Description * - ``soc-at-start`` - - ``"3.1"`` - - The (estimated) state of charge at the beginning of the schedule (defaults to 0). + - ``"3.1 kWh"`` + - The (estimated) state of charge at the beginning of the schedule (defaults to 0). [#quantity_field]_ * - ``soc-unit`` - ``"kWh"`` or ``"MWh"`` - - The unit in which all SoC related flex-model values are to be interpreted. + - The unit used to interpret any SoC related flex-model value that does not mention a unit itself (only applies to numeric values, so not to string values). + However, we advise to mention the unit in each field explicitly (for instance, ``"3.1 kWh"`` rather than ``3.1``). + Enumerated option only. * - ``soc-min`` - - ``"2.5"`` - - A constant lower boundary for all values in the schedule (defaults to 0). + - ``"2.5 kWh"`` + - A constant lower boundary for all values in the schedule (defaults to 0). [#quantity_field]_ * - ``soc-max`` - - ``"7"`` - - A constant upper boundary for all values in the schedule (defaults to max soc target, if provided) + - ``"7 kWh"`` + - A constant upper boundary for all values in the schedule (defaults to max soc target, if provided). [#quantity_field]_ * - ``soc-minima`` - - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: 8.2}]`` - - Set point(s) that form lower boundaries, e.g. to target a full car battery in the morning. Can be single values or a range (defaults to NaN values). + - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: "8.2 kWh"}]`` + - Set points that form lower boundaries, e.g. to target a full car battery in the morning (defaults to NaN values). * - ``soc-maxima`` - - ``{"value": 51, "start": "2024-02-05T12:00:00+01:00","end": "2024-02-05T13:30:00+01:00"}`` - - Set point(s) that form upper boundaries at certain times. Can be single values or a range (defaults to NaN values). + - ``{"value": "51 kWh", "start": "2024-02-05T12:00:00+01:00", "end": "2024-02-05T13:30:00+01:00"}`` + - Set points that form upper boundaries at certain times (defaults to NaN values). * - ``soc-targets`` - - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: 3.2}]`` + - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: "3.2 kWh"}]`` - Exact set point(s) that the scheduler needs to realize (defaults to NaN values). * - ``soc-gain`` - ``.1kWh`` - - Encode SoC gain per time step. A constant gain every time step, or see [#sensor_field]_. + - SoC gain per time step, e.g. from a secondary energy source (defaults to zero). * - ``soc-usage`` - - ``{"sensor": 23}`` - - Encode SoC reduction per time step. A constant loss every time step, or see [#sensor_field]_. + - ``{"sensor": 23}`` + - SoC reduction per time step, e.g. from a load or heat sink (defaults to zero). * - ``roundtrip-efficiency`` - ``"90%"`` - - Below 100%, this represents roundtrip losses (of charging & discharging), usually used for batteries. Can be percent or ratio ``[0,1]`` (defaults to 100%). + - Below 100%, this represents roundtrip losses (of charging & discharging), usually used for batteries. Can be percent or ratio ``[0,1]`` (defaults to 100%). [#quantity_field]_ * - ``charging-efficiency`` - ``".9"`` - - Apply efficiency losses only at time of charging, not across roundtrip (defaults to 100%). A constant percentage at every step, or see [#sensor_field]_. + - Apply efficiency losses only at time of charging, not across roundtrip (defaults to 100%). * - ``discharging-efficiency`` - ``"90%"`` - - Apply efficiency losses only at time of discharging, not across roundtrip (defaults to 100%). A constant percentage at every step, or see [#sensor_field]_. + - Apply efficiency losses only at time of discharging, not across roundtrip (defaults to 100%). * - ``storage-efficiency`` - ``"99.9%"`` - - This can encode losses over time, so each time step the energy is held longer leads to higher losses (defaults to 100%). A constant percentage at every step, or see [#sensor_field]_. Also read [#storage_efficiency]_ about applying this value per time step across longer time spans. + - This can encode losses over time, so each time step the energy is held longer leads to higher losses (defaults to 100%). Also read [#storage_efficiency]_ about applying this value per time step across longer time spans. * - ``prefer-charging-sooner`` - ``True`` - - Policy to apply if conditions are stable (defaults to True, which also signals a preference to discharge later) + - Tie-breaking policy to apply if conditions are stable (defaults to True, which also signals a preference to discharge later). Boolean option only. * - ``power-capacity`` - ``50kW`` - - Device-level power constraint. How much power can be applied to this asset (defaults to the Sensor attribute ``capacity_in_mw``). A constant limit, or see [#sensor_field]_. + - Device-level power constraint. How much power can be applied to this asset (defaults to the Sensor attribute ``capacity_in_mw``). * - ``consumption-capacity`` - ``{"sensor": 56}`` - - Device-level power constraint on consumption. How much power can be drawn by this asset. A constant limit, or see [#sensor_field]_. + - Device-level power constraint on consumption. How much power can be drawn by this asset. * - ``production-capacity`` - ``0kW`` (only consumption) - - Device-level power constraint on production. How much power can be supplied by this asset. A constant limit, or see [#sensor_field]_. + - Device-level power constraint on production. How much power can be supplied by this asset. -.. [#sensor_field] For some fields, it is possible to supply a sensor instead of one fixed value (``{"sensor": 51}``), which allows for more dynamic contexts, for instance power limits that change over time. +.. [#quantity_field] Can only be set as a fixed quantity. .. [#storage_efficiency] The storage efficiency (e.g. 95% or 0.95) to use for the schedule is applied over each time step equal to the sensor resolution. For example, a storage efficiency of 95 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of :math:`0.95^{1/24} = 0.997865`. +For more details on the possible formats for field values, see :ref:`variable_quantities`. + Usually, not the whole flexibility model is needed. FlexMeasures can infer missing values in the flex model, and even get them (as default) from the sensor's attributes. You can add new storage schedules with the CLI command ``flexmeasures add schedule for-storage``. @@ -177,11 +207,10 @@ Finally, are you interested in the linear programming details behind the storage You can also review the current flex-model for storage in the code, at ``flexmeasures.data.schemas.scheduling.storage.StorageFlexModelSchema``. - Shiftable loads (processes) ^^^^^^^^^^^^^^^^^^^^^^^^^^ -For *processes* that can be shifted or interrupted, but have to happen at a constant rate (of consumption), FlexMeasures provides the ``ShiftableLoad`` scheduler. +For *processes* that can be shifted or interrupted, but have to happen at a constant rate (of consumption), FlexMeasures provides the ``ProcessScheduler``. Some examples from practice (usually industry) could be: - A centrifuge's daily work of combing through sludge water. Depends on amount of sludge present. @@ -208,13 +237,15 @@ Some examples from practice (usually industry) could be: - ``[{"start": "2015-01-02T08:00:00+01:00", "duration": "PT2H"}]`` - Time periods in which the load cannot be scheduled to run. * - ``process_type`` - - ``INFLEXIBLE``, ``BREAKABLE`` or ``SHIFTABLE`` - - Is the load inflexible? Or is there flexibility, to interrupt or shift it? + - ``INFLEXIBLE``, ``SHIFTABLE`` or ``BREAKABLE`` + - Is the load inflexible and should it run as soon as possible? Or can the process's start time be shifted? Or can it even be broken up into smaller segments? You can review the current flex-model for processes in the code, at ``flexmeasures.data.schemas.scheduling.process.ProcessSchedulerFlexModelSchema``. You can add new shiftable-process schedules with the CLI command ``flexmeasures add schedule for-process``. +.. note:: Currently, the ``ProcessScheduler`` uses only the ``consumption-price`` field of the flex-context, so it ignores any site capacities and inflexible devices. + Work on other schedulers -------------------------- @@ -222,6 +253,6 @@ Work on other schedulers We believe the two schedulers (and their flex-models) we describe here are covering a lot of use cases already. Here are some thoughts on further innovation: -- Writing your own scheduler. You can always write your own scheduler(see :ref:`plugin_customization`). You then might want to add your own flex model, as well. FlexMeasures will let the scheduler decide which flexibility model is relevant and how it should be validated. +- Writing your own scheduler. You can always write your own scheduler (see :ref:`plugin_customization`). You then might want to add your own flex model, as well. FlexMeasures will let the scheduler decide which flexibility model is relevant and how it should be validated. - We also aim to model situations with more than one flexible asset, and that have different types of flexibility (e.g. EV charging and smart heating in the same site). This is ongoing architecture design work, and therefore happens in development settings, until we are happy with the outcomes. Thoughts welcome :) - Aggregating flexibility of a group of assets (e.g. a neighborhood) and optimizing its aggregated usage (e.g. for grid congestion support) is also an exciting direction for expansion. diff --git a/documentation/host/docker.rst b/documentation/host/docker.rst index 7ca9f2067..6b2caf9a1 100644 --- a/documentation/host/docker.rst +++ b/documentation/host/docker.rst @@ -37,17 +37,18 @@ Running the image (as a container) might work like this (remember to get the ima .. code-block:: bash - $ docker run --env SQLALCHEMY_DATABASE_URI=postgresql://user:pass@localhost:5432/dbname --env SECRET_KEY=blabla --env FLEXMEASURES_ENV=development -d --net=host lfenergy/flexmeasures + $ docker run --env SQLALCHEMY_DATABASE_URI=postgresql://user:pass@localhost:5432/dbname --env SECRET_KEY=blabla --env FLEXMEASURES_ENV=development -p 5000:5000 -d --net=host lfenergy/flexmeasures .. note:: Don't know what your image is called (its "tag")? We used ``lfenergy/flexmeasures`` here, as that should be the name when pulling it from Docker Hub. You can run ``docker images`` to see which images you have. +.. include:: ../notes/macOS-docker-port-note.rst + The two minimal environment variables to run the container successfully are ``SQLALCHEMY_DATABASE_URI`` and the ``SECRET_KEY``, see :ref:`configuration`. ``FLEXMEASURES_ENV=development`` is needed if you do not have an SSL certificate set up (the default mode is ``production``, and in that mode FlexMeasures requires https for security reasons). If you see too much output, you can also set ``LOGGING_LEVEL=INFO``. In this example, we connect to a postgres database running on our local computer, so we use the host network. In the docker-compose section below, we use a Docker container for the database, as well. Browsing ``http://localhost:5000`` should work now and ask you to log in. - -Of course, you might not have created a user. You can use ``docker exec -it bash`` to go inside the container and use the :ref:`cli` to create everything you need. +Of course, you might not have created a user. You can use ``docker exec -it bash`` to go inside the container and use the :ref:`cli` to create everything you need. .. _docker_configuration: diff --git a/documentation/notes/macOS-docker-port-note.rst b/documentation/notes/macOS-docker-port-note.rst new file mode 100644 index 000000000..31730dc85 --- /dev/null +++ b/documentation/notes/macOS-docker-port-note.rst @@ -0,0 +1,4 @@ +.. note:: For newer versions of macOS, port 5000 is in use by default by Control Center. You can turn this off by going to System Preferences > Sharing and untick the "Airplay Receiver" box. + If you don't want to do this for some reason, you can change the host port in the ``docker run`` command to some other port. + For example, to set it to port 5001, change ``-p 5000:5000`` in the command to ``-p 5001:5000``. + If you do this, remember that you will have to go to http://localhost:5001 in your browser when you want to inspect the FlexMeasures UI. diff --git a/documentation/notes/macOS-port-note.rst b/documentation/notes/macOS-port-note.rst new file mode 100644 index 000000000..9004264a9 --- /dev/null +++ b/documentation/notes/macOS-port-note.rst @@ -0,0 +1,10 @@ +.. note:: For newer versions of macOS, port 5000 is in use by default by Control Center. + You can turn this off by going to System Preferences > Sharing and untick the "Airplay Receiver" box. + If you don't want to do this for some reason, you can change the port for locally running FlexMeasures by setting the ``FLASK_RUN_PORT`` environment variable. + For example, to set it to port 5001: + + .. code-block:: bash + + $ export FLASK_RUN_PORT=5001 # You can also add this to your local .env + + If you do this, remember that you will have to go to http://localhost:5001 in your browser when you want to inspect the FlexMeasures UI. diff --git a/documentation/tut/flex-model-v2g.rst b/documentation/tut/flex-model-v2g.rst index 3d44e703b..46e0d6b99 100644 --- a/documentation/tut/flex-model-v2g.rst +++ b/documentation/tut/flex-model-v2g.rst @@ -33,9 +33,8 @@ Constraining the cycling to occur within a static 25-85% SoC range can be modell { "flex-model": { - "soc-min": 15, - "soc-max": 51, - "soc-unit": "kWh" + "soc-min": "15 kWh", + "soc-max": "51 kWh" } } @@ -50,16 +49,15 @@ To enable a temporary target SoC of more than 85% (for car reservations, see the { "flex-model": { - "soc-min": 15, - "soc-max": 60, + "soc-min": "15 kWh", + "soc-max": "60 kWh", "soc-maxima": [ { - "value": 51, + "value": "51 kWh", "start": "2024-02-04T10:35:00+01:00", "end": "2024-02-05T04:25:00+01:00" } - ], - "soc-unit": "kWh" + ] } } @@ -80,7 +78,7 @@ Given a reservation for 8 AM on February 5th, constraint 2 can be modelled throu "flex-model": { "soc-minima": [ { - "value": 57, + "value": "57 kWh", "datetime": "2024-02-05T08:00:00+01:00" } ] @@ -88,7 +86,7 @@ Given a reservation for 8 AM on February 5th, constraint 2 can be modelled throu } This constraint also signals that if the car is not plugged out of the Charge Point at 8 AM, the scheduler is in principle allowed to start discharging immediately afterwards. -To make sure the car remains at 95% SoC for some time, additional soc-minima constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM: +To make sure the car remains at or above 95% SoC for some time, additional soc-minima constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM: .. code-block:: json @@ -96,7 +94,7 @@ To make sure the car remains at 95% SoC for some time, additional soc-minima con "flex-model": { "soc-minima": [ { - "value": 57, + "value": "57 kWh", "start": "2024-02-05T08:00:00+01:00", "end": "2024-02-05T08:15:00+01:00" } @@ -104,20 +102,42 @@ To make sure the car remains at 95% SoC for some time, additional soc-minima con } } +The car may still charge and discharge within those 15 minutes, but it won't go below 95%. +Alternatively, to keep the car from discharging altogether during that time, limit the ``production-capacity`` (likewise, use the ``consumption-capacity`` to prevent any charging): + +.. code-block:: json + + { + "flex-model": { + "soc-minima": [ + { + "value": "57 kWh", + "datetime": "2024-02-05T08:00:00+01:00" + } + ], + "production-capacity": [ + { + "value": "0 kW", + "start": "2024-02-05T08:00:00+01:00", + "end": "2024-02-05T08:15:00+01:00" + } + ] + } + } .. _earning_by_cycling: Earning by cycling ================== -To provide an incentive for cycling the battery in response to market prices, the ``consumption-price-sensor`` and ``production-price-sensor`` fields of the flex context may be used, which define the sensor IDs under which the price data is stored that is relevant to the given site: +To provide an incentive for cycling the battery in response to market prices, the ``consumption-price`` and ``production-price`` fields of the flex context may be used, which define the sensor IDs under which the price data is stored that is relevant to the given site: .. code-block:: json { "flex-context": { - "consumption-price-sensor": 41, - "production-price-sensor": 42 + "consumption-price": {"sensor": 41}, + "production-price": {"sensor": 42} } } diff --git a/documentation/tut/forecasting_scheduling.rst b/documentation/tut/forecasting_scheduling.rst index 15152e90b..f7a1cfb1f 100644 --- a/documentation/tut/forecasting_scheduling.rst +++ b/documentation/tut/forecasting_scheduling.rst @@ -108,18 +108,17 @@ Here, we extend that (storage) example with an additional target value, represen { "start": "2015-06-02T10:00:00+00:00", "flex-model": { - "soc-at-start": 12.1, - "soc-unit": "kWh" + "soc-at-start": "12.1 kWh", "soc-targets": [ { - "value": 25, + "value": "25 kWh", "datetime": "2015-06-02T16:00:00+00:00" } } } -We now have described the state of charge at 10am to be ``12.1``. In addition, we requested that it should be ``25`` at 4pm. +We now have described the state of charge at 10am to be ``"12.1 kWh"``. In addition, we requested that it should be ``"25 kWh"`` at 4pm. For instance, this could mean that a car should be charged at 90% at that time. If FlexMeasures receives this message, a scheduling job will be made and put into the queue. In turn, the scheduling job creates a proposed schedule. We'll look a bit deeper into those further down in :ref:`getting_schedules`. @@ -149,7 +148,7 @@ Prognoses (the USEF term used for power forecasts) are used by FlexMeasures to d You can access forecasts via the FlexMeasures API at `[GET] /sensors/data <../api/v3_0.html#get--api-v3_0-sensors-data>`_. Getting them might be useful if you want to use prognoses in your own system, or to check their accuracy against meter data, i.e. the realised power measurements. -The FlexMeasures UI also lists forecast accuracy, and visualises prognoses and meter data next to each other. +The FlexMeasures UI also visualizes prognoses and meter data next to each other. A prognosis can be requested at a URL looking like this: diff --git a/documentation/tut/posting_data.rst b/documentation/tut/posting_data.rst index ac60bafea..6eb63b641 100644 --- a/documentation/tut/posting_data.rst +++ b/documentation/tut/posting_data.rst @@ -287,14 +287,12 @@ The endpoint also allows to limit the flexibility range and also to set target v { "start": "2015-06-02T10:00:00+00:00", "flex-model": { - "soc-at-start": 12.1, - "soc-unit": "kWh" + "soc-at-start": "12.1 kWh" } } -.. note:: At the moment, FlexMeasures only supports flexibility models suitable for batteries and car chargers here (asset types "battery", "one-way_evse" or "two-way_evse"). - This will be expanded to other flexible assets as needed. +.. note:: More details on supported flex models can be found in :ref:`flex_models_and_schedulers`. -.. note:: Flexibility states are persisted on sensor attributes. To record a more complete history of the state of charge, set up a separate sensor and post data to it using `[POST] /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`). +.. note:: Flexibility states are persisted on sensor attributes. To record a more complete history of the state of charge, set up a separate sensor and post data to it using `[POST] /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`). In :ref:`how_queue_scheduling`, we'll cover what happens when FlexMeasures is triggered to create a new schedule, and how those schedules can be retrieved via the API, so they can be used to steer assets. \ No newline at end of file diff --git a/documentation/tut/toy-example-setup.rst b/documentation/tut/toy-example-setup.rst index dcedf58c4..7cac2a7e6 100644 --- a/documentation/tut/toy-example-setup.rst +++ b/documentation/tut/toy-example-setup.rst @@ -17,7 +17,7 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s .. code-block:: bash - # setup an account with a user and an energy market (ID 1) + # setup an account with a user, assets for battery & solar and an energy market (ID 1) $ flexmeasures add toy-account # load prices to optimise the schedule against $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam @@ -45,6 +45,9 @@ Install Flexmeasures and the database $ docker pull postgres $ docker network create flexmeasures_network + .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. + ``alias docker='sudo docker'`` enables you to still run this tutorial. + After running these commands, we can start the Postgres database and the FlexMeasures app with the following commands: .. code-block:: bash @@ -52,14 +55,16 @@ Install Flexmeasures and the database $ docker run --rm --name flexmeasures-tutorial-db -e POSTGRES_PASSWORD=fm-db-passwd -e POSTGRES_DB=flexmeasures-db -d --network=flexmeasures_network postgres:latest $ docker run --rm --name flexmeasures-tutorial-fm --env SQLALCHEMY_DATABASE_URI=postgresql://postgres:fm-db-passwd@flexmeasures-tutorial-db:5432/flexmeasures-db --env SECRET_KEY=notsecret --env FLEXMEASURES_ENV=development --env LOGGING_LEVEL=INFO -d --network=flexmeasures_network -p 5000:5000 lfenergy/flexmeasures + When the app has started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. + + .. include:: ../notes/macOS-docker-port-note.rst + To establish the FlexMeasures database structure, execute: .. code-block:: bash $ docker exec flexmeasures-tutorial-fm bash -c "flexmeasures db upgrade" - .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. ``alias docker='sudo docker'`` enables you to still run this tutorial. - Now - what's *very important* to remember is this: The rest of this tutorial will happen *inside* the ``flexmeasures-tutorial-fm`` container! This is how you hop inside the container and run a terminal there: .. code-block:: bash @@ -82,8 +87,6 @@ Install Flexmeasures and the database $ docker start flexmeasures-tutorial-db $ docker start flexmeasures-tutorial-fm - .. note:: For newer versions of MacOS, port 5000 is in use by default by Control Center. You can turn this off by going to System Preferences > Sharing and untick the "Airplay Receiver" box. If you don't want to do this for some reason, you can change the host port in the ``docker run`` command to some other port, for example 5001. To do this, change ``-p 5000:5000`` in the command to ``-p 5001:5000``. If you do this, remember that you will have to go to ``localhost:5001`` in your browser when you want to inspect the FlexMeasures UI. - .. note:: Got docker-compose? You could run this tutorial with 5 containers :) ― Go to :ref:`docker-compose-tutorial`. .. tab:: On your PC @@ -116,6 +119,17 @@ Install Flexmeasures and the database $ make clean-db db_name=flexmeasures-db [db_user=flexmeasures] + To start the web application, you can run: + + .. code-block:: bash + + $ flexmeasures run + + When started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. + + .. include:: ../notes/macOS-port-note.rst + + Add some structural data --------------------------------------- diff --git a/documentation/views/asset-data.rst b/documentation/views/asset-data.rst index 6a92279a7..43dd86185 100644 --- a/documentation/views/asset-data.rst +++ b/documentation/views/asset-data.rst @@ -7,10 +7,9 @@ Assets & sensor data Asset page ------------ -The asset page allows to see data from the asset's sensors, and also to edit attributes of the asset, like its location. -Other attributes are stored as a JSON string, which can be edited here as well. -This is meant for meta information that may be used to customize views or functionality, e.g. by plugins. -This includes the possibility to specify which sensors the asset page should show. For instance, here we include a price sensor from a public asset, by setting ``{"sensor_to_show": [3, 2]}`` (sensor 3 on top, followed by sensor 2 below). +The asset page allows to plot data from the asset's sensors, show sensors and child assets and also to edit attributes of the asset, like its location. + +For instance, in the picture below we include a price sensor from a public asset, then plot the asset's only sensor below that. .. image:: https://github.com/FlexMeasures/screenshots/raw/main/screenshot_asset.png @@ -20,6 +19,24 @@ This includes the possibility to specify which sensors the asset page should sho | | + +The asset page as data dashboard +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The data charts are maybe the most interesting feature - turning it into a data dashboard. When the most interesting sensors are shown, the replay button on the right creates a very meaningful dynamic insight! + +With the attribute `sensors_to_show` one can specify from which sensors the asset page should show data. In the example above, this happened by setting ``{"sensor_to_show": [3, 2]}`` (sensor 3 on top, followed by sensor 2 below). + +It is also possible to overlay data for multiple sensors within one plot, by setting the `sensors_to_show` attribute to a nested list. For example, ``{"sensor_to_show": [3, [2, 4]]}`` would show the data for sensor 4 laid over the data for sensor 2. +While it is possible to show an arbitrary number of sensors this way, we recommend showing only the most crucial ones for faster loading, less page scrolling, and generally, a quick grasp of what the asset is up to. + +Finally, it is possible to set custom titles for sensor graphs, by setting within `sensors_to_show` a dictionary with a title and sensor or sensors. For example, ``{"title": "Outdoor Temperature", "sensor": 1}`` or ``{"title": "Energy Demand", "sensors": [2, 3]}`` will display the specified title for the corresponding sensor data. + + + +Status page +^^^^^^^^^^^^ + For each asset, you can also visit a status page to see if your data connectivity and recent jobs are okay. This is how data connectivity status looks like on the building asset from our tutorial: .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/screenshot_building_status.png @@ -35,13 +52,6 @@ This is how the audit log looks for the history of actions taken on an asset: :align: center .. :scale: 40% -| -| - -.. note:: It is possible to overlay data for multiple sensors, by setting the `sensors_to_show` attribute to a nested list. For example, ``{"sensor_to_show": [3, [2, 4]]}`` would show the data for sensor 4 laid over the data for sensor 2. -.. note:: While it is possible to show an arbitrary number of sensors this way, we recommend showing only the most crucial ones for faster loading, less page scrolling, and generally, a quick grasp of what the asset is up to. -.. note:: Asset attributes can be edited through the CLI as well, with the CLI command ``flexmeasures edit attribute``. - Sensor page ------------- diff --git a/flexmeasures/api/common/schemas/generic_assets.py b/flexmeasures/api/common/schemas/generic_assets.py index dfda954c5..cdd5fe81b 100644 --- a/flexmeasures/api/common/schemas/generic_assets.py +++ b/flexmeasures/api/common/schemas/generic_assets.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from flask import abort from marshmallow import fields from sqlalchemy import select diff --git a/flexmeasures/api/common/schemas/search.py b/flexmeasures/api/common/schemas/search.py new file mode 100644 index 000000000..fa24b0e01 --- /dev/null +++ b/flexmeasures/api/common/schemas/search.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from shlex import join, split + +from marshmallow import fields, ValidationError + + +class SearchFilterField(fields.Str): + """Field that represents a search filter.""" + + def _deserialize(self, value, attr, data, **kwargs) -> list[str]: + try: + search_terms = split(value) + except ValueError as e: + raise ValidationError(str(e)) + return search_terms + + def _serialize(self, value: list[str], attr, obj, **kwargs) -> str: + return join(value) diff --git a/flexmeasures/api/common/schemas/sensor_data.py b/flexmeasures/api/common/schemas/sensor_data.py index a6c341488..0c31de267 100644 --- a/flexmeasures/api/common/schemas/sensor_data.py +++ b/flexmeasures/api/common/schemas/sensor_data.py @@ -5,7 +5,7 @@ from flask_login import current_user from isodate import datetime_isoformat from marshmallow import fields, post_load, validates_schema, ValidationError -from marshmallow.validate import OneOf +from marshmallow.validate import OneOf, Length from marshmallow_polyfield import PolyField from timely_beliefs import BeliefsDataFrame import pandas as pd @@ -63,7 +63,7 @@ def select_schema_to_ensure_list_of_floats( This ensures that we are not requiring the same flexibility from users who are retrieving data. """ if isinstance(values, list): - return fields.List(fields.Float(allow_none=True)) + return fields.List(fields.Float(allow_none=True), validate=Length(min=1)) else: return SingleValueField() @@ -268,6 +268,7 @@ def check_resolution_compatibility_of_sensor_data(self, data, **kwargs): Currently, only upsampling is supported (e.g. converting hourly events to 15-minute events). """ required_resolution = data["sensor"].event_resolution + if required_resolution == timedelta(hours=0): # For instantaneous sensors, any event frequency is compatible return @@ -280,6 +281,19 @@ def check_resolution_compatibility_of_sensor_data(self, data, **kwargs): f"Resolution of {inferred_resolution} is incompatible with the sensor's required resolution of {required_resolution}." ) + @validates_schema + def check_multiple_instantenous_values(self, data, **kwargs): + """Ensure that we are not getting multiple instantaneous values that overlap. + That is, two values spanning the same moment (a zero duration). + """ + + if len(data["values"]) > 1 and data["duration"] / len( + data["values"] + ) == timedelta(0): + raise ValidationError( + "Cannot save multiple instantaneous values that overlap. That is, two values spanning the same moment (a zero duration). Try sending a single value or definining a non-zero duration." + ) + @post_load() def post_load_sequence(self, data: dict, **kwargs) -> BeliefsDataFrame: """If needed, upsample and convert units, then deserialize to a BeliefsDataFrame.""" @@ -344,11 +358,18 @@ def load_bdf(sensor_data: dict) -> BeliefsDataFrame: source = get_or_create_source(current_user) num_values = len(sensor_data["values"]) event_resolution = sensor_data["duration"] / num_values - dt_index = pd.date_range( - sensor_data["start"], - periods=num_values, - freq=event_resolution, - ) + + if event_resolution == timedelta(hours=0): + dt_index = pd.date_range( + sensor_data["start"], + periods=num_values, + ) + else: + dt_index = pd.date_range( + sensor_data["start"], + periods=num_values, + freq=event_resolution, + ) s = pd.Series(sensor_data["values"], index=dt_index) # Work out what the recording time should be diff --git a/flexmeasures/api/common/schemas/sensors.py b/flexmeasures/api/common/schemas/sensors.py index dccc23c4d..30c1e75d2 100644 --- a/flexmeasures/api/common/schemas/sensors.py +++ b/flexmeasures/api/common/schemas/sensors.py @@ -1,5 +1,5 @@ from flask import abort -from marshmallow import fields +from marshmallow import fields, ValidationError from sqlalchemy import select from flexmeasures.data import db @@ -9,6 +9,7 @@ EntityAddressException, ) from flexmeasures.data.models.time_series import Sensor +from flexmeasures.utils.unit_utils import is_valid_unit class EntityAddressValidationError(FMValidationError): @@ -80,3 +81,15 @@ def _serialize(self, value: Sensor, attr, data, **kwargs): return value.entity_address_fm0 else: return value.entity_address + + +class UnitField(fields.Str): + """Field that represents a unit.""" + + def _deserialize(self, value, attr, data, **kwargs) -> str: + if not is_valid_unit(value): + raise ValidationError(f"Invalid unit: {value}") + return value + + def _serialize(self, value: str, attr, obj, **kwargs) -> str: + return value diff --git a/flexmeasures/api/common/schemas/users.py b/flexmeasures/api/common/schemas/users.py index db50b9d1d..20c4c8a3a 100644 --- a/flexmeasures/api/common/schemas/users.py +++ b/flexmeasures/api/common/schemas/users.py @@ -38,8 +38,8 @@ class UserIdField(fields.Integer): """ def __init__(self, *args, **kwargs): - kwargs["load_default"] = ( - lambda: current_user if not current_user.is_anonymous else None + kwargs["load_default"] = lambda: ( + current_user if not current_user.is_anonymous else None ) super().__init__(*args, **kwargs) diff --git a/flexmeasures/api/v3_0/accounts.py b/flexmeasures/api/v3_0/accounts.py index 341ecf667..06406c888 100644 --- a/flexmeasures/api/v3_0/accounts.py +++ b/flexmeasures/api/v3_0/accounts.py @@ -1,15 +1,27 @@ +from __future__ import annotations + from flask_classful import FlaskView, route -from webargs.flaskparser import use_kwargs +from flexmeasures.data import db +from webargs.flaskparser import use_kwargs, use_args from flask_security import current_user, auth_required from flask_json import as_json +from sqlalchemy import or_, select, func + +from marshmallow import fields +import marshmallow.validate as validate +from flask_sqlalchemy.pagination import SelectPagination + from flexmeasures.auth.policy import user_has_admin_access from flexmeasures.auth.decorators import permission_required_for_context from flexmeasures.data.models.audit_log import AuditLog -from flexmeasures.data.models.user import Account +from flexmeasures.data.models.user import Account, User +from flexmeasures.data.models.generic_assets import GenericAsset from flexmeasures.data.services.accounts import get_accounts, get_audit_log_records from flexmeasures.api.common.schemas.users import AccountIdField from flexmeasures.data.schemas.account import AccountSchema +from flexmeasures.api.common.schemas.search import SearchFilterField +from flexmeasures.utils.time_utils import server_now """ API endpoints to manage accounts. @@ -30,8 +42,25 @@ class AccountAPI(FlaskView): decorators = [auth_required()] @route("", methods=["GET"]) + @use_kwargs( + { + "page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=None + ), + "per_page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=10 + ), + "filter": SearchFilterField(required=False, load_default=None), + }, + location="query", + ) @as_json - def index(self): + def index( + self, + page: int | None = None, + per_page: int | None = None, + filter: list[str] | None = None, + ): """API endpoint to list all accounts accessible to the current user. .. :quickref: Account; Download account list @@ -39,20 +68,37 @@ def index(self): This endpoint returns all accessible accounts. Accessible accounts are your own account and accounts you are a consultant for, or all accounts for admins. + The endpoint supports pagination of the asset list using the `page` and `per_page` query parameters. + + - If the `page` parameter is not provided, all assets are returned, without pagination information. The result will be a list of assets. + - If a `page` parameter is provided, the response will be paginated, showing a specific number of assets per page as defined by `per_page` (default is 10). + - If a search 'filter' such as 'solar "ACME corp"' is provided, the response will filter out assets where each search term is either present in their name or account name. + The response schema for pagination is inspired by https://datatables.net/manual/server-side#Returned-data + **Example response** An example of one account being returned: .. sourcecode:: json - [ + { + "data" : [ { 'id': 1, 'name': 'Test Account' 'account_roles': [1, 3], 'consultancy_account_id': 2, + 'primary_color': '#1a3443' + 'secondary_color': '#f1a122' + 'logo_url': 'https://example.com/logo.png' } - ] + ], + "num-records" : 1, + "filtered-records" : 1 + + } + + If no pagination is requested, the response only consists of the list under the "data" key. :reqheader Authorization: The authentication token :reqheader Content-Type: application/json @@ -72,7 +118,48 @@ def index(self): else [] ) - return accounts_schema.dump(accounts), 200 + query = db.session.query(Account).filter( + Account.id.in_([a.id for a in accounts]) + ) + + if filter: + search_terms = filter[0].split(" ") + query = query.filter( + or_(*[Account.name.ilike(f"%{term}%") for term in search_terms]) + ) + + if page: + select_pagination: SelectPagination = db.paginate( + query, per_page=per_page, page=page + ) + + accounts_reponse: list = [] + for account in select_pagination.items: + user_count_query = select(func.count(User.id)).where( + User.account_id == account.id + ) + asset_count_query = select(func.count(GenericAsset.id)).where( + GenericAsset.account_id == account.id + ) + user_count = db.session.execute(user_count_query).scalar() + asset_count = db.session.execute(asset_count_query).scalar() + accounts_reponse.append( + { + **account_schema.dump(account), + "user_count": user_count, + "asset_count": asset_count, + } + ) + + response = { + "data": accounts_reponse, + "num-records": select_pagination.total, + "filtered-records": select_pagination.total, + } + else: + response = accounts_schema.dump(query.all(), many=True) + + return response, 200 @route("/", methods=["GET"]) @use_kwargs({"account": AccountIdField(data_key="id")}, location="path") @@ -109,6 +196,120 @@ def get(self, id: int, account: Account): return account_schema.dump(account), 200 + @route("/", methods=["PATCH"]) + @use_args(partial_account_schema) + @use_kwargs({"account": AccountIdField(data_key="id")}, location="path") + @permission_required_for_context("update", ctx_arg_name="account") + @as_json + def patch(self, account_data: dict, id: int, account: Account): + """Update an account given its identifier. + + .. :quickref: Account; Update an account + + This endpoint sets data for an existing account. + + The following fields are not allowed to be updated: + - id + + The following fields are only editable if user role is admin: + - consultancy_account_id + + **Example request** + + .. sourcecode:: json + + { + 'name': 'Test Account' + 'primary_color': '#1a3443' + 'secondary_color': '#f1a122' + 'logo_url': 'https://example.com/logo.png' + 'consultancy_account_id': 2, + } + + + **Example response** + + The whole account is returned in the response: + + .. sourcecode:: json + + { + 'id': 1, + 'name': 'Test Account' + 'account_roles': [1, 3], + 'primary_color': '#1a3443' + 'secondary_color': '#f1a122' + 'logo_url': 'https://example.com/logo.png' + 'consultancy_account_id': 2, + } + + :reqheader Authorization: The authentication token + :reqheader Content-Type: application/json + :resheader Content-Type: application/json + :status 200: UPDATED + :status 400: INVALID_REQUEST, REQUIRED_INFO_MISSING, UNEXPECTED_PARAMS + :status 401: UNAUTHORIZED + :status 403: INVALID_SENDER + :status 422: UNPROCESSABLE_ENTITY + """ + + # Get existing consultancy_account_id + existing_consultancy_account_id = ( + account.consultancy_account.id if account.consultancy_account else None + ) + + if not user_has_admin_access(current_user, "update"): + # Remove consultancy_account_id from account_data if no admin access + account_data.pop("consultancy_account_id", None) + else: + # Check if consultancy_account_id has changed + new_consultancy_account_id = account_data.get("consultancy_account_id") + if existing_consultancy_account_id != new_consultancy_account_id: + new_consultant_account = db.session.query(Account).get( + new_consultancy_account_id + ) + # Validate new consultant account + if ( + not new_consultant_account + or new_consultant_account.id == account.id + ): + return {"errors": ["Invalid consultancy_account_id"]}, 422 + + # Track modified fields + fields_to_check = [ + "name", + "primary_color", + "secondary_color", + "logo_url", + "consultancy_account_id", + ] + modified_fields = { + field: getattr(account, field) + for field in fields_to_check + if account_data.get(field) != getattr(account, field) + } + + # Compile modified fields string + modified_fields_str = ", ".join(modified_fields.keys()) + + for k, v in account_data.items(): + setattr(account, k, v) + + event_message = f"Account Updated, Field: {modified_fields_str}" + + # Add Audit log + account_audit_log = AuditLog( + event_datetime=server_now(), + event=event_message, + active_user_id=current_user.id, + active_user_name=current_user.username, + affected_user_id=current_user.id, + affected_account_id=account.id, + ) + db.session.add(account_audit_log) + db.session.commit() + return account_schema.dump(account), 200 + @route("//auditlog", methods=["GET"]) @use_kwargs({"account": AccountIdField(data_key="id")}, location="path") @permission_required_for_context( diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py index 89cf3afb6..6e973827f 100644 --- a/flexmeasures/api/v3_0/assets.py +++ b/flexmeasures/api/v3_0/assets.py @@ -1,35 +1,61 @@ +from __future__ import annotations import json +from humanize import naturaldelta from flask import current_app from flask_classful import FlaskView, route +from flask_login import current_user from flask_security import auth_required from flask_json import as_json +from flask_sqlalchemy.pagination import SelectPagination + from marshmallow import fields +import marshmallow.validate as validate + from webargs.flaskparser import use_kwargs, use_args -from sqlalchemy import select, delete +from sqlalchemy import select, delete, func from flexmeasures.auth.decorators import permission_required_for_context from flexmeasures.data import db from flexmeasures.data.models.user import Account from flexmeasures.data.models.audit_log import AssetAuditLog from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.data.queries.generic_assets import query_assets_by_search_terms from flexmeasures.data.schemas import AwareDateTimeField from flexmeasures.data.schemas.generic_assets import GenericAssetSchema as AssetSchema from flexmeasures.api.common.schemas.generic_assets import AssetIdField +from flexmeasures.api.common.schemas.search import SearchFilterField from flexmeasures.api.common.schemas.users import AccountIdField from flexmeasures.utils.coding_utils import flatten_unique from flexmeasures.ui.utils.view_utils import set_session_variables +from flexmeasures.auth.policy import check_access +from werkzeug.exceptions import Forbidden, Unauthorized +from flexmeasures.data.schemas.sensors import SensorSchema +from flexmeasures.data.models.time_series import Sensor asset_schema = AssetSchema() assets_schema = AssetSchema(many=True) +sensor_schema = SensorSchema() +sensors_schema = SensorSchema(many=True) partial_asset_schema = AssetSchema(partial=True, exclude=["account_id"]) +def get_accessible_accounts() -> list[Account]: + accounts = [] + for _account in db.session.scalars(select(Account)).all(): + try: + check_access(_account, "read") + accounts.append(_account) + except (Forbidden, Unauthorized): + pass + + return accounts + + class AssetAPI(FlaskView): """ This API view exposes generic assets. - Under development until it replaces the original Asset API. """ route_base = "/assets" @@ -39,38 +65,188 @@ class AssetAPI(FlaskView): @route("", methods=["GET"]) @use_kwargs( { - "account": AccountIdField( - data_key="account_id", load_default=AccountIdField.load_current + "account": AccountIdField(data_key="account_id", load_default=None), + "all_accessible": fields.Bool( + data_key="all_accessible", load_default=False + ), + "include_public": fields.Bool( + data_key="include_public", load_default=False ), + "page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=None + ), + "per_page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=10 + ), + "filter": SearchFilterField(required=False, load_default=None), }, location="query", ) - @permission_required_for_context("read", ctx_arg_name="account") @as_json - def index(self, account: Account): - """List all assets owned by a certain account. + def index( + self, + account: Account | None, + all_accessible: bool, + include_public: bool, + page: int | None = None, + per_page: int | None = None, + filter: list[str] | None = None, + ): + """List all assets owned by user's accounts, or a certain account or all accessible accounts. .. :quickref: Asset; Download asset list - This endpoint returns all accessible assets for the account of the user. - The `account_id` query parameter can be used to list assets from a different account. + This endpoint returns all accessible assets by accounts. + The `account_id` query parameter can be used to list assets from any account (if the user is allowed to read them). Per default, the user's account is used. + Alternatively, the `all_accessible` query parameter can be used to list assets from all accounts the current_user has read-access to, plus all public assets. Defaults to `false`. + The `include_public` query parameter can be used to include public assets in the response. Defaults to `false`. + + The endpoint supports pagination of the asset list using the `page` and `per_page` query parameters. + + - If the `page` parameter is not provided, all assets are returned, without pagination information. The result will be a list of assets. + - If a `page` parameter is provided, the response will be paginated, showing a specific number of assets per page as defined by `per_page` (default is 10). + - If a search 'filter' such as 'solar "ACME corp"' is provided, the response will filter out assets where each search term is either present in their name or account name. + The response schema for pagination is inspired by https://datatables.net/manual/server-side#Returned-data + **Example response** - An example of one asset being returned: + An example of one asset being returned in a paginated response: .. sourcecode:: json - [ - { - "id": 1, - "name": "Test battery", - "latitude": 10, - "longitude": 100, - "account_id": 2, - "generic_asset_type_id": 1 - } - ] + { + "data" : [ + { + "id": 1, + "name": "Test battery", + "latitude": 10, + "longitude": 100, + "account_id": 2, + "generic_asset_type": {"id": 1, "name": "battery"} + } + ], + "num-records" : 1, + "filtered-records" : 1 + + } + + If no pagination is requested, the response only consists of the list under the "data" key. + + :reqheader Authorization: The authentication token + :reqheader Content-Type: application/json + :resheader Content-Type: application/json + :status 200: PROCESSED + :status 400: INVALID_REQUEST + :status 401: UNAUTHORIZED + :status 403: INVALID_SENDER + :status 422: UNPROCESSABLE_ENTITY + """ + + # find out which accounts are relevant + if all_accessible: + accounts = get_accessible_accounts() + else: + if account is None: + account = current_user.account + check_access(account, "read") + accounts = [account] + + filter_statement = GenericAsset.account_id.in_([a.id for a in accounts]) + + # add public assets if the request asks for all the accessible assets + if all_accessible or include_public: + filter_statement = filter_statement | GenericAsset.account_id.is_(None) + + query = query_assets_by_search_terms( + search_terms=filter, filter_statement=filter_statement + ) + if page is None: + response = asset_schema.dump(db.session.scalars(query).all(), many=True) + else: + if per_page is None: + per_page = 10 + + select_pagination: SelectPagination = db.paginate( + query, per_page=per_page, page=page + ) + num_records = db.session.scalar( + select(func.count(GenericAsset.id)).filter(filter_statement) + ) + response = { + "data": asset_schema.dump(select_pagination.items, many=True), + "num-records": num_records, + "filtered-records": select_pagination.total, + } + + return response, 200 + + @route( + "//sensors", + methods=["GET"], + ) + @use_kwargs( + { + "asset": AssetIdField(data_key="id"), + }, + location="path", + ) + @use_kwargs( + { + "page": fields.Int( + required=False, validate=validate.Range(min=1), dump_default=1 + ), + "per_page": fields.Int( + required=False, validate=validate.Range(min=1), dump_default=10 + ), + }, + location="query", + ) + @as_json + def asset_sensors( + self, + id: int, + asset: GenericAsset | None, + page: int | None = None, + per_page: int | None = None, + ): + """ + List all sensors under an asset. + + .. :quickref: Asset; Return all sensors under an asset. + + This endpoint returns all sensors under an asset. + + The endpoint supports pagination of the asset list using the `page` and `per_page` query parameters. + + - If the `page` parameter is not provided, all sensors are returned, without pagination information. The result will be a list of sensors. + - If a `page` parameter is provided, the response will be paginated, showing a specific number of assets per page as defined by `per_page` (default is 10). + The response schema for pagination is inspired by https://datatables.net/manual/server-side#Returned-data + + + **Example response** + + An example of one asset being returned in a paginated response: + + .. sourcecode:: json + + { + "data" : [ + { + "id": 1, + "name": "Test battery", + "latitude": 10, + "longitude": 100, + "account_id": 2, + "generic_asset_type": {"id": 1, "name": "battery"} + } + ], + "num-records" : 1, + "filtered-records" : 1 + + } + + If no pagination is requested, the response only consists of the list under the "data" key. :reqheader Authorization: The authentication token :reqheader Content-Type: application/json @@ -81,7 +257,33 @@ def index(self, account: Account): :status 403: INVALID_SENDER :status 422: UNPROCESSABLE_ENTITY """ - return assets_schema.dump(account.generic_assets), 200 + query_statement = Sensor.generic_asset_id == asset.id + + query = select(Sensor).filter(query_statement) + + select_pagination: SelectPagination = db.paginate( + query, per_page=per_page, page=page + ) + + num_records = db.session.scalar( + select(func.count(Sensor.id)).where(query_statement) + ) + + sensors_response: list = [ + { + **sensor_schema.dump(sensor), + "event_resolution": naturaldelta(sensor.event_resolution), + } + for sensor in select_pagination.items + ] + + response = { + "data": sensors_response, + "num-records": num_records, + "filtered-records": select_pagination.total, + } + + return response, 200 @route("/public", methods=["GET"]) @as_json @@ -252,14 +454,16 @@ def patch(self, asset_data: dict, id: int, db_asset: GenericAsset): for attr_key, attr_value in v.items(): if current_attributes.get(attr_key) != attr_value: audit_log_data.append( - f"Attribute name: {attr_key}, Old value: {current_attributes.get(attr_key)}, New value: {attr_value}" + f"Updated Attr: {attr_key}, From: {current_attributes.get(attr_key)}, To: {attr_value}" ) continue audit_log_data.append( - f"Field name: {k}, Old value: {getattr(db_asset, k)}, New value: {v}" + f"Updated Field: {k}, From: {getattr(db_asset, k)}, To: {v}" ) - audit_log_event = f"Updated asset '{db_asset.name}': {db_asset.id} fields: {'; '.join(audit_log_data)}" - AssetAuditLog.add_record(db_asset, audit_log_event) + + # Iterate over each field or attribute updates and create a separate audit log entry for each. + for event in audit_log_data: + AssetAuditLog.add_record(db_asset, event) for k, v in asset_data.items(): setattr(db_asset, k, v) @@ -307,6 +511,7 @@ def delete(self, id: int, asset: GenericAsset): "beliefs_after": AwareDateTimeField(format="iso", required=False), "beliefs_before": AwareDateTimeField(format="iso", required=False), "include_data": fields.Boolean(required=False), + "combine_legend": fields.Boolean(required=False, load_default=True), "dataset_name": fields.Str(required=False), "height": fields.Str(required=False), "width": fields.Str(required=False), diff --git a/flexmeasures/api/v3_0/sensors.py b/flexmeasures/api/v3_0/sensors.py index 63b912c20..cba456b38 100644 --- a/flexmeasures/api/v3_0/sensors.py +++ b/flexmeasures/api/v3_0/sensors.py @@ -1,17 +1,19 @@ from __future__ import annotations +import isodate from datetime import datetime, timedelta +from werkzeug.exceptions import Unauthorized from flask import current_app, url_for from flask_classful import FlaskView, route from flask_json import as_json -from flask_security import auth_required -import isodate +from flask_security import auth_required, current_user from marshmallow import fields, ValidationError +import marshmallow.validate as validate from rq.job import Job, NoSuchJobError from timely_beliefs import BeliefsDataFrame from webargs.flaskparser import use_args, use_kwargs -from sqlalchemy import delete +from sqlalchemy import delete, select, or_ from flexmeasures.api.common.responses import ( request_processed, @@ -29,6 +31,7 @@ ) from flexmeasures.api.common.schemas.users import AccountIdField from flexmeasures.api.common.utils.api_utils import save_and_enqueue +from flexmeasures.auth.policy import check_access from flexmeasures.auth.decorators import permission_required_for_context from flexmeasures.data import db from flexmeasures.data.models.audit_log import AssetAuditLog @@ -37,8 +40,11 @@ from flexmeasures.data.models.time_series import Sensor, TimedBelief from flexmeasures.data.queries.utils import simplify_index from flexmeasures.data.schemas.sensors import SensorSchema, SensorIdField +from flexmeasures.data.schemas import AssetIdField +from flexmeasures.api.common.schemas.search import SearchFilterField +from flexmeasures.api.common.schemas.sensors import UnitField from flexmeasures.data.schemas.times import AwareDateTimeField, PlanningDurationField -from flexmeasures.data.services.sensors import get_sensors +from flexmeasures.data.services.sensors import get_sensor_stats from flexmeasures.data.services.scheduling import ( create_scheduling_job, get_data_source_for_job, @@ -62,30 +68,70 @@ class SensorAPI(FlaskView): @route("", methods=["GET"]) @use_kwargs( { - "account": AccountIdField( - data_key="account_id", load_default=AccountIdField.load_current + "account": AccountIdField(data_key="account_id", required=False), + "asset": AssetIdField(data_key="asset_id", required=False), + "include_consultancy_clients": fields.Boolean( + required=False, load_default=False + ), + "include_public_assets": fields.Boolean(required=False, load_default=False), + "page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=None ), + "per_page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=10 + ), + "filter": SearchFilterField(required=False, load_default=None), + "unit": UnitField(required=False, load_default=None), }, location="query", ) - @permission_required_for_context("read", ctx_arg_name="account") @as_json - def index(self, account: Account): + def index( + self, + account: Account | None = None, + asset: GenericAsset | None = None, + include_consultancy_clients: bool = False, + include_public_assets: bool = False, + page: int | None = None, + per_page: int | None = None, + filter: list[str] | None = None, + unit: str | None = None, + ): """API endpoint to list all sensors of an account. .. :quickref: Sensor; Download sensor list This endpoint returns all accessible sensors. - Accessible sensors are sensors in the same account as the current user. + By default, "accessible sensors" means all sensors in the same account as the current user (if they have read permission to the account). + + You can also specify an `account` (an ID parameter), if the user has read access to that account. In this case, all assets under the + specified account will be retrieved, and the sensors associated with these assets will be returned. + + Alternatively, you can filter by asset hierarchy by providing the `asset` parameter (ID). When this is set, all sensors on the specified + asset and its sub-assets are retrieved, provided the user has read access to the asset. + + NOTE: You can't set both account and asset at the same time, you can only have one set. The only exception is if the asset being specified is + part of the account that was set, then we allow to see sensors under that asset but then ignore the account (account = None). + + Finally, you can use the `include_consultancy_clients` parameter to include sensors from accounts for which the current user account is a consultant. + This is only possible if the user has the role of a consultant. + Only admins can use this endpoint to fetch sensors from a different account (by using the `account_id` query parameter). + The `filter` parameter allows you to search for sensors by name or account name. + The `unit` parameter allows you to filter by unit. + + For the pagination of the sensor list, you can use the `page` and `per_page` query parameters, the `page` parameter is used to trigger + pagination, and the `per_page` parameter is used to specify the number of records per page. The default value for `page` is 1 and for `per_page` is 10. + **Example response** An example of one sensor being returned: .. sourcecode:: json - [ + { + "data" : [ { "entity_address": "ea1.2021-01.io.flexmeasures.company:fm1.42", "event_resolution": PT15M, @@ -95,7 +141,12 @@ def index(self, account: Account): "unit": "m\u00b3/h" "id": 2 } - ] + ], + "num-records" : 1, + "filtered-records" : 1 + } + + If no pagination is requested, the response only consists of the list under the "data" key. :reqheader Authorization: The authentication token :reqheader Content-Type: application/json @@ -106,8 +157,109 @@ def index(self, account: Account): :status 403: INVALID_SENDER :status 422: UNPROCESSABLE_ENTITY """ - sensors = get_sensors(account=account) - return sensors_schema.dump(sensors), 200 + if account is None and asset is None: + if current_user.is_anonymous: + raise Unauthorized + account = current_user.account + + if account is not None and asset is not None: + if asset.account_id != account.id: + return { + "message": "Please provide either an account or an asset ID, not both" + }, 422 + else: + account = None + + if asset is not None: + check_access(asset, "read") + + asset_tree = ( + db.session.query(GenericAsset.id, GenericAsset.parent_asset_id) + .filter(GenericAsset.id == asset.id) + .cte(name="asset_tree", recursive=True) + ) + + recursive_part = db.session.query( + GenericAsset.id, GenericAsset.parent_asset_id + ).join(asset_tree, GenericAsset.parent_asset_id == asset_tree.c.id) + + asset_tree = asset_tree.union(recursive_part) + + child_assets = db.session.query(asset_tree).all() + + filter_statement = GenericAsset.id.in_( + [asset.id] + [a.id for a in child_assets] + ) + elif account is not None: + check_access(account, "read") + + account_ids: list = [account.id] + + if include_consultancy_clients: + if current_user.has_role("consultant"): + consultancy_accounts = ( + db.session.query(Account) + .filter(Account.consultancy_account_id == account.id) + .all() + ) + account_ids.extend([acc.id for acc in consultancy_accounts]) + + filter_statement = GenericAsset.account_id.in_(account_ids) + else: + filter_statement = None + + if include_public_assets: + filter_statement = or_( + filter_statement, + GenericAsset.account_id.is_(None), + ) + + sensor_query = ( + select(Sensor) + .join(GenericAsset, Sensor.generic_asset_id == GenericAsset.id) + .outerjoin(Account, GenericAsset.owner) + .filter(filter_statement) + ) + + if filter is not None: + sensor_query = sensor_query.filter( + or_( + *( + or_( + Sensor.name.ilike(f"%{term}%"), + Account.name.ilike(f"%{term}%"), + GenericAsset.name.ilike(f"%{term}%"), + ) + for term in filter + ) + ) + ) + + if unit: + sensor_query = sensor_query.filter(Sensor.unit == unit) + + sensors = ( + db.session.scalars(sensor_query).all() + if page is None + else db.paginate(sensor_query, per_page=per_page, page=page).items + ) + + sensors = [sensor for sensor in sensors if check_access(sensor, "read") is None] + + sensors_response = sensors_schema.dump(sensors) + + # Return appropriate response for paginated or non-paginated data + if page is None: + return sensors_response, 200 + else: + num_records = len(db.session.execute(sensor_query).scalars().all()) + select_pagination = db.paginate(sensor_query, per_page=per_page, page=page) + response = { + "data": sensors_response, + "num-records": num_records, + "filtered-records": select_pagination.total, + } + return response, 200 @route("/data", methods=["POST"]) @use_args( @@ -279,8 +431,7 @@ def trigger_schedule( { "start": "2015-06-02T10:00:00+00:00", "flex-model": { - "soc-at-start": 12.1, - "soc-unit": "kWh" + "soc-at-start": "12.1 kWh" } } @@ -311,17 +462,16 @@ def trigger_schedule( "start": "2015-06-02T10:00:00+00:00", "duration": "PT24H", "flex-model": { - "soc-at-start": 12.1, - "soc-unit": "kWh", + "soc-at-start": "12.1 kWh", "soc-targets": [ { - "value": 25, + "value": "25 kWh", "datetime": "2015-06-02T16:00:00+00:00" }, ], "soc-minima": {"sensor" : 300}, - "soc-min": 10, - "soc-max": 25, + "soc-min": "10 kWh", + "soc-max": "25 kWh", "charging-efficiency": "120%", "discharging-efficiency": {"sensor": 98}, "storage-efficiency": 0.9999, @@ -330,8 +480,8 @@ def trigger_schedule( "production-capacity" : "30 kW" }, "flex-context": { - "consumption-price-sensor": 9, - "production-price-sensor": 10, + "consumption-price": {"sensor": 9}, + "production-price": {"sensor": 10}, "inflexible-device-sensors": [13, 14, 15], "site-power-capacity": "100kW", "site-production-capacity": "80kW", @@ -366,7 +516,7 @@ def trigger_schedule( """ end_of_schedule = start_of_schedule + duration scheduler_kwargs = dict( - sensor=sensor, + asset_or_sensor=sensor, start=start_of_schedule, end=end_of_schedule, resolution=sensor.event_resolution, @@ -769,3 +919,42 @@ def delete(self, id: int, sensor: Sensor): db.session.commit() current_app.logger.info("Deleted sensor '%s'." % sensor_name) return {}, 204 + + @route("//stats", methods=["GET"]) + @use_kwargs({"sensor": SensorIdField(data_key="id")}, location="path") + @permission_required_for_context("read", ctx_arg_name="sensor") + @as_json + def get_stats(self, id, sensor): + """Fetch stats for a given sensor. + + .. :quickref: Sensor; Get sensor stats + + This endpoint fetches sensor stats for all the historical data. + + Example response + + .. sourcecode:: json + + { + "some data source": { + "min_event_start": "2015-06-02T10:00:00+00:00", + "max_event_start": "2015-10-02T10:00:00+00:00", + "min_value": 0.0, + "max_value": 100.0, + "mean_value": 50.0, + "sum_values": 500.0, + "count_values": 10 + } + } + + :reqheader Authorization: The authentication token + :reqheader Content-Type: application/json + :resheader Content-Type: application/json + :status 200: PROCESSED + :status 400: INVALID_REQUEST, REQUIRED_INFO_MISSING, UNEXPECTED_PARAMS + :status 401: UNAUTHORIZED + :status 403: INVALID_SENDER + :status 422: UNPROCESSABLE_ENTITY + """ + + return get_sensor_stats(sensor), 200 diff --git a/flexmeasures/api/v3_0/tests/conftest.py b/flexmeasures/api/v3_0/tests/conftest.py index 74a70383f..a8c534e1b 100644 --- a/flexmeasures/api/v3_0/tests/conftest.py +++ b/flexmeasures/api/v3_0/tests/conftest.py @@ -2,12 +2,14 @@ from datetime import timedelta +import numpy as np import pandas as pd import pytest from flask_security import SQLAlchemySessionUserDatastore, hash_password from sqlalchemy import select, delete from flexmeasures import Sensor, Source, User, UserRole +from flexmeasures.data.models.data_sources import DataSource from flexmeasures.data.models.generic_assets import GenericAssetType, GenericAsset from flexmeasures.data.models.time_series import TimedBelief @@ -150,6 +152,11 @@ def add_incineration_line(db, test_supplier_user) -> dict[str, Sensor]: ) db.session.add(gas_sensor) add_gas_measurements(db, test_supplier_user.data_source[0], gas_sensor) + other_source = DataSource(name="Other source", type="demo script") + db.session.add(other_source) + db.session.flush() + add_gas_measurements(db, other_source, gas_sensor, values=[91.3, np.nan, 92.1]) + temperature_sensor = Sensor( name="some temperature sensor", unit="°C", @@ -161,16 +168,28 @@ def add_incineration_line(db, test_supplier_user) -> dict[str, Sensor]: db, test_supplier_user.data_source[0], temperature_sensor ) + empty_temperature_sensor = Sensor( + name="empty temperature sensor", + unit="°C", + event_resolution=timedelta(0), + generic_asset=incineration_asset, + ) + db.session.add(empty_temperature_sensor) + db.session.flush() # assign sensor ids - return {gas_sensor.name: gas_sensor, temperature_sensor.name: temperature_sensor} + return { + gas_sensor.name: gas_sensor, + temperature_sensor.name: temperature_sensor, + empty_temperature_sensor.name: empty_temperature_sensor, + } -def add_gas_measurements(db, source: Source, sensor: Sensor): +def add_gas_measurements(db, source: Source, sensor: Sensor, values=None): event_starts = [ pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes) for minutes in range(0, 30, 10) ] - event_values = [91.3, 91.7, 92.1] + event_values = list(values) if values else [91.3, 91.7, 92.1] beliefs = [ TimedBelief( sensor=sensor, diff --git a/flexmeasures/api/v3_0/tests/test_assets_api.py b/flexmeasures/api/v3_0/tests/test_assets_api.py index c6292a81c..d44716036 100644 --- a/flexmeasures/api/v3_0/tests/test_assets_api.py +++ b/flexmeasures/api/v3_0/tests/test_assets_api.py @@ -9,6 +9,7 @@ from flexmeasures.data.services.users import find_user_by_email from flexmeasures.api.tests.utils import get_auth_token, UserContext, AccountContext from flexmeasures.api.v3_0.tests.utils import get_asset_post_data +from flexmeasures.utils.unit_utils import is_valid_unit @pytest.mark.parametrize( @@ -66,11 +67,12 @@ def test_get_asset_nonaccount_access(client, setup_api_test_data, requesting_use @pytest.mark.parametrize( - "requesting_user, account_name, num_assets", + "requesting_user, account_name, num_assets, use_pagination", [ - ("test_admin_user@seita.nl", "Prosumer", 1), - ("test_admin_user@seita.nl", "Supplier", 2), - ("test_consultant@seita.nl", "ConsultancyClient", 1), + ("test_admin_user@seita.nl", "Prosumer", 1, False), + ("test_admin_user@seita.nl", "Supplier", 2, False), + ("test_consultant@seita.nl", "ConsultancyClient", 1, False), + ("test_admin_user@seita.nl", "Prosumer", 1, True), ], indirect=["requesting_user"], ) @@ -80,13 +82,17 @@ def test_get_assets( setup_accounts, account_name, num_assets, + use_pagination, requesting_user, ): """ Get assets per account. Our user here is admin, so is allowed to see all assets. + Pagination is tested only in passing, we should test filtering and page > 1 """ query = {"account_id": setup_accounts[account_name].id} + if use_pagination: + query["page"] = 1 get_assets_response = client.get( url_for("AssetAPI:index"), @@ -94,17 +100,47 @@ def test_get_assets( ) print("Server responded with:\n%s" % get_assets_response.json) assert get_assets_response.status_code == 200 - assert len(get_assets_response.json) == num_assets + + if use_pagination: + assets = get_assets_response.json["data"] + assert get_assets_response.json["num-records"] == num_assets + assert get_assets_response.json["filtered-records"] == num_assets + else: + assets = get_assets_response.json + + assert len(assets) == num_assets if account_name == "Supplier": # one deep dive turbine = {} - for asset in get_assets_response.json: + for asset in assets: if asset["name"] == "Test wind turbine": turbine = asset assert turbine assert turbine["account_id"] == setup_accounts["Supplier"].id +@pytest.mark.parametrize("requesting_user", ["test_admin_user@seita.nl"], indirect=True) +def test_fetch_asset_sensors(client, setup_api_test_data, requesting_user): + """ + Retrieve all sensors associated with a specific asset. + + This test checks for these metadata fields and the number of sensors returned, as well as + confirming that the response is a list of dictionaries, each containing a valid unit. + """ + asset_id = setup_api_test_data["some gas sensor"].generic_asset_id + response = client.get(url_for("AssetAPI:asset_sensors", id=asset_id)) + + print("Server responded with:\n%s" % response.json) + + assert response.status_code == 200 + assert response.json["status"] == 200 + assert isinstance(response.json["data"], list) + assert isinstance(response.json["data"][0], dict) + assert is_valid_unit(response.json["data"][0]["unit"]) + assert response.json["num-records"] == 3 + assert response.json["filtered-records"] == 3 + + @pytest.mark.parametrize("requesting_user", ["test_admin_user@seita.nl"], indirect=True) def test_get_asset_with_children(client, add_asset_with_children, requesting_user): """ @@ -190,7 +226,17 @@ def test_alter_an_asset( print(f"Editing Response: {asset_edit_response.json}") assert asset_edit_response.status_code == 200 - audit_log_event = f"Updated asset '{prosumer_asset.name}': {prosumer_asset.id} fields: Field name: name, Old value: {name}, New value: other; Field name: latitude, Old value: {latitude}, New value: 11.1" + audit_log_event = f"Updated Field: name, From: {name}, To: other" + assert db.session.execute( + select(AssetAuditLog).filter_by( + event=audit_log_event, + active_user_id=requesting_user.id, + active_user_name=requesting_user.username, + affected_asset_id=prosumer_asset.id, + ) + ).scalar_one_or_none() + + audit_log_event = f"Updated Field: latitude, From: {latitude}, To: 11.1" assert db.session.execute( select(AssetAuditLog).filter_by( event=audit_log_event, @@ -211,11 +257,11 @@ def test_alter_an_asset( ('{"sensors_to_show": [1, [0, 2]]}', "No sensor found"), # no sensor with ID 0 ( '{"sensors_to_show": [1, [2, [3, 4]]]}', - "should only contain", + "All elements in a list within 'sensors_to_show' must be integers.", ), # nesting level max 1 ( '{"sensors_to_show": [1, "2"]}', - "should only contain", + "Invalid item type in 'sensors_to_show'. Expected int, list, or dict.", ), # non-integer sensor ID ], ) diff --git a/flexmeasures/api/v3_0/tests/test_sensor_data.py b/flexmeasures/api/v3_0/tests/test_sensor_data.py index 577c43980..2a245cc12 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_data.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_data.py @@ -3,7 +3,8 @@ from datetime import timedelta from flask import url_for import pytest - +from sqlalchemy import event +from sqlalchemy.engine import Engine from flexmeasures import Sensor, Source, User from flexmeasures.api.v3_0.tests.utils import make_sensor_data_request_for_gas_sensor @@ -184,9 +185,19 @@ def test_post_invalid_sensor_data( @pytest.mark.parametrize( "requesting_user", ["test_supplier_user_4@seita.nl"], indirect=True ) -def test_post_sensor_data_twice(client, setup_api_test_data, requesting_user): +def test_post_sensor_data_twice(client, setup_api_test_data, requesting_user, db): post_data = make_sensor_data_request_for_gas_sensor() + @event.listens_for(Engine, "handle_error") + def receive_handle_error(exception_context): + """ + Check that the error that we are getting is of type IntegrityError. + """ + error_info = exception_context.sqlalchemy_exception + + # If the assert failed, we would get a 500 status code + assert error_info.__class__.__name__ == "IntegrityError" + # Check that 1st time posting the data succeeds response = client.post( url_for("SensorAPI:post_data"), @@ -213,3 +224,54 @@ def test_post_sensor_data_twice(client, setup_api_test_data, requesting_user): print(response.json) assert response.status_code == 403 assert "data represents a replacement" in response.json["message"] + + # at this point, the transaction has failed and needs to be rolled back. + db.session.rollback() + + +@pytest.mark.parametrize( + "num_values, status_code, message, saved_rows", + [ + (1, 200, "Request has been processed.", 1), + ( + 2, + 422, + "Cannot save multiple instantaneous values that overlap. That is, two values spanning the same moment (a zero duration). Try sending a single value or definining a non-zero duration.", + 0, + ), + ], +) +@pytest.mark.parametrize( + "requesting_user", ["test_supplier_user_4@seita.nl"], indirect=True +) +def test_post_sensor_instantaneous_data( + client, + setup_api_test_data, + num_values, + status_code, + message, + saved_rows, + requesting_user, +): + post_data = make_sensor_data_request_for_gas_sensor( + sensor_name="empty temperature sensor", + num_values=num_values, + unit="°C", + duration="PT0H", + ) + sensor = setup_api_test_data["empty temperature sensor"] + rows = len(sensor.search_beliefs()) + + # Check that 1st time posting the data succeeds + response = client.post( + url_for("SensorAPI:post_data"), + json=post_data, + ) + + assert response.status_code == status_code + if status_code == 422: + assert response.json["message"]["json"]["_schema"][0] == message + else: + assert response.json["message"] == message + + assert len(sensor.search_beliefs()) - rows == saved_rows diff --git a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py index be7b67760..1ccb94210 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py @@ -46,7 +46,7 @@ def test_get_schedule_wrong_job_id( message_for_trigger_schedule(), "soc-min", "not-a-float", - "Not a valid number", + "Cannot convert value 'not-a-float' to a valid quantity.", ), (message_for_trigger_schedule(), "soc-unit", "MWH", "Must be one of"), # todo: add back test in case we stop grandfathering ignoring too-far-into-the-future targets, or amend otherwise diff --git a/flexmeasures/api/v3_0/tests/test_sensors_api.py b/flexmeasures/api/v3_0/tests/test_sensors_api.py index 0dcb75480..2163ed12f 100644 --- a/flexmeasures/api/v3_0/tests/test_sensors_api.py +++ b/flexmeasures/api/v3_0/tests/test_sensors_api.py @@ -1,6 +1,7 @@ from __future__ import annotations import pytest +import math from flask import url_for from sqlalchemy import select, func @@ -11,10 +12,206 @@ from flexmeasures.data.models.audit_log import AssetAuditLog from flexmeasures.data.schemas.sensors import SensorSchema from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.tests.utils import QueryCounter +from flexmeasures.utils.unit_utils import is_valid_unit + sensor_schema = SensorSchema() +@pytest.mark.parametrize( + "requesting_user, search_by, search_value, exp_sensor_name, exp_num_results, include_consultancy_clients, use_pagination, expected_status_code, filter_account_id, filter_asset_id, asset_id_of_of_first_sensor_result", + [ + ( + "test_supplier_user_4@seita.nl", + "unit", + "°C", + "some temperature sensor", + 2, + True, + False, + 200, + None, + 5, + None, + ), + ( + "test_prosumer_user@seita.nl", + None, + None, + "power", + 2, + False, + False, + 200, + None, + 7, + 8, # We test that the endpoint returns the sensor on a battery asset (ID: 8) while we filter for the building asset (ID: 7) that includes it + ), + ( + "test_supplier_user_4@seita.nl", + "unit", + "m³/h", + "some gas sensor", + 1, + True, + False, + 200, + None, + 5, + None, + ), + ( + "test_supplier_user_4@seita.nl", + None, + None, + None, + None, + None, + None, + 422, # Error expected due to both asset_id and account_id being provided + 1, + 5, + None, + ), + ( + "test_dummy_account_admin@seita.nl", + None, + None, + None, + None, + None, + None, + 403, # Error expected as the user lacks access to the specified asset + None, + 5, + None, + ), + ( + "test_supplier_user_4@seita.nl", + None, + None, + None, + None, + None, + None, + 403, # Error expected as the user lacks access to the specified account + 1, + None, + None, + ), + ( + "test_supplier_user_4@seita.nl", + None, + None, + "some temperature sensor", + 3, + True, + True, + 200, + None, + 5, + None, + ), + ( + "test_supplier_user_4@seita.nl", + "filter", + "'some temperature sensor'", + "some temperature sensor", + 1, + False, + False, + 200, + None, + 5, + None, + ), + ], + indirect=["requesting_user"], +) +def test_fetch_sensors( + client, + setup_api_test_data, + add_battery_assets, + requesting_user, + search_by, + search_value, + exp_sensor_name, + exp_num_results, + include_consultancy_clients, + use_pagination, + expected_status_code, + filter_account_id, + filter_asset_id, + asset_id_of_of_first_sensor_result, +): + """ + Retrieve all sensors. + + Our user here is admin, so is allowed to see all sensors. + Pagination is tested only in passing, we should test filtering and page > 1 + + The `filter_asset_id` specifies the asset_id to filter for. + + The `asset_id_of_of_first_sensor_result` specifies the asset_id of the first sensor + in the result list. This sensors is expected to be from a child asset of the asset + specified in `filter_asset_id`. + + The `filter_account_id` specifies the account_id to filter for. + + `check_errors` is used to test the error handling of the endpoint. + """ + query = {search_by: search_value} + + if use_pagination: + query["page"] = 1 + + if search_by == "unit": + query["unit"] = search_value + elif search_by == "filter": + query["filter"] = search_value + + if include_consultancy_clients: + query["include_consultancy_clients"] = True + + if filter_account_id: + query["account_id"] = filter_account_id + + if filter_asset_id: + query["asset_id"] = filter_asset_id + + response = client.get( + url_for("SensorAPI:index"), + query_string=query, + ) + + print("Server responded with:\n%s" % response.json) + + assert response.status_code == expected_status_code + if expected_status_code == 200: + if use_pagination: + assert isinstance(response.json["data"][0], dict) + assert is_valid_unit(response.json["data"][0]["unit"]) + assert response.json["num-records"] == exp_num_results + assert response.json["filtered-records"] == exp_num_results + else: + assert isinstance(response.json, list) + assert is_valid_unit(response.json[0]["unit"]) + assert response.json[0]["name"] == exp_sensor_name + assert len(response.json) == exp_num_results + + if asset_id_of_of_first_sensor_result is not None: + assert ( + response.json[0]["generic_asset_id"] + == asset_id_of_of_first_sensor_result + ) + elif filter_asset_id: + assert response.json[0]["generic_asset_id"] == filter_asset_id + + if search_by == "unit": + assert response.json[0]["unit"] == search_value + + @pytest.mark.parametrize( "requesting_user", ["test_supplier_user_4@seita.nl"], indirect=True ) @@ -77,10 +274,12 @@ def test_post_a_sensor(client, setup_api_test_data, requesting_user, db): assert response.status_code == 201 assert response.json["name"] == "power" assert response.json["event_resolution"] == "PT1H" + assert response.json["generic_asset_id"] == post_data["generic_asset_id"] sensor: Sensor = db.session.execute( - select(Sensor).filter_by(name="power") + select(Sensor).filter_by(name="power", unit="kWh") ).scalar_one_or_none() + assert sensor is not None assert sensor.unit == "kWh" assert sensor.attributes["capacity_in_mw"] == 0.0074 @@ -240,3 +439,57 @@ def test_delete_a_sensor(client, setup_api_test_data, requesting_user, db): active_user_name=requesting_user.username, ) ).scalar_one_or_none() + + +@pytest.mark.parametrize( + "requesting_user", ["test_supplier_user_4@seita.nl"], indirect=True +) +def test_fetch_sensor_stats( + client, setup_api_test_data: dict[str, Sensor], requesting_user, db +): + # gas sensor is set up in add_gas_measurements + sensor_id = 1 + with QueryCounter(db.session.connection()) as counter1: + response = client.get( + url_for("SensorAPI:get_stats", id=sensor_id), + ) + print("Server responded with:\n%s" % response.json) + assert response.status_code == 200 + response_content = response.json + + del response_content["status"] + assert sorted(list(response_content.keys())) == [ + "Other source", + "Test Supplier User", + ] + for source, record in response_content.items(): + assert record["First event start"] == "2021-05-01T22:00:00+00:00" + assert record["Last event end"] == "2021-05-01T22:30:00+00:00" + assert record["Min value"] == 91.3 + assert record["Max value"] == 92.1 + if source == "Test Supplier User": + # values are: 91.3, 91.7, 92.1 + sum_values = 275.1 + count_values = 3 + else: + # values are: 91.3, NaN, 92.1 + sum_values = 183.4 + count_values = 3 + mean_value = 91.7 + assert math.isclose( + record["Mean value"], mean_value, rel_tol=1e-5 + ), f"mean_value is close to {mean_value}" + assert math.isclose( + record["Sum over values"], sum_values, rel_tol=1e-5 + ), f"sum_values is close to {sum_values}" + assert record["Number of values"] == count_values + + with QueryCounter(db.session.connection()) as counter2: + response = client.get( + url_for("SensorAPI:get_stats", id=sensor_id), + ) + print("Server responded with:\n%s" % response.json) + assert response.status_code == 200 + + # Check stats cache works and stats query is executed only once + assert counter1.count == counter2.count + 1 diff --git a/flexmeasures/api/v3_0/tests/utils.py b/flexmeasures/api/v3_0/tests/utils.py index 96c4095ab..1971dc132 100644 --- a/flexmeasures/api/v3_0/tests/utils.py +++ b/flexmeasures/api/v3_0/tests/utils.py @@ -9,12 +9,13 @@ def make_sensor_data_request_for_gas_sensor( duration: str = "PT1H", unit: str = "m³", include_a_null: bool = False, + sensor_name: str = "some gas sensor", ) -> dict: """Creates request to post sensor data for a gas sensor. This particular gas sensor measures units of m³/h with a 10-minute resolution. """ sensor = db.session.execute( - select(Sensor).filter_by(name="some gas sensor") + select(Sensor).filter_by(name=sensor_name) ).scalar_one_or_none() values = num_values * [-11.28] if include_a_null: @@ -68,9 +69,9 @@ def message_for_trigger_schedule( "duration": "PT24H", # Will be extended in case of targets that would otherwise lie beyond the schedule's end } if unknown_prices: - message[ - "start" - ] = "2040-01-01T00:00:00+01:00" # We have no beliefs in our test database about 2040 prices + message["start"] = ( + "2040-01-01T00:00:00+01:00" # We have no beliefs in our test database about 2040 prices + ) message["flex-model"] = { "soc-at-start": 12.1, # in kWh, according to soc-unit diff --git a/flexmeasures/api/v3_0/users.py b/flexmeasures/api/v3_0/users.py index e37f920d7..8015f45ff 100644 --- a/flexmeasures/api/v3_0/users.py +++ b/flexmeasures/api/v3_0/users.py @@ -1,25 +1,33 @@ +from __future__ import annotations from flask_classful import FlaskView, route from marshmallow import fields +import marshmallow.validate as validate from sqlalchemy.exc import IntegrityError +from sqlalchemy import and_, select, func +from flask_sqlalchemy.pagination import SelectPagination from webargs.flaskparser import use_kwargs from flask_security import current_user, auth_required from flask_security.recoverable import send_reset_password_instructions from flask_json import as_json from werkzeug.exceptions import Forbidden +from flexmeasures.auth.policy import check_access from flexmeasures.data.models.audit_log import AuditLog from flexmeasures.data.models.user import User as UserModel, Account from flexmeasures.api.common.schemas.users import AccountIdField, UserIdField +from flexmeasures.api.common.schemas.search import SearchFilterField +from flexmeasures.api.v3_0.assets import get_accessible_accounts +from flexmeasures.data.queries.users import query_users_by_search_terms +from flexmeasures.data.schemas.account import AccountSchema from flexmeasures.data.schemas.users import UserSchema from flexmeasures.data.services.users import ( - get_users, set_random_password, remove_cookie_and_token_access, get_audit_log_records, ) from flexmeasures.auth.decorators import permission_required_for_context from flexmeasures.data import db -from flexmeasures.utils.time_utils import server_now +from flexmeasures.utils.time_utils import server_now, naturalized_datetime_str """ API endpoints to manage users. @@ -31,6 +39,7 @@ user_schema = UserSchema() users_schema = UserSchema(many=True) partial_user_schema = UserSchema(partial=True) +account_schema = AccountSchema() class UserAPI(FlaskView): @@ -41,22 +50,36 @@ class UserAPI(FlaskView): @route("", methods=["GET"]) @use_kwargs( { - "account": AccountIdField( - data_key="account_id", load_default=AccountIdField.load_current - ), + "account": AccountIdField(data_key="account_id", load_default=None), "include_inactive": fields.Bool(load_default=False), + "page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=None + ), + "per_page": fields.Int( + required=False, validate=validate.Range(min=1), load_default=1 + ), + "filter": SearchFilterField(required=False, load_default=None), }, location="query", ) - @permission_required_for_context("read", ctx_arg_name="account") @as_json - def index(self, account: Account, include_inactive: bool = False): - """API endpoint to list all users of an account. + def index( + self, + account: Account, + include_inactive: bool = False, + page: int | None = None, + per_page: int | None = None, + filter: str | None = None, + ): + """ + API endpoint to list all users. .. :quickref: User; Download user list This endpoint returns all accessible users. By default, only active users are returned. + The `account_id` query parameter can be used to filter the users of + a given account. The `include_inactive` query parameter can be used to also fetch inactive users. Accessible users are users in the same account as the current user. @@ -89,8 +112,63 @@ def index(self, account: Account, include_inactive: bool = False): :status 403: INVALID_SENDER :status 422: UNPROCESSABLE_ENTITY """ - users = get_users(account_name=account.name, only_active=not include_inactive) - return users_schema.dump(users), 200 + if account is not None: + check_access(account, "read") + accounts = [account] + else: + accounts = get_accessible_accounts() + + filter_statement = UserModel.account_id.in_([a.id for a in accounts]) + + if include_inactive is False: + filter_statement = and_(filter_statement, UserModel.active.is_(True)) + + query = query_users_by_search_terms( + search_terms=filter, filter_statement=filter_statement + ) + + if page is not None: + num_records = db.session.scalar( + select(func.count(UserModel.id)).where(filter_statement) + ) + paginated_users: SelectPagination = db.paginate( + query, per_page=per_page, page=page + ) + + users_response: list = [ + { + **user_schema.dump(user), + "account": account_schema.dump(user.account), + "flexmeasures_roles": [ + role.name for role in user.flexmeasures_roles + ], + "last_login_at": naturalized_datetime_str(user.last_login_at), + "last_seen_at": naturalized_datetime_str(user.last_seen_at), + } + for user in paginated_users.items + ] + response: dict | list = { + "data": users_response, + "num-records": num_records, + "filtered-records": paginated_users.total, + } + else: + users = db.session.execute(query).scalars().all() + + response = [ + { + **user_schema.dump(user), + "account": account_schema.dump(user.account), + "flexmeasures_roles": [ + role.name for role in user.flexmeasures_roles + ], + "last_login_at": naturalized_datetime_str(user.last_login_at), + "last_seen_at": naturalized_datetime_str(user.last_seen_at), + } + for user in users + ] + + return response, 200 @route("/") @use_kwargs({"user": UserIdField(data_key="id")}, location="path") diff --git a/flexmeasures/auth/decorators.py b/flexmeasures/auth/decorators.py index 2655bb42a..76e51b23d 100644 --- a/flexmeasures/auth/decorators.py +++ b/flexmeasures/auth/decorators.py @@ -2,7 +2,6 @@ Auth decorators for endpoints """ - from __future__ import annotations from typing import Callable diff --git a/flexmeasures/auth/error_handling.py b/flexmeasures/auth/error_handling.py index 9d504f815..9b67337bb 100644 --- a/flexmeasures/auth/error_handling.py +++ b/flexmeasures/auth/error_handling.py @@ -5,6 +5,7 @@ Names of Responses have to be kept as they were called in original W3 protocols. See explanation below. """ + from __future__ import annotations from typing import Callable diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py index 7f1ac3e3c..cb1f7046b 100755 --- a/flexmeasures/cli/data_add.py +++ b/flexmeasures/cli/data_add.py @@ -62,7 +62,7 @@ LongitudeField, SensorIdField, TimeIntervalField, - QuantityOrSensor, + VariableQuantityField, ) from flexmeasures.data.schemas.sources import DataSourceIdField from flexmeasures.data.schemas.times import TimeIntervalSchema @@ -83,6 +83,7 @@ from flexmeasures.utils import flexmeasures_inflection from flexmeasures.utils.time_utils import server_now, apply_offset_chain from flexmeasures.utils.unit_utils import convert_units, ur +from flexmeasures.cli.utils import validate_color_cli, validate_url_cli from flexmeasures.data.utils import save_to_db from flexmeasures.data.services.utils import get_asset_or_sensor_ref from flexmeasures.data.models.reporting import Reporter @@ -147,7 +148,10 @@ def new_account_role(name: str, description: str): if role is not None: click.secho(f"Account role '{name}' already exists.", **MsgStyle.ERROR) raise click.Abort() - role = AccountRole(name=name, description=description) + role = AccountRole( + name=name, + description=description, + ) db.session.add(role) db.session.commit() click.secho( @@ -160,13 +164,35 @@ def new_account_role(name: str, description: str): @with_appcontext @click.option("--name", required=True) @click.option("--roles", help="e.g. anonymous,Prosumer,CPO") +@click.option( + "--primary-color", + callback=validate_color_cli, + help="Primary color to use in UI, in hex format. Defaults to FlexMeasures' primary color (#1a3443)", +) +@click.option( + "--secondary-color", + callback=validate_color_cli, + help="Secondary color to use in UI, in hex format. Defaults to FlexMeasures' secondary color (#f1a122)", +) +@click.option( + "--logo-url", + callback=validate_url_cli, + help="Logo URL to use in UI. Defaults to FlexMeasures' logo URL", +) @click.option( "--consultancy", "consultancy_account", type=AccountIdField(required=False), help="ID of the consultancy account, whose consultants will have read access to this account", ) -def new_account(name: str, roles: str, consultancy_account: Account | None): +def new_account( + name: str, + roles: str, + consultancy_account: Account | None, + primary_color: str | None, + secondary_color: str | None, + logo_url: str | None, +): """ Create an account for a tenant in the FlexMeasures platform. """ @@ -176,7 +202,36 @@ def new_account(name: str, roles: str, consultancy_account: Account | None): if account is not None: click.secho(f"Account '{name}' already exists.", **MsgStyle.ERROR) raise click.Abort() - account = Account(name=name, consultancy_account=consultancy_account) + + # make sure both colors or none are given + if (primary_color and not secondary_color) or ( + not primary_color and secondary_color + ): + click.secho( + "Please provide both primary_color and secondary_color, or leave both fields blank.", + **MsgStyle.ERROR, + ) + raise click.Abort() + + # Add '#' if color is given and doesn't already start with it + primary_color = ( + f"#{primary_color}" + if primary_color and not primary_color.startswith("#") + else primary_color + ) + secondary_color = ( + f"#{secondary_color}" + if secondary_color and not secondary_color.startswith("#") + else secondary_color + ) + + account = Account( + name=name, + consultancy_account=consultancy_account, + primary_color=primary_color, + secondary_color=secondary_color, + logo_url=logo_url, + ) db.session.add(account) if roles: for role_name in roles.split(","): @@ -687,9 +742,9 @@ def add_beliefs( header=None, skiprows=skiprows, nrows=nrows, - usecols=[datecol, valuecol] - if beliefcol is None - else [datecol, beliefcol, valuecol], + usecols=( + [datecol, valuecol] if beliefcol is None else [datecol, beliefcol, valuecol] + ), parse_dates=True, na_values=na_values, keep_default_na=keep_default_na, @@ -1123,7 +1178,7 @@ def create_schedule(ctx): @click.option( "--site-power-capacity", "site_power_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Site consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1133,7 +1188,7 @@ def create_schedule(ctx): @click.option( "--site-consumption-capacity", "site_consumption_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Site consumption power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1143,7 +1198,7 @@ def create_schedule(ctx): @click.option( "--site-production-capacity", "site_production_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Site production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1208,7 +1263,7 @@ def create_schedule(ctx): @click.option( "--charging-efficiency", "charging_efficiency", - type=QuantityOrSensor("%"), + type=VariableQuantityField("%"), required=False, default=None, help="Storage charging efficiency to use for the schedule." @@ -1218,7 +1273,7 @@ def create_schedule(ctx): @click.option( "--discharging-efficiency", "discharging_efficiency", - type=QuantityOrSensor("%"), + type=VariableQuantityField("%"), required=False, default=None, help="Storage discharging efficiency to use for the schedule." @@ -1228,7 +1283,7 @@ def create_schedule(ctx): @click.option( "--soc-gain", "soc_gain", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Specify the State of Charge (SoC) gain as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1238,7 +1293,7 @@ def create_schedule(ctx): @click.option( "--soc-usage", "soc_usage", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Specify the State of Charge (SoC) usage as a quantity in power units (e.g. 1 MW or 1000 kW) " @@ -1248,7 +1303,7 @@ def create_schedule(ctx): @click.option( "--storage-power-capacity", "storage_power_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Storage consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1258,7 +1313,7 @@ def create_schedule(ctx): @click.option( "--storage-consumption-capacity", "storage_consumption_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Storage consumption power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1268,7 +1323,7 @@ def create_schedule(ctx): @click.option( "--storage-production-capacity", "storage_production_capacity", - type=QuantityOrSensor("MW"), + type=VariableQuantityField("MW"), required=False, default=None, help="Storage production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)" @@ -1278,7 +1333,7 @@ def create_schedule(ctx): @click.option( "--storage-efficiency", "storage_efficiency", - type=QuantityOrSensor("%", default_src_unit="dimensionless"), + type=VariableQuantityField("%", default_src_unit="dimensionless"), required=False, default="100%", help="Storage efficiency (e.g. 95% or 0.95) to use for the schedule," @@ -1421,9 +1476,9 @@ def add_schedule_for_storage( # noqa C901 else: unit = "MW" - scheduling_kwargs[key][field_name] = QuantityOrSensor(unit)._serialize( - value, None, None - ) + scheduling_kwargs[key][field_name] = VariableQuantityField( + unit + )._serialize(value, None, None) if as_job: job = create_scheduling_job(asset_or_sensor=power_sensor, **scheduling_kwargs) diff --git a/flexmeasures/cli/data_edit.py b/flexmeasures/cli/data_edit.py index cb0b22ce4..aaf2eb7c3 100644 --- a/flexmeasures/cli/data_edit.py +++ b/flexmeasures/cli/data_edit.py @@ -282,9 +282,11 @@ def transfer_ownership(asset: Asset, new_owner: Account): def transfer_ownership_recursive(asset: Asset, account: Account): AssetAuditLog.add_record( asset, - f"Transferred ownership for asset '{asset.name}': {asset.id} from '{asset.owner.name}': {asset.owner.id} to '{account.name}': {account.id}" - if asset.owner is not None - else f"Assign ownership to public asset '{asset.name}': {asset.id} to '{account.name}': {account.id}", + ( + f"Transferred ownership for asset '{asset.name}': {asset.id} from '{asset.owner.name}': {asset.owner.id} to '{account.name}': {account.id}" + if asset.owner is not None + else f"Assign ownership to public asset '{asset.name}': {asset.id} to '{account.name}': {account.id}" + ), ) asset.owner = account diff --git a/flexmeasures/cli/data_show.py b/flexmeasures/cli/data_show.py index 94d9da8fa..4ca91de6f 100644 --- a/flexmeasures/cli/data_show.py +++ b/flexmeasures/cli/data_show.py @@ -666,17 +666,17 @@ def plot_beliefs( title += f"\nOnly beliefs made before: {belief_time_before}." if source: title += f"\nSource: {source.description}" - title += f"\nThe time resolution (x-axis) is {naturaldelta(resolution)}." uniplot.plot( - [df[col] for col in df.columns], + ys=[df[col] for col in df.columns], + xs=[df.index for _ in df.columns], title=title, color=True, lines=True, y_unit=shared_unit, - legend_labels=df.columns - if shared_unit - else [f"{col} in {s.unit}" for col in df.columns], + legend_labels=( + df.columns if shared_unit else [f"{col} in {s.unit}" for col in df.columns] + ), ) if filepath is not None: df.columns = pd.MultiIndex.from_arrays( diff --git a/flexmeasures/cli/testing.py b/flexmeasures/cli/testing.py index f219cc76b..a444bde6f 100644 --- a/flexmeasures/cli/testing.py +++ b/flexmeasures/cli/testing.py @@ -30,6 +30,7 @@ They are not registered as app command per default, as we don't need to show them to users. """ + # un-comment to use as CLI function # @app.cli.command() def test_making_forecasts(): diff --git a/flexmeasures/cli/tests/test_data_add.py b/flexmeasures/cli/tests/test_data_add.py index da43c1785..8c65ae1d2 100644 --- a/flexmeasures/cli/tests/test_data_add.py +++ b/flexmeasures/cli/tests/test_data_add.py @@ -514,12 +514,12 @@ def test_add_storage_schedule( if storage_power_capacity is not None: if storage_power_capacity == "sensor": - cli_input_params[ - "storage-consumption-capacity" - ] = f"sensor:{power_capacity_sensor}" - cli_input_params[ - "storage-production-capacity" - ] = f"sensor:{power_capacity_sensor}" + cli_input_params["storage-consumption-capacity"] = ( + f"sensor:{power_capacity_sensor}" + ) + cli_input_params["storage-production-capacity"] = ( + f"sensor:{power_capacity_sensor}" + ) else: cli_input_params["storage-consumption-capacity"] = "700kW" @@ -527,9 +527,9 @@ def test_add_storage_schedule( if storage_efficiency is not None: if storage_efficiency == "sensor": - cli_input_params[ - "storage-efficiency" - ] = f"sensor:{storage_efficiency_sensor}" + cli_input_params["storage-efficiency"] = ( + f"sensor:{storage_efficiency_sensor}" + ) else: cli_input_params["storage-efficiency"] = "90%" diff --git a/flexmeasures/cli/tests/test_data_edit.py b/flexmeasures/cli/tests/test_data_edit.py index dbaa2ad76..8bb62099e 100644 --- a/flexmeasures/cli/tests/test_data_edit.py +++ b/flexmeasures/cli/tests/test_data_edit.py @@ -28,7 +28,7 @@ def test_add_one_sensor_attribute(app, db, setup_markets): result = runner.invoke(edit_attribute, to_flags(cli_input)) assert result.exit_code == 0 and "Success" in result.output, result.exception - event = f"Updated sensor '{sensor.name}': {sensor.id} attribute 'some new attribute' to 3.0 from None" + event = f"Updated sensor '{sensor.name}': {sensor.id}; Attr 'some new attribute' To 3.0 From None" assert db.session.execute( select(AssetAuditLog).filter_by( affected_asset_id=sensor.generic_asset_id, @@ -61,7 +61,7 @@ def test_update_one_asset_attribute(app, db, setup_generic_assets): result = runner.invoke(edit_attribute, to_flags(cli_input)) assert result.exit_code == 0 and "Success" in result.output, result.exception - event = f"Updated asset '{asset.name}': {asset.id} attribute 'some-attribute' to some-new-value from some-value" + event = f"Updated asset '{asset.name}': {asset.id}; Attr 'some-attribute' To some-new-value From some-value" assert db.session.execute( select(AssetAuditLog).filter_by( affected_asset_id=asset.id, diff --git a/flexmeasures/cli/utils.py b/flexmeasures/cli/utils.py index 319cc4cb3..2c536a117 100644 --- a/flexmeasures/cli/utils.py +++ b/flexmeasures/cli/utils.py @@ -12,6 +12,7 @@ from click_default_group import DefaultGroup from flexmeasures.utils.time_utils import get_most_recent_hour, get_timezone +from flexmeasures.utils.validation_utils import validate_color_hex, validate_url from flexmeasures import Sensor @@ -154,7 +155,7 @@ def get_timerange_from_flag( last_7_days: bool = False, last_month: bool = False, last_year: bool = False, - timezone: pytz.BaseTzInfo = get_timezone(), + timezone: pytz.BaseTzInfo | None = None, ) -> tuple[datetime, datetime]: """This function returns a time range [start,end] of the last-X period. See input parameters for more details. @@ -168,6 +169,9 @@ def get_timerange_from_flag( :returns: start:datetime, end:datetime """ + if timezone is None: + timezone = get_timezone() + current_hour = get_most_recent_hour().astimezone(timezone) if last_hour: # last finished hour @@ -298,3 +302,41 @@ def get_sensor_aliases( } return aliases + + +def validate_color_cli(ctx, param, value): + """ + Optional parameter validation + + Validates that a given value is a valid hex color code. + + Parameters: + :param ctx: Click context. + :param param: Click parameter name. + :param value: The color code to validate. + """ + + try: + validate_color_hex(value) + except ValueError as e: + click.secho(str(e), **MsgStyle.ERROR) + raise click.Abort() + + +def validate_url_cli(ctx, param, value): + """ + Optional parameter validation + + Validates that a given value is a valid URL format using regex. + + Parameters: + :param ctx: Click context. + :param param: Click parameter name. + :param value: The URL to validate. + """ + + try: + validate_url(value) + except ValueError as e: + click.secho(str(e), **MsgStyle.ERROR) + raise click.Abort() diff --git a/flexmeasures/conftest.py b/flexmeasures/conftest.py index d7a3cf4f1..b2a42bee9 100644 --- a/flexmeasures/conftest.py +++ b/flexmeasures/conftest.py @@ -504,9 +504,11 @@ def create_assets( for asset_name in ["wind-asset-1", "wind-asset-2", "solar-asset-1"]: asset = GenericAsset( name=asset_name, - generic_asset_type=setup_asset_types["wind"] - if "wind" in asset_name - else setup_asset_types["solar"], + generic_asset_type=( + setup_asset_types["wind"] + if "wind" in asset_name + else setup_asset_types["solar"] + ), owner=setup_accounts["Prosumer"], latitude=10, longitude=100, @@ -819,6 +821,19 @@ def create_test_battery_assets( ) db.session.add(test_battery_sensor) + test_battery_sensor_kw = Sensor( + name="power (kW)", + generic_asset=test_battery, + event_resolution=timedelta(minutes=15), + unit="kW", + attributes=dict( + daily_seasonality=True, + weekly_seasonality=True, + yearly_seasonality=True, + ), + ) + db.session.add(test_battery_sensor_kw) + test_battery_no_prices = GenericAsset( name="Test battery with no known prices", owner=setup_accounts["Prosumer"], diff --git a/flexmeasures/data/config.py b/flexmeasures/data/config.py index 246c233e2..f8c98c6c8 100644 --- a/flexmeasures/data/config.py +++ b/flexmeasures/data/config.py @@ -11,7 +11,9 @@ from flexmeasures.data.models import naming_convention -db: sa = None # typed attributes unavailable in flask-sqlalchemy, see https://github.com/pallets/flask-sqlalchemy/issues/867 +db: sa = ( + None # typed attributes unavailable in flask-sqlalchemy, see https://github.com/pallets/flask-sqlalchemy/issues/867 +) Base = None # type: ignore session_options = None diff --git a/flexmeasures/data/migrations/versions/01fe99da5716_initial.py b/flexmeasures/data/migrations/versions/01fe99da5716_initial.py index 0ddd5ed5c..349973282 100644 --- a/flexmeasures/data/migrations/versions/01fe99da5716_initial.py +++ b/flexmeasures/data/migrations/versions/01fe99da5716_initial.py @@ -5,6 +5,7 @@ Create Date: 2018-03-20 10:29:28.864971 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/02ddbbff29a7_naming_conventions.py b/flexmeasures/data/migrations/versions/02ddbbff29a7_naming_conventions.py index 70e94176b..1f31f3e5d 100644 --- a/flexmeasures/data/migrations/versions/02ddbbff29a7_naming_conventions.py +++ b/flexmeasures/data/migrations/versions/02ddbbff29a7_naming_conventions.py @@ -5,6 +5,7 @@ Create Date: 2020-09-17 11:05:37.195404 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/038bab973c40_add_unique_constraint_for_sensor_names_.py b/flexmeasures/data/migrations/versions/038bab973c40_add_unique_constraint_for_sensor_names_.py index 9abe3701f..d5b229a65 100644 --- a/flexmeasures/data/migrations/versions/038bab973c40_add_unique_constraint_for_sensor_names_.py +++ b/flexmeasures/data/migrations/versions/038bab973c40_add_unique_constraint_for_sensor_names_.py @@ -7,6 +7,7 @@ Create Date: 2022-02-16 22:35:26.330950 """ + from alembic import op # revision identifiers, used by Alembic. diff --git a/flexmeasures/data/migrations/versions/04f0e2d2924a_add_source_id_as_primary_key_for_timed_beliefs.py b/flexmeasures/data/migrations/versions/04f0e2d2924a_add_source_id_as_primary_key_for_timed_beliefs.py index fe2d80d88..8e0ff78ca 100644 --- a/flexmeasures/data/migrations/versions/04f0e2d2924a_add_source_id_as_primary_key_for_timed_beliefs.py +++ b/flexmeasures/data/migrations/versions/04f0e2d2924a_add_source_id_as_primary_key_for_timed_beliefs.py @@ -5,6 +5,7 @@ Create Date: 2021-04-10 13:53:22.561718 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/0af134879301_merge.py b/flexmeasures/data/migrations/versions/0af134879301_merge.py new file mode 100644 index 000000000..8fb3a9efe --- /dev/null +++ b/flexmeasures/data/migrations/versions/0af134879301_merge.py @@ -0,0 +1,21 @@ +"""merge + +Revision ID: 0af134879301 +Revises: +Create Date: 2024-08-23 10:58:49.239866 + +""" + +# revision identifiers, used by Alembic. +revision = "0af134879301" +down_revision = ("9b2b90ee5dbf", "ea84cefba5c0") +branch_labels = None +depends_on = None + + +def upgrade(): + pass + + +def downgrade(): + pass diff --git a/flexmeasures/data/migrations/versions/11b735abebe7_create_power_table_and_drop_measurement_table.py b/flexmeasures/data/migrations/versions/11b735abebe7_create_power_table_and_drop_measurement_table.py index 9b5d70487..790294686 100644 --- a/flexmeasures/data/migrations/versions/11b735abebe7_create_power_table_and_drop_measurement_table.py +++ b/flexmeasures/data/migrations/versions/11b735abebe7_create_power_table_and_drop_measurement_table.py @@ -5,6 +5,7 @@ Create Date: 2018-05-29 14:51:30.331230 """ + from alembic import op import sqlalchemy as sa from sqlalchemy.dialects import postgresql diff --git a/flexmeasures/data/migrations/versions/126d65cbe6b4_asset_audit_log.py b/flexmeasures/data/migrations/versions/126d65cbe6b4_asset_audit_log.py index 6f20cd245..6a0a30739 100644 --- a/flexmeasures/data/migrations/versions/126d65cbe6b4_asset_audit_log.py +++ b/flexmeasures/data/migrations/versions/126d65cbe6b4_asset_audit_log.py @@ -5,6 +5,7 @@ Create Date: 2024-05-27 15:48:07.399594 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/1a4f0e5c4b86_unique_userids_in_ds.py b/flexmeasures/data/migrations/versions/1a4f0e5c4b86_unique_userids_in_ds.py index 8926d43bc..3056ef83b 100644 --- a/flexmeasures/data/migrations/versions/1a4f0e5c4b86_unique_userids_in_ds.py +++ b/flexmeasures/data/migrations/versions/1a4f0e5c4b86_unique_userids_in_ds.py @@ -5,6 +5,7 @@ Create Date: 2018-08-16 12:47:38.492823 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/1ae32ffc8c3f_rename_data_source_unique_constraint.py b/flexmeasures/data/migrations/versions/1ae32ffc8c3f_rename_data_source_unique_constraint.py index 9ab2a7387..4eebbeeac 100644 --- a/flexmeasures/data/migrations/versions/1ae32ffc8c3f_rename_data_source_unique_constraint.py +++ b/flexmeasures/data/migrations/versions/1ae32ffc8c3f_rename_data_source_unique_constraint.py @@ -5,6 +5,7 @@ Create Date: 2021-11-11 16:54:09.302274 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/1b64acf01809_forecasting_job_table.py b/flexmeasures/data/migrations/versions/1b64acf01809_forecasting_job_table.py index d7059ae62..54eef6cd5 100644 --- a/flexmeasures/data/migrations/versions/1b64acf01809_forecasting_job_table.py +++ b/flexmeasures/data/migrations/versions/1b64acf01809_forecasting_job_table.py @@ -5,6 +5,7 @@ Create Date: 2018-08-31 15:09:09.462860 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/1bcccdf0c3e1_unique_usernames.py b/flexmeasures/data/migrations/versions/1bcccdf0c3e1_unique_usernames.py index 5047dca67..9cca5c7a1 100644 --- a/flexmeasures/data/migrations/versions/1bcccdf0c3e1_unique_usernames.py +++ b/flexmeasures/data/migrations/versions/1bcccdf0c3e1_unique_usernames.py @@ -5,6 +5,7 @@ Create Date: 2018-08-08 13:39:56.732107 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/1e8d27922f56_create_price_table.py b/flexmeasures/data/migrations/versions/1e8d27922f56_create_price_table.py index 03123d67d..be0c0d0fc 100644 --- a/flexmeasures/data/migrations/versions/1e8d27922f56_create_price_table.py +++ b/flexmeasures/data/migrations/versions/1e8d27922f56_create_price_table.py @@ -5,6 +5,7 @@ Create Date: 2018-05-28 13:30:54.227839 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/202505c5cb06_assets_inflexible_sensors_table_.py b/flexmeasures/data/migrations/versions/202505c5cb06_assets_inflexible_sensors_table_.py index f5a83908d..cc8f16c8d 100644 --- a/flexmeasures/data/migrations/versions/202505c5cb06_assets_inflexible_sensors_table_.py +++ b/flexmeasures/data/migrations/versions/202505c5cb06_assets_inflexible_sensors_table_.py @@ -5,6 +5,7 @@ Create Date: 2024-05-20 22:23:05.406911 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/22ce09690d23_mix_in_timely_beliefs_sensor_with_asset_market_and_weather_sensor.py b/flexmeasures/data/migrations/versions/22ce09690d23_mix_in_timely_beliefs_sensor_with_asset_market_and_weather_sensor.py index e99ac01c2..912669c02 100644 --- a/flexmeasures/data/migrations/versions/22ce09690d23_mix_in_timely_beliefs_sensor_with_asset_market_and_weather_sensor.py +++ b/flexmeasures/data/migrations/versions/22ce09690d23_mix_in_timely_beliefs_sensor_with_asset_market_and_weather_sensor.py @@ -5,6 +5,7 @@ Create Date: 2021-01-31 14:31:16.370110 """ + from alembic import op import json import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/26373d8266db_owner_deletion_deletes_assets_and_power.py b/flexmeasures/data/migrations/versions/26373d8266db_owner_deletion_deletes_assets_and_power.py index ba6d130b2..a5c34ffa4 100644 --- a/flexmeasures/data/migrations/versions/26373d8266db_owner_deletion_deletes_assets_and_power.py +++ b/flexmeasures/data/migrations/versions/26373d8266db_owner_deletion_deletes_assets_and_power.py @@ -5,6 +5,7 @@ Create Date: 2018-08-08 13:45:19.717975 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/2ac7fb39ce0c_add_attribute_column_to_data_source.py b/flexmeasures/data/migrations/versions/2ac7fb39ce0c_add_attribute_column_to_data_source.py index 8698bc3a5..1f7459429 100644 --- a/flexmeasures/data/migrations/versions/2ac7fb39ce0c_add_attribute_column_to_data_source.py +++ b/flexmeasures/data/migrations/versions/2ac7fb39ce0c_add_attribute_column_to_data_source.py @@ -5,6 +5,7 @@ Create Date: 2023-06-05 23:41:31.788961 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/2c9a32614784_rename_data_source_columns_in_power_price_and_weather_tables.py b/flexmeasures/data/migrations/versions/2c9a32614784_rename_data_source_columns_in_power_price_and_weather_tables.py index 861bcb0f9..81087aa77 100644 --- a/flexmeasures/data/migrations/versions/2c9a32614784_rename_data_source_columns_in_power_price_and_weather_tables.py +++ b/flexmeasures/data/migrations/versions/2c9a32614784_rename_data_source_columns_in_power_price_and_weather_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-07-26 15:58:07.780000 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/30f7b63069e1_delete_asset_if_sensor_is_deleted.py b/flexmeasures/data/migrations/versions/30f7b63069e1_delete_asset_if_sensor_is_deleted.py index 0d315ff38..c22d7f70e 100644 --- a/flexmeasures/data/migrations/versions/30f7b63069e1_delete_asset_if_sensor_is_deleted.py +++ b/flexmeasures/data/migrations/versions/30f7b63069e1_delete_asset_if_sensor_is_deleted.py @@ -5,6 +5,7 @@ Create Date: 2022-03-18 14:44:56.718765 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/30fe2267e7d5_add_optional_DataSource_columns_for_model_and_version.py b/flexmeasures/data/migrations/versions/30fe2267e7d5_add_optional_DataSource_columns_for_model_and_version.py index eac278c35..671097733 100644 --- a/flexmeasures/data/migrations/versions/30fe2267e7d5_add_optional_DataSource_columns_for_model_and_version.py +++ b/flexmeasures/data/migrations/versions/30fe2267e7d5_add_optional_DataSource_columns_for_model_and_version.py @@ -5,6 +5,7 @@ Create Date: 2021-10-11 10:54:24.348371 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/31f251554682_merge.py b/flexmeasures/data/migrations/versions/31f251554682_merge.py index 016048ee7..49653aa8f 100644 --- a/flexmeasures/data/migrations/versions/31f251554682_merge.py +++ b/flexmeasures/data/migrations/versions/31f251554682_merge.py @@ -6,7 +6,6 @@ """ - # revision identifiers, used by Alembic. revision = "31f251554682" down_revision = ("b2b43f0eec40", "61bfc6e45c4d") diff --git a/flexmeasures/data/migrations/versions/3d56402cde15_drop_login_columns_in_bvp_users_table.py b/flexmeasures/data/migrations/versions/3d56402cde15_drop_login_columns_in_bvp_users_table.py index ac3f56232..859b49ae8 100644 --- a/flexmeasures/data/migrations/versions/3d56402cde15_drop_login_columns_in_bvp_users_table.py +++ b/flexmeasures/data/migrations/versions/3d56402cde15_drop_login_columns_in_bvp_users_table.py @@ -5,6 +5,7 @@ Create Date: 2018-04-10 16:42:38.892648 """ + from alembic import op import sqlalchemy as sa from sqlalchemy.dialects import postgresql diff --git a/flexmeasures/data/migrations/versions/3db3e71d101d_make_datasource_a_subclass_of_timely_beliefs_beliefsource.py b/flexmeasures/data/migrations/versions/3db3e71d101d_make_datasource_a_subclass_of_timely_beliefs_beliefsource.py index dacce316f..f4df5b081 100644 --- a/flexmeasures/data/migrations/versions/3db3e71d101d_make_datasource_a_subclass_of_timely_beliefs_beliefsource.py +++ b/flexmeasures/data/migrations/versions/3db3e71d101d_make_datasource_a_subclass_of_timely_beliefs_beliefsource.py @@ -8,6 +8,7 @@ Create Date: 2020-08-10 15:31:28.391337 """ + from alembic import op # revision identifiers, used by Alembic. diff --git a/flexmeasures/data/migrations/versions/3e43d3274d16_Asset_soc_udi_event_id.py b/flexmeasures/data/migrations/versions/3e43d3274d16_Asset_soc_udi_event_id.py index b37c04a61..d63dd0631 100644 --- a/flexmeasures/data/migrations/versions/3e43d3274d16_Asset_soc_udi_event_id.py +++ b/flexmeasures/data/migrations/versions/3e43d3274d16_Asset_soc_udi_event_id.py @@ -5,6 +5,7 @@ Create Date: 2018-09-21 15:07:00.816216 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/3eb0564948ca_add_search_index_for_single_belief_.py b/flexmeasures/data/migrations/versions/3eb0564948ca_add_search_index_for_single_belief_.py new file mode 100644 index 000000000..9b54c7d2a --- /dev/null +++ b/flexmeasures/data/migrations/versions/3eb0564948ca_add_search_index_for_single_belief_.py @@ -0,0 +1,30 @@ +"""add search index for single-belief search + +Revision ID: 3eb0564948ca +Revises: 126d65cbe6b4 +Create Date: 2024-05-12 15:45:25.337949 + +""" + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = "3eb0564948ca" +down_revision = "126d65cbe6b4" +branch_labels = None +depends_on = None + + +def upgrade(): + with op.batch_alter_table("timed_belief", schema=None) as batch_op: + batch_op.create_index( + "timed_belief_search_session_singleevent_idx", + ["event_start", "sensor_id"], + unique=False, + ) + + +def downgrade(): + with op.batch_alter_table("timed_belief", schema=None) as batch_op: + batch_op.drop_index("timed_belief_search_session_singleevent_idx") diff --git a/flexmeasures/data/migrations/versions/40d6c8e4be94_add_parent_asset_id_column.py b/flexmeasures/data/migrations/versions/40d6c8e4be94_add_parent_asset_id_column.py index 62744aeff..fc190aa32 100644 --- a/flexmeasures/data/migrations/versions/40d6c8e4be94_add_parent_asset_id_column.py +++ b/flexmeasures/data/migrations/versions/40d6c8e4be94_add_parent_asset_id_column.py @@ -5,6 +5,7 @@ Create Date: 2023-09-19 17:05:00.020779 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/45d937300b0f_create_measurement_table.py b/flexmeasures/data/migrations/versions/45d937300b0f_create_measurement_table.py index b4eb9dfa2..188a5aa10 100644 --- a/flexmeasures/data/migrations/versions/45d937300b0f_create_measurement_table.py +++ b/flexmeasures/data/migrations/versions/45d937300b0f_create_measurement_table.py @@ -5,6 +5,7 @@ Create Date: 2018-05-07 18:18:59.555454 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/4b5aa7856932_added_primary_and_secondary_color_field_.py b/flexmeasures/data/migrations/versions/4b5aa7856932_added_primary_and_secondary_color_field_.py new file mode 100644 index 000000000..04e3af7c6 --- /dev/null +++ b/flexmeasures/data/migrations/versions/4b5aa7856932_added_primary_and_secondary_color_field_.py @@ -0,0 +1,32 @@ +"""Added primary and secondary color field to account model + +Revision ID: 4b5aa7856932 +Revises: 3eb0564948ca +Create Date: 2024-08-08 13:38:17.197805 + +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "4b5aa7856932" +down_revision = "3eb0564948ca" +branch_labels = None +depends_on = None + + +def upgrade(): + with op.batch_alter_table("account", schema=None) as batch_op: + batch_op.add_column( + sa.Column("primary_color", sa.String(length=7), nullable=True) + ) + batch_op.add_column( + sa.Column("secondary_color", sa.String(length=7), nullable=True) + ) + + +def downgrade(): + with op.batch_alter_table("account", schema=None) as batch_op: + batch_op.drop_column("secondary_color") + batch_op.drop_column("primary_color") diff --git a/flexmeasures/data/migrations/versions/4b6cebbdf473_create_weather_sensor_type_and_weather_sensor_and_weather_tables.py b/flexmeasures/data/migrations/versions/4b6cebbdf473_create_weather_sensor_type_and_weather_sensor_and_weather_tables.py index 2e5f5d9ab..789f63f5b 100644 --- a/flexmeasures/data/migrations/versions/4b6cebbdf473_create_weather_sensor_type_and_weather_sensor_and_weather_tables.py +++ b/flexmeasures/data/migrations/versions/4b6cebbdf473_create_weather_sensor_type_and_weather_sensor_and_weather_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-06-07 12:13:44.504308 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/50cf294e007d_complete_adding_units_to_all_generic_asset_tables_and_display_names_to_all_generic_asset_tables_and_generic_asset_type_tables.py b/flexmeasures/data/migrations/versions/50cf294e007d_complete_adding_units_to_all_generic_asset_tables_and_display_names_to_all_generic_asset_tables_and_generic_asset_type_tables.py index d32a3e7c3..1ff4a5cab 100644 --- a/flexmeasures/data/migrations/versions/50cf294e007d_complete_adding_units_to_all_generic_asset_tables_and_display_names_to_all_generic_asset_tables_and_generic_asset_type_tables.py +++ b/flexmeasures/data/migrations/versions/50cf294e007d_complete_adding_units_to_all_generic_asset_tables_and_display_names_to_all_generic_asset_tables_and_generic_asset_type_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-10-12 11:12:03.525000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/521034349543_add_consultancy_account_to_account.py b/flexmeasures/data/migrations/versions/521034349543_add_consultancy_account_to_account.py index 6d37d0308..9113efa09 100644 --- a/flexmeasures/data/migrations/versions/521034349543_add_consultancy_account_to_account.py +++ b/flexmeasures/data/migrations/versions/521034349543_add_consultancy_account_to_account.py @@ -5,6 +5,7 @@ Create Date: 2023-10-13 11:07:08.013181 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/524800c11eec_add_warning_and_error_as_annotations_.py b/flexmeasures/data/migrations/versions/524800c11eec_add_warning_and_error_as_annotations_.py new file mode 100644 index 000000000..854a5245b --- /dev/null +++ b/flexmeasures/data/migrations/versions/524800c11eec_add_warning_and_error_as_annotations_.py @@ -0,0 +1,54 @@ +"""add warning and error as annotations types + +Revision ID: 524800c11eec +Revises: 4b5aa7856932 +Create Date: 2024-08-19 15:10:24.323594 + +""" + +from alembic import op +from sqlalchemy.sql import text + + +# revision identifiers, used by Alembic. +revision = "524800c11eec" +down_revision = "4b5aa7856932" +branch_labels = None +depends_on = None + + +def upgrade(): + conn = op.get_bind() + + # Check if the 'warning' value exists in the 'annotation_type' enum + result = conn.execute( + text( + """ + SELECT 1 + FROM pg_enum + WHERE enumlabel = 'warning' + AND enumtypid = (SELECT oid FROM pg_type WHERE typname = 'annotation_type') + """ + ) + ) + if result.rowcount == 0: + op.execute("ALTER TYPE annotation_type ADD VALUE 'warning'") + + # Check if the 'error' value exists in the 'annotation_type' enum + result = conn.execute( + text( + """ + SELECT 1 + FROM pg_enum + WHERE enumlabel = 'error' + AND enumtypid = (SELECT oid FROM pg_type WHERE typname = 'annotation_type') + """ + ) + ) + if result.rowcount == 0: + op.execute("ALTER TYPE annotation_type ADD VALUE 'error'") + + +def downgrade(): + # Enum values cannot be removed in Postgres, so no downgrade action is provided + pass diff --git a/flexmeasures/data/migrations/versions/550a9020f1bf_default_resolution_for_existing_sensors.py b/flexmeasures/data/migrations/versions/550a9020f1bf_default_resolution_for_existing_sensors.py index f22eaec47..ed660ec8a 100644 --- a/flexmeasures/data/migrations/versions/550a9020f1bf_default_resolution_for_existing_sensors.py +++ b/flexmeasures/data/migrations/versions/550a9020f1bf_default_resolution_for_existing_sensors.py @@ -5,6 +5,7 @@ Create Date: 2020-11-05 17:48:49.670289 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/564e8df4e3a9_stop_using_bvp_in_table_names.py b/flexmeasures/data/migrations/versions/564e8df4e3a9_stop_using_bvp_in_table_names.py index 61dc4129f..091601512 100644 --- a/flexmeasures/data/migrations/versions/564e8df4e3a9_stop_using_bvp_in_table_names.py +++ b/flexmeasures/data/migrations/versions/564e8df4e3a9_stop_using_bvp_in_table_names.py @@ -5,6 +5,7 @@ Create Date: 2021-01-12 21:44:43.069141 """ + from alembic import op # revision identifiers, used by Alembic. diff --git a/flexmeasures/data/migrations/versions/565e092a6c5e_introduce_the_GenericAssetType_table.py b/flexmeasures/data/migrations/versions/565e092a6c5e_introduce_the_GenericAssetType_table.py index b57af2032..481c5e19f 100644 --- a/flexmeasures/data/migrations/versions/565e092a6c5e_introduce_the_GenericAssetType_table.py +++ b/flexmeasures/data/migrations/versions/565e092a6c5e_introduce_the_GenericAssetType_table.py @@ -5,6 +5,7 @@ Create Date: 2021-07-20 16:16:50.872449 """ + import json from alembic import context, op diff --git a/flexmeasures/data/migrations/versions/5d39829d91af_create_data_sources_table.py b/flexmeasures/data/migrations/versions/5d39829d91af_create_data_sources_table.py index 0600c19ff..347eee231 100644 --- a/flexmeasures/data/migrations/versions/5d39829d91af_create_data_sources_table.py +++ b/flexmeasures/data/migrations/versions/5d39829d91af_create_data_sources_table.py @@ -5,6 +5,7 @@ Create Date: 2018-07-09 17:17:36.276000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/61bfc6e45c4d_add_soc_columns_in_asset_table.py b/flexmeasures/data/migrations/versions/61bfc6e45c4d_add_soc_columns_in_asset_table.py index f62a855e5..bb2c1d185 100644 --- a/flexmeasures/data/migrations/versions/61bfc6e45c4d_add_soc_columns_in_asset_table.py +++ b/flexmeasures/data/migrations/versions/61bfc6e45c4d_add_soc_columns_in_asset_table.py @@ -5,6 +5,7 @@ Create Date: 2018-09-10 11:10:34.068000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/650b085c0ad3_consolidate_data_source_after_storage_.py b/flexmeasures/data/migrations/versions/650b085c0ad3_consolidate_data_source_after_storage_.py index 26fbe36b4..3133bdc5d 100644 --- a/flexmeasures/data/migrations/versions/650b085c0ad3_consolidate_data_source_after_storage_.py +++ b/flexmeasures/data/migrations/versions/650b085c0ad3_consolidate_data_source_after_storage_.py @@ -5,6 +5,7 @@ Create Date: 2022-11-16 07:07:44.281943 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/6938f16617ab_add_search_index_for_beliefs.py b/flexmeasures/data/migrations/versions/6938f16617ab_add_search_index_for_beliefs.py index e48400929..11c1f9a95 100644 --- a/flexmeasures/data/migrations/versions/6938f16617ab_add_search_index_for_beliefs.py +++ b/flexmeasures/data/migrations/versions/6938f16617ab_add_search_index_for_beliefs.py @@ -5,6 +5,7 @@ Create Date: 2024-03-01 09:55:34.910868 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/6cf5b241b85f_copy_attributes_from_old_data_models_to_GenericAsset.py b/flexmeasures/data/migrations/versions/6cf5b241b85f_copy_attributes_from_old_data_models_to_GenericAsset.py index ba3c360f3..db70e87d9 100644 --- a/flexmeasures/data/migrations/versions/6cf5b241b85f_copy_attributes_from_old_data_models_to_GenericAsset.py +++ b/flexmeasures/data/migrations/versions/6cf5b241b85f_copy_attributes_from_old_data_models_to_GenericAsset.py @@ -5,6 +5,7 @@ Create Date: 2021-11-11 17:18:15.395915 """ + import json from datetime import datetime diff --git a/flexmeasures/data/migrations/versions/7113b0f00678_drop_forecasting_jobs_table_and_make_display_names_nullable.py b/flexmeasures/data/migrations/versions/7113b0f00678_drop_forecasting_jobs_table_and_make_display_names_nullable.py index 5c9ba2edc..7596214bd 100644 --- a/flexmeasures/data/migrations/versions/7113b0f00678_drop_forecasting_jobs_table_and_make_display_names_nullable.py +++ b/flexmeasures/data/migrations/versions/7113b0f00678_drop_forecasting_jobs_table_and_make_display_names_nullable.py @@ -5,6 +5,7 @@ Create Date: 2020-06-04 11:29:46.507095 """ + from alembic import op import sqlalchemy as sa from sqlalchemy.dialects import postgresql diff --git a/flexmeasures/data/migrations/versions/75f53d2dbfae_new_field_last_seen_in_user_model.py b/flexmeasures/data/migrations/versions/75f53d2dbfae_new_field_last_seen_in_user_model.py index 0e3ff82c2..3dee9a07b 100644 --- a/flexmeasures/data/migrations/versions/75f53d2dbfae_new_field_last_seen_in_user_model.py +++ b/flexmeasures/data/migrations/versions/75f53d2dbfae_new_field_last_seen_in_user_model.py @@ -5,6 +5,7 @@ Create Date: 2022-11-27 00:15:26.403169 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/7987667dbd43_add_asset_type_hover_label.py b/flexmeasures/data/migrations/versions/7987667dbd43_add_asset_type_hover_label.py index a4a48eb26..277e10c43 100644 --- a/flexmeasures/data/migrations/versions/7987667dbd43_add_asset_type_hover_label.py +++ b/flexmeasures/data/migrations/versions/7987667dbd43_add_asset_type_hover_label.py @@ -5,6 +5,7 @@ Create Date: 2020-06-04 11:36:42.684918 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/7f8b8920355f_create_annotation_table.py b/flexmeasures/data/migrations/versions/7f8b8920355f_create_annotation_table.py index f2b3a94f9..56db29aa5 100644 --- a/flexmeasures/data/migrations/versions/7f8b8920355f_create_annotation_table.py +++ b/flexmeasures/data/migrations/versions/7f8b8920355f_create_annotation_table.py @@ -5,6 +5,7 @@ Create Date: 2022-01-29 20:23:29.996133 """ + from alembic import op import click import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/81cbbf42357b_added_audit_log_table.py b/flexmeasures/data/migrations/versions/81cbbf42357b_added_audit_log_table.py index 0eff6190b..1611515e4 100644 --- a/flexmeasures/data/migrations/versions/81cbbf42357b_added_audit_log_table.py +++ b/flexmeasures/data/migrations/versions/81cbbf42357b_added_audit_log_table.py @@ -5,6 +5,7 @@ Create Date: 2024-04-22 12:40:20.483528 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/830e72a8b218_migrate_sensor_relationships_for_power_price_weather.py b/flexmeasures/data/migrations/versions/830e72a8b218_migrate_sensor_relationships_for_power_price_weather.py index c39390fd8..e5a828edb 100644 --- a/flexmeasures/data/migrations/versions/830e72a8b218_migrate_sensor_relationships_for_power_price_weather.py +++ b/flexmeasures/data/migrations/versions/830e72a8b218_migrate_sensor_relationships_for_power_price_weather.py @@ -5,6 +5,7 @@ Create Date: 2021-12-02 14:58:06.581092 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/8abe32ffa204_add_owner_id_column_in_asset_table.py b/flexmeasures/data/migrations/versions/8abe32ffa204_add_owner_id_column_in_asset_table.py index 3bb3ce066..01813bb28 100644 --- a/flexmeasures/data/migrations/versions/8abe32ffa204_add_owner_id_column_in_asset_table.py +++ b/flexmeasures/data/migrations/versions/8abe32ffa204_add_owner_id_column_in_asset_table.py @@ -5,6 +5,7 @@ Create Date: 2018-04-23 16:54:45.199226 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/8fcc5fec67bc_make_data_source_id_columns_primary_keys.py b/flexmeasures/data/migrations/versions/8fcc5fec67bc_make_data_source_id_columns_primary_keys.py index ae8f87e57..75684bd3f 100644 --- a/flexmeasures/data/migrations/versions/8fcc5fec67bc_make_data_source_id_columns_primary_keys.py +++ b/flexmeasures/data/migrations/versions/8fcc5fec67bc_make_data_source_id_columns_primary_keys.py @@ -5,6 +5,7 @@ Create Date: 2018-07-30 15:39:30.583000 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/919dc9f1dc1f_merge.py b/flexmeasures/data/migrations/versions/919dc9f1dc1f_merge.py index 6c555deac..d680334f1 100644 --- a/flexmeasures/data/migrations/versions/919dc9f1dc1f_merge.py +++ b/flexmeasures/data/migrations/versions/919dc9f1dc1f_merge.py @@ -6,7 +6,6 @@ """ - # revision identifiers, used by Alembic. revision = "919dc9f1dc1f" down_revision = ("db00b66be82c", "5d39829d91af") diff --git a/flexmeasures/data/migrations/versions/91a938bfa5a8_add_horizon_columns_to_power_price_and_weather_tables.py b/flexmeasures/data/migrations/versions/91a938bfa5a8_add_horizon_columns_to_power_price_and_weather_tables.py index 11c07884d..0f131668b 100644 --- a/flexmeasures/data/migrations/versions/91a938bfa5a8_add_horizon_columns_to_power_price_and_weather_tables.py +++ b/flexmeasures/data/migrations/versions/91a938bfa5a8_add_horizon_columns_to_power_price_and_weather_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-06-08 15:46:28.950464 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/9254559dcac2_create_market_type_and_market_tables.py b/flexmeasures/data/migrations/versions/9254559dcac2_create_market_type_and_market_tables.py index ebc8eaf23..c3a744db6 100644 --- a/flexmeasures/data/migrations/versions/9254559dcac2_create_market_type_and_market_tables.py +++ b/flexmeasures/data/migrations/versions/9254559dcac2_create_market_type_and_market_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-05-28 13:02:55.210603 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/96f2db5bed30_add_account_roles.py b/flexmeasures/data/migrations/versions/96f2db5bed30_add_account_roles.py index 2abef020e..944be2ee6 100644 --- a/flexmeasures/data/migrations/versions/96f2db5bed30_add_account_roles.py +++ b/flexmeasures/data/migrations/versions/96f2db5bed30_add_account_roles.py @@ -5,6 +5,7 @@ Create Date: 2021-08-30 11:33:40.481140 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/994170c26bc6_add_account_table.py b/flexmeasures/data/migrations/versions/994170c26bc6_add_account_table.py index c36c90be8..57c7bb3fa 100644 --- a/flexmeasures/data/migrations/versions/994170c26bc6_add_account_table.py +++ b/flexmeasures/data/migrations/versions/994170c26bc6_add_account_table.py @@ -5,6 +5,7 @@ Create Date: 2021-08-11 19:21:07.083253 """ + from typing import List, Tuple, Optional import os import json diff --git a/flexmeasures/data/migrations/versions/9b2b90ee5dbf_add_ondelete_cascade_to_foreign_keys_in_account_sensor_and_asset_relationships.py b/flexmeasures/data/migrations/versions/9b2b90ee5dbf_add_ondelete_cascade_to_foreign_keys_in_account_sensor_and_asset_relationships.py new file mode 100644 index 000000000..264785082 --- /dev/null +++ b/flexmeasures/data/migrations/versions/9b2b90ee5dbf_add_ondelete_cascade_to_foreign_keys_in_account_sensor_and_asset_relationships.py @@ -0,0 +1,146 @@ +"""Add ondelete cascade to foreign keys in GenericAssetAnnotationRelationship + +Revision ID: 9b2b90ee5dbf +Revises: 524800c11eec +Create Date: 2024-08-22 12:22:45.240872 + +""" + +from alembic import op + +# revision identifiers, used by Alembic. +revision = "9b2b90ee5dbf" +down_revision = "524800c11eec" +branch_labels = None +depends_on = None + + +def upgrade(): + with op.batch_alter_table("annotations_assets", schema=None) as batch_op: + batch_op.drop_constraint( + "annotations_assets_generic_asset_id_generic_asset_fkey", type_="foreignkey" + ) + batch_op.drop_constraint( + "annotations_assets_annotation_id_annotation_fkey", type_="foreignkey" + ) + batch_op.create_foreign_key( + batch_op.f("annotations_assets_generic_asset_id_generic_asset_fkey"), + "generic_asset", + ["generic_asset_id"], + ["id"], + ondelete="CASCADE", + ) + batch_op.create_foreign_key( + batch_op.f("annotations_assets_annotation_id_annotation_fkey"), + "annotation", + ["annotation_id"], + ["id"], + ondelete="CASCADE", + ) + with op.batch_alter_table("annotations_accounts", schema=None) as batch_op: + batch_op.drop_constraint( + "annotations_accounts_account_id_account_fkey", type_="foreignkey" + ) + batch_op.drop_constraint( + "annotations_accounts_annotation_id_annotation_fkey", type_="foreignkey" + ) + batch_op.create_foreign_key( + batch_op.f("annotations_accounts_account_id_account_fkey"), + "account", + ["account_id"], + ["id"], + ondelete="CASCADE", + ) + batch_op.create_foreign_key( + batch_op.f("annotations_accounts_annotation_id_annotation_fkey"), + "annotation", + ["annotation_id"], + ["id"], + ondelete="CASCADE", + ) + + with op.batch_alter_table("annotations_sensors", schema=None) as batch_op: + batch_op.drop_constraint( + "annotations_sensors_annotation_id_annotation_fkey", type_="foreignkey" + ) + batch_op.drop_constraint( + "annotations_sensors_sensor_id_sensor_fkey", type_="foreignkey" + ) + batch_op.create_foreign_key( + batch_op.f("annotations_sensors_annotation_id_annotation_fkey"), + "annotation", + ["annotation_id"], + ["id"], + ondelete="CASCADE", + ) + batch_op.create_foreign_key( + batch_op.f("annotations_sensors_sensor_id_sensor_fkey"), + "sensor", + ["sensor_id"], + ["id"], + ondelete="CASCADE", + ) + + +def downgrade(): + with op.batch_alter_table("annotations_assets", schema=None) as batch_op: + batch_op.drop_constraint( + batch_op.f("annotations_assets_annotation_id_annotation_fkey"), + type_="foreignkey", + ) + batch_op.drop_constraint( + batch_op.f("annotations_assets_generic_asset_id_generic_asset_fkey"), + type_="foreignkey", + ) + batch_op.create_foreign_key( + "annotations_assets_annotation_id_annotation_fkey", + "annotation", + ["annotation_id"], + ["id"], + ) + batch_op.create_foreign_key( + "annotations_assets_generic_asset_id_generic_asset_fkey", + "generic_asset", + ["generic_asset_id"], + ["id"], + ) + + with op.batch_alter_table("annotations_sensors", schema=None) as batch_op: + batch_op.drop_constraint( + batch_op.f("annotations_sensors_sensor_id_sensor_fkey"), type_="foreignkey" + ) + batch_op.drop_constraint( + batch_op.f("annotations_sensors_annotation_id_annotation_fkey"), + type_="foreignkey", + ) + batch_op.create_foreign_key( + "annotations_sensors_sensor_id_sensor_fkey", "sensor", ["sensor_id"], ["id"] + ) + batch_op.create_foreign_key( + "annotations_sensors_annotation_id_annotation_fkey", + "annotation", + ["annotation_id"], + ["id"], + ) + + with op.batch_alter_table("annotations_accounts", schema=None) as batch_op: + batch_op.drop_constraint( + batch_op.f("annotations_accounts_annotation_id_annotation_fkey"), + type_="foreignkey", + ) + batch_op.drop_constraint( + batch_op.f("annotations_accounts_account_id_account_fkey"), + type_="foreignkey", + ) + batch_op.create_foreign_key( + "annotations_accounts_annotation_id_annotation_fkey", + "annotation", + ["annotation_id"], + ["id"], + ) + batch_op.create_foreign_key( + "annotations_accounts_account_id_account_fkey", + "account", + ["account_id"], + ["id"], + ) diff --git a/flexmeasures/data/migrations/versions/9c7fc8e46f1e_add_location_columns_to_weather_sensor_table.py b/flexmeasures/data/migrations/versions/9c7fc8e46f1e_add_location_columns_to_weather_sensor_table.py index ae6cc0089..38afe3390 100644 --- a/flexmeasures/data/migrations/versions/9c7fc8e46f1e_add_location_columns_to_weather_sensor_table.py +++ b/flexmeasures/data/migrations/versions/9c7fc8e46f1e_add_location_columns_to_weather_sensor_table.py @@ -5,6 +5,7 @@ Create Date: 2018-07-27 12:38:27.006000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/a328412b4623_add_timezone_column_in_bvp_users_table.py b/flexmeasures/data/migrations/versions/a328412b4623_add_timezone_column_in_bvp_users_table.py index 075626fd6..59dde1977 100644 --- a/flexmeasures/data/migrations/versions/a328412b4623_add_timezone_column_in_bvp_users_table.py +++ b/flexmeasures/data/migrations/versions/a328412b4623_add_timezone_column_in_bvp_users_table.py @@ -5,6 +5,7 @@ Create Date: 2018-04-24 15:31:18.190845 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/a528c3c81506_unique_generic_sensor_ids.py b/flexmeasures/data/migrations/versions/a528c3c81506_unique_generic_sensor_ids.py index 5b51ddf37..7ac335dee 100644 --- a/flexmeasures/data/migrations/versions/a528c3c81506_unique_generic_sensor_ids.py +++ b/flexmeasures/data/migrations/versions/a528c3c81506_unique_generic_sensor_ids.py @@ -52,6 +52,7 @@ (downgrade) a 1,2,6 <- 1,2,6 m 3,4,8 <- 9,10,14 w 1,6,7 <- 15,20,21 (- max_market_id) """ + from alembic import op import sqlalchemy as sa from sqlalchemy import orm diff --git a/flexmeasures/data/migrations/versions/a5b970eadb3b_time_series_indexes.py b/flexmeasures/data/migrations/versions/a5b970eadb3b_time_series_indexes.py index 1f6e82efe..8f132fa30 100644 --- a/flexmeasures/data/migrations/versions/a5b970eadb3b_time_series_indexes.py +++ b/flexmeasures/data/migrations/versions/a5b970eadb3b_time_series_indexes.py @@ -5,6 +5,7 @@ Create Date: 2020-11-05 17:15:31.767627 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/a60cc43aef5e_delete_children_assets_on_cascade_when_.py b/flexmeasures/data/migrations/versions/a60cc43aef5e_delete_children_assets_on_cascade_when_.py index 3668deb6b..56725c36c 100644 --- a/flexmeasures/data/migrations/versions/a60cc43aef5e_delete_children_assets_on_cascade_when_.py +++ b/flexmeasures/data/migrations/versions/a60cc43aef5e_delete_children_assets_on_cascade_when_.py @@ -5,6 +5,7 @@ Create Date: 2023-10-11 14:04:19.447773 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/a918360f7d63_add_unique_contraints_on_.py b/flexmeasures/data/migrations/versions/a918360f7d63_add_unique_contraints_on_.py index 8f0414819..8ff6d230b 100644 --- a/flexmeasures/data/migrations/versions/a918360f7d63_add_unique_contraints_on_.py +++ b/flexmeasures/data/migrations/versions/a918360f7d63_add_unique_contraints_on_.py @@ -5,6 +5,7 @@ Create Date: 2022-01-01 22:08:50.163734 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/ac2613fffc74_add_market_id_column_to_asset_table.py b/flexmeasures/data/migrations/versions/ac2613fffc74_add_market_id_column_to_asset_table.py index 42595db0e..6ad644cbd 100644 --- a/flexmeasures/data/migrations/versions/ac2613fffc74_add_market_id_column_to_asset_table.py +++ b/flexmeasures/data/migrations/versions/ac2613fffc74_add_market_id_column_to_asset_table.py @@ -5,6 +5,7 @@ Create Date: 2018-10-23 15:49:36.312000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/ac5e340cccea_change_scope_of_name_account_id_unique_constraint.py b/flexmeasures/data/migrations/versions/ac5e340cccea_change_scope_of_name_account_id_unique_constraint.py index 0539feeb6..950240cca 100644 --- a/flexmeasures/data/migrations/versions/ac5e340cccea_change_scope_of_name_account_id_unique_constraint.py +++ b/flexmeasures/data/migrations/versions/ac5e340cccea_change_scope_of_name_account_id_unique_constraint.py @@ -8,6 +8,7 @@ Create Date: 2023-10-05 15:13:36.641051 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py b/flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py index 818907cd0..417beb932 100644 --- a/flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py +++ b/flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py @@ -5,6 +5,7 @@ Create Date: 2023-11-30 10:31:46.125670 """ + from alembic import op import sqlalchemy as sa from sqlalchemy.dialects import postgresql diff --git a/flexmeasures/data/migrations/versions/b087ce8b529f_create_latest_task_run_table.py b/flexmeasures/data/migrations/versions/b087ce8b529f_create_latest_task_run_table.py index 6a94ec732..620f2516b 100644 --- a/flexmeasures/data/migrations/versions/b087ce8b529f_create_latest_task_run_table.py +++ b/flexmeasures/data/migrations/versions/b087ce8b529f_create_latest_task_run_table.py @@ -5,6 +5,7 @@ Create Date: 2018-06-14 12:06:05.037547 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/b2b43f0eec40_weathersensors_unique_type_location.py b/flexmeasures/data/migrations/versions/b2b43f0eec40_weathersensors_unique_type_location.py index 968019fb1..bfe6bfb37 100644 --- a/flexmeasures/data/migrations/versions/b2b43f0eec40_weathersensors_unique_type_location.py +++ b/flexmeasures/data/migrations/versions/b2b43f0eec40_weathersensors_unique_type_location.py @@ -5,6 +5,7 @@ Create Date: 2018-09-12 11:14:46.486640 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/b6d49ed7cceb_introduce_the_GenericAsset_table.py b/flexmeasures/data/migrations/versions/b6d49ed7cceb_introduce_the_GenericAsset_table.py index 3adf80079..e3b019438 100644 --- a/flexmeasures/data/migrations/versions/b6d49ed7cceb_introduce_the_GenericAsset_table.py +++ b/flexmeasures/data/migrations/versions/b6d49ed7cceb_introduce_the_GenericAsset_table.py @@ -5,6 +5,7 @@ Create Date: 2021-07-20 20:15:28.019102 """ + import json from alembic import context, op diff --git a/flexmeasures/data/migrations/versions/b797328ac32d_add_user_fs_uniquifier_for_faster_auth_.py b/flexmeasures/data/migrations/versions/b797328ac32d_add_user_fs_uniquifier_for_faster_auth_.py index 31e11f5be..946e53e14 100644 --- a/flexmeasures/data/migrations/versions/b797328ac32d_add_user_fs_uniquifier_for_faster_auth_.py +++ b/flexmeasures/data/migrations/versions/b797328ac32d_add_user_fs_uniquifier_for_faster_auth_.py @@ -5,6 +5,7 @@ Create Date: 2020-08-24 19:01:04.337956 """ + import uuid from alembic import op diff --git a/flexmeasures/data/migrations/versions/bddc5e9f72a3_add_event_resolution_field_to_asset_.py b/flexmeasures/data/migrations/versions/bddc5e9f72a3_add_event_resolution_field_to_asset_.py index 1b114b4d3..1ba0a52b1 100644 --- a/flexmeasures/data/migrations/versions/bddc5e9f72a3_add_event_resolution_field_to_asset_.py +++ b/flexmeasures/data/migrations/versions/bddc5e9f72a3_add_event_resolution_field_to_asset_.py @@ -5,6 +5,7 @@ Create Date: 2020-10-07 14:12:45.761789 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/c1d316c60985_merge.py b/flexmeasures/data/migrations/versions/c1d316c60985_merge.py index 6eb97c690..f404eaf7b 100644 --- a/flexmeasures/data/migrations/versions/c1d316c60985_merge.py +++ b/flexmeasures/data/migrations/versions/c1d316c60985_merge.py @@ -6,7 +6,6 @@ """ - # revision identifiers, used by Alembic. revision = "c1d316c60985" down_revision = ("a918360f7d63", "e690d373a3d9") diff --git a/flexmeasures/data/migrations/versions/c349f52c700d_update_data_sources.py b/flexmeasures/data/migrations/versions/c349f52c700d_update_data_sources.py index 499a0e9cb..e652349a8 100644 --- a/flexmeasures/data/migrations/versions/c349f52c700d_update_data_sources.py +++ b/flexmeasures/data/migrations/versions/c349f52c700d_update_data_sources.py @@ -5,6 +5,7 @@ Create Date: 2023-12-14 10:31:02.612590 """ + from alembic import op import sqlalchemy as sa from sqlalchemy.dialects import postgresql diff --git a/flexmeasures/data/migrations/versions/c41beee0c904_rename_DataSource_type_for_forecasters_and_schedulers.py b/flexmeasures/data/migrations/versions/c41beee0c904_rename_DataSource_type_for_forecasters_and_schedulers.py index 4510f7294..9a5e9cdf8 100644 --- a/flexmeasures/data/migrations/versions/c41beee0c904_rename_DataSource_type_for_forecasters_and_schedulers.py +++ b/flexmeasures/data/migrations/versions/c41beee0c904_rename_DataSource_type_for_forecasters_and_schedulers.py @@ -5,6 +5,7 @@ Create Date: 2022-11-30 21:33:09.046751 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/d3440de27ab9_create_bvp_roles_and_bvp_users_and_bvp_roles_users_tables.py b/flexmeasures/data/migrations/versions/d3440de27ab9_create_bvp_roles_and_bvp_users_and_bvp_roles_users_tables.py index ee2133a2c..0e445abfd 100644 --- a/flexmeasures/data/migrations/versions/d3440de27ab9_create_bvp_roles_and_bvp_users_and_bvp_roles_users_tables.py +++ b/flexmeasures/data/migrations/versions/d3440de27ab9_create_bvp_roles_and_bvp_users_and_bvp_roles_users_tables.py @@ -5,6 +5,7 @@ Create Date: 2018-03-27 16:50:13.265722 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/d814c0688ae0_merge.py b/flexmeasures/data/migrations/versions/d814c0688ae0_merge.py index ff3204329..f43f6446b 100644 --- a/flexmeasures/data/migrations/versions/d814c0688ae0_merge.py +++ b/flexmeasures/data/migrations/versions/d814c0688ae0_merge.py @@ -6,7 +6,6 @@ """ - # revision identifiers, used by Alembic. revision = "d814c0688ae0" down_revision = ("75f53d2dbfae", "c41beee0c904") diff --git a/flexmeasures/data/migrations/versions/db00b66be82c_add_horizon_columns_as_primary_keys.py b/flexmeasures/data/migrations/versions/db00b66be82c_add_horizon_columns_as_primary_keys.py index 411757599..9ad669bd5 100644 --- a/flexmeasures/data/migrations/versions/db00b66be82c_add_horizon_columns_as_primary_keys.py +++ b/flexmeasures/data/migrations/versions/db00b66be82c_add_horizon_columns_as_primary_keys.py @@ -5,6 +5,7 @@ Create Date: 2018-07-10 10:31:58.915035 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/db1f67336324_add_price_unit_and_display_name_columns_to_market_table.py b/flexmeasures/data/migrations/versions/db1f67336324_add_price_unit_and_display_name_columns_to_market_table.py index 4e0145d3e..3f2943ffa 100644 --- a/flexmeasures/data/migrations/versions/db1f67336324_add_price_unit_and_display_name_columns_to_market_table.py +++ b/flexmeasures/data/migrations/versions/db1f67336324_add_price_unit_and_display_name_columns_to_market_table.py @@ -5,6 +5,7 @@ Create Date: 2018-10-07 13:50:23.690000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/e0c2f9aff251_rename_source_id_column_in_data_sources_table.py b/flexmeasures/data/migrations/versions/e0c2f9aff251_rename_source_id_column_in_data_sources_table.py index 03b8fbbf6..4c228a018 100644 --- a/flexmeasures/data/migrations/versions/e0c2f9aff251_rename_source_id_column_in_data_sources_table.py +++ b/flexmeasures/data/migrations/versions/e0c2f9aff251_rename_source_id_column_in_data_sources_table.py @@ -5,6 +5,7 @@ Create Date: 2018-07-20 16:08:50.641000 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/e4c9cf837311_sensor_generic_asset_id_should_not_be_.py b/flexmeasures/data/migrations/versions/e4c9cf837311_sensor_generic_asset_id_should_not_be_.py index 08bb98702..38cc795a2 100644 --- a/flexmeasures/data/migrations/versions/e4c9cf837311_sensor_generic_asset_id_should_not_be_.py +++ b/flexmeasures/data/migrations/versions/e4c9cf837311_sensor_generic_asset_id_should_not_be_.py @@ -5,6 +5,7 @@ Create Date: 2021-08-15 20:42:29.729532 """ + from alembic import op diff --git a/flexmeasures/data/migrations/versions/e62ac5f519d7_create_table_for_timed_beliefs.py b/flexmeasures/data/migrations/versions/e62ac5f519d7_create_table_for_timed_beliefs.py index 95fb2ddb3..550e6b46f 100644 --- a/flexmeasures/data/migrations/versions/e62ac5f519d7_create_table_for_timed_beliefs.py +++ b/flexmeasures/data/migrations/versions/e62ac5f519d7_create_table_for_timed_beliefs.py @@ -5,6 +5,7 @@ Create Date: 2021-03-28 16:26:45.025994 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/e690d373a3d9_copy_Power_Price_Weather_time_series_data_to_TimedBeliefs_table.py b/flexmeasures/data/migrations/versions/e690d373a3d9_copy_Power_Price_Weather_time_series_data_to_TimedBeliefs_table.py index cdb055b7e..c470c7b93 100644 --- a/flexmeasures/data/migrations/versions/e690d373a3d9_copy_Power_Price_Weather_time_series_data_to_TimedBeliefs_table.py +++ b/flexmeasures/data/migrations/versions/e690d373a3d9_copy_Power_Price_Weather_time_series_data_to_TimedBeliefs_table.py @@ -8,6 +8,7 @@ Create Date: 2021-12-27 15:01:38.967237 """ + from alembic import op import sqlalchemy as sa diff --git a/flexmeasures/data/migrations/versions/ea84cefba5c0_added_logo_url_field_to_account_model.py b/flexmeasures/data/migrations/versions/ea84cefba5c0_added_logo_url_field_to_account_model.py new file mode 100644 index 000000000..43701efb5 --- /dev/null +++ b/flexmeasures/data/migrations/versions/ea84cefba5c0_added_logo_url_field_to_account_model.py @@ -0,0 +1,26 @@ +"""added logo_url field to account model + +Revision ID: ea84cefba5c0 +Revises: 4b5aa7856932 +Create Date: 2024-08-19 14:07:14.187428 + +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "ea84cefba5c0" +down_revision = "4b5aa7856932" +branch_labels = None +depends_on = None + + +def upgrade(): + with op.batch_alter_table("account", schema=None) as batch_op: + batch_op.add_column(sa.Column("logo_url", sa.String(length=255), nullable=True)) + + +def downgrade(): + with op.batch_alter_table("account", schema=None) as batch_op: + batch_op.drop_column("logo_url") diff --git a/flexmeasures/data/models/annotations.py b/flexmeasures/data/models/annotations.py index 0c6a13441..f06b4ab3f 100644 --- a/flexmeasures/data/models/annotations.py +++ b/flexmeasures/data/models/annotations.py @@ -30,7 +30,15 @@ class Annotation(db.Model): backref=db.backref("annotations", lazy=True), ) type = db.Column( - db.Enum("alert", "holiday", "label", "feedback", name="annotation_type"), + db.Enum( + "alert", + "holiday", + "label", + "feedback", + "warning", + "error", + name="annotation_type", + ), nullable=False, ) content = db.Column(db.String(1024), nullable=False) @@ -134,8 +142,10 @@ class AccountAnnotationRelationship(db.Model): __tablename__ = "annotations_accounts" id = db.Column(db.Integer(), primary_key=True) - account_id = db.Column(db.Integer, db.ForeignKey("account.id")) - annotation_id = db.Column(db.Integer, db.ForeignKey("annotation.id")) + account_id = db.Column(db.Integer, db.ForeignKey("account.id", ondelete="CASCADE")) + annotation_id = db.Column( + db.Integer, db.ForeignKey("annotation.id", ondelete="CASCADE") + ) __table_args__ = ( db.UniqueConstraint( "annotation_id", @@ -151,8 +161,12 @@ class GenericAssetAnnotationRelationship(db.Model): __tablename__ = "annotations_assets" id = db.Column(db.Integer(), primary_key=True) - generic_asset_id = db.Column(db.Integer, db.ForeignKey("generic_asset.id")) - annotation_id = db.Column(db.Integer, db.ForeignKey("annotation.id")) + generic_asset_id = db.Column( + db.Integer, db.ForeignKey("generic_asset.id", ondelete="CASCADE") + ) + annotation_id = db.Column( + db.Integer, db.ForeignKey("annotation.id", ondelete="CASCADE") + ) __table_args__ = ( db.UniqueConstraint( "annotation_id", @@ -168,8 +182,10 @@ class SensorAnnotationRelationship(db.Model): __tablename__ = "annotations_sensors" id = db.Column(db.Integer(), primary_key=True) - sensor_id = db.Column(db.Integer, db.ForeignKey("sensor.id")) - annotation_id = db.Column(db.Integer, db.ForeignKey("annotation.id")) + sensor_id = db.Column(db.Integer, db.ForeignKey("sensor.id", ondelete="CASCADE")) + annotation_id = db.Column( + db.Integer, db.ForeignKey("annotation.id", ondelete="CASCADE") + ) __table_args__ = ( db.UniqueConstraint( "annotation_id", diff --git a/flexmeasures/data/models/audit_log.py b/flexmeasures/data/models/audit_log.py index 41768f7b1..caf24267b 100644 --- a/flexmeasures/data/models/audit_log.py +++ b/flexmeasures/data/models/audit_log.py @@ -142,16 +142,18 @@ def add_record_for_attribute_update( old_value = asset_or_sensor.attributes.get(attribute_key) if entity_type == "sensor": - event = f"Updated sensor '{asset_or_sensor.name}': {asset_or_sensor.id} " + event = f"Updated sensor '{asset_or_sensor.name}': {asset_or_sensor.id}; " affected_asset_id = (asset_or_sensor.generic_asset_id,) else: - event = f"Updated asset '{asset_or_sensor.name}': {asset_or_sensor.id} " + event = f"Updated asset '{asset_or_sensor.name}': {asset_or_sensor.id}; " affected_asset_id = asset_or_sensor.id - event += f"attribute '{attribute_key}' to {attribute_value} from {old_value}" + event += f"Attr '{attribute_key}' To {attribute_value} From {old_value}" audit_log = cls( event_datetime=server_now(), - event=event, + event=truncate_string( + event, 255 + ), # we truncate the event string if it 255 characters by adding ellipses in the middle active_user_id=current_user_id, active_user_name=current_user_name, affected_asset_id=affected_asset_id, @@ -170,11 +172,22 @@ def add_record( :param event: event to log """ current_user_id, current_user_name = get_current_user_id_name() + audit_log = AssetAuditLog( event_datetime=server_now(), - event=event, + event=truncate_string( + event, 255 + ), # we truncate the event string if it exceed 255 characters by adding ellipses in the middle active_user_id=current_user_id, active_user_name=current_user_name, affected_asset_id=asset.id, ) db.session.add(audit_log) + + +def truncate_string(value: str, max_length: int) -> str: + """Truncate a string and add ellipses in the middle if it exceeds max_length.""" + if len(value) <= max_length: + return value + half_length = (max_length - 5) // 2 + return f"{value[:half_length]} ... {value[-half_length:]}" diff --git a/flexmeasures/data/models/charts/belief_charts.py b/flexmeasures/data/models/charts/belief_charts.py index 5242789c5..990481fa0 100644 --- a/flexmeasures/data/models/charts/belief_charts.py +++ b/flexmeasures/data/models/charts/belief_charts.py @@ -5,7 +5,6 @@ from flexmeasures.data.models.charts.defaults import FIELD_DEFINITIONS, REPLAY_RULER from flexmeasures.utils.flexmeasures_inflection import ( capitalize, - join_words_into_a_list, ) from flexmeasures.utils.coding_utils import flatten_unique from flexmeasures.utils.unit_utils import ( @@ -15,12 +14,25 @@ ) -def bar_chart( +def create_bar_chart_or_histogram_specs( sensor: "Sensor", # noqa F821 event_starts_after: datetime | None = None, event_ends_before: datetime | None = None, + chart_type: str = "bar_chart", **override_chart_specs: dict, ): + """ + This function generates the specifications required to visualize sensor data either as a bar chart or a histogram. + The chart type (bar_chart or histogram) can be specified, and various field definitions are set up based on the sensor attributes and + event time range. The resulting specifications can be customized further through additional keyword arguments. + + The function handles the following: + - Determines unit and formats for the sensor data. + - Configures event value and event start field definitions. + - Sets the appropriate mark type and interpolation based on sensor attributes. + - Defines chart specifications for both bar charts and histograms, including titles, axis configurations, and tooltips. + - Merges any additional specifications provided through keyword arguments into the final chart specifications. + """ unit = sensor.unit if sensor.unit else "a.u." event_value_field_definition = dict( title=f"{capitalize(sensor.sensor_type)} ({unit})", @@ -50,9 +62,31 @@ def bar_chart( if sensor.event_resolution == timedelta(0) and sensor.has_attribute("interpolate"): mark_type = "area" mark_interpolate = sensor.get_attribute("interpolate") + replay_ruler = REPLAY_RULER.copy() + if chart_type == "histogram": + description = "A histogram showing the distribution of sensor data." + x = { + **event_value_field_definition, + "bin": True, + } + y = { + "aggregate": "count", + "title": "Count", + } + replay_ruler["encoding"] = { + "detail": { + "field": "belief_time", + "type": "temporal", + "title": None, + }, + } + else: + description = (f"A simple {mark_type} chart showing sensor data.",) + x = event_start_field_definition + y = event_value_field_definition + chart_specs = { - "description": f"A simple {mark_type} chart showing sensor data.", - # the sensor type is already shown as the y-axis title (avoid redundant info) + "description": description, "title": capitalize(sensor.name) if sensor.name != sensor.sensor_type else None, "layer": [ { @@ -63,13 +97,17 @@ def bar_chart( "width": {"band": 0.999}, }, "encoding": { - "x": event_start_field_definition, - "y": event_value_field_definition, + "x": x, + "y": y, "color": FIELD_DEFINITIONS["source_name"], "detail": FIELD_DEFINITIONS["source"], "opacity": {"value": 0.7}, "tooltip": [ - FIELD_DEFINITIONS["full_date"], + ( + FIELD_DEFINITIONS["full_date"] + if chart_type != "histogram" + else None + ), { **event_value_field_definition, **dict(title=f"{capitalize(sensor.sensor_type)}"), @@ -84,8 +122,11 @@ def bar_chart( "as": "source_name_and_id", }, ], + "selection": { + "scroll": {"type": "interval", "bind": "scales", "encodings": ["x"]} + }, }, - REPLAY_RULER, + replay_ruler, ], } for k, v in override_chart_specs.items(): @@ -93,6 +134,48 @@ def bar_chart( return chart_specs +def histogram( + sensor: "Sensor", # noqa F821 + event_starts_after: datetime | None = None, + event_ends_before: datetime | None = None, + **override_chart_specs: dict, +): + """ + Generates specifications for a histogram chart using sensor data. This function leverages + the `create_bar_chart_or_histogram_specs` helper function, specifying `chart_type` as 'histogram'. + """ + + chart_type = "histogram" + chart_specs = create_bar_chart_or_histogram_specs( + sensor, + event_starts_after, + event_ends_before, + chart_type, + **override_chart_specs, + ) + return chart_specs + + +def bar_chart( + sensor: "Sensor", # noqa F821 + event_starts_after: datetime | None = None, + event_ends_before: datetime | None = None, + **override_chart_specs: dict, +): + """ + Generates specifications for a bar chart using sensor data. This function leverages + the `create_bar_chart_or_histogram_specs` helper function to create the specifications. + """ + + chart_specs = create_bar_chart_or_histogram_specs( + sensor, + event_starts_after, + event_ends_before, + **override_chart_specs, + ) + return chart_specs + + def daily_heatmap( sensor: "Sensor", # noqa F821 event_starts_after: datetime | None = None, @@ -395,9 +478,10 @@ def create_fall_dst_transition_layer( def chart_for_multiple_sensors( - sensors_to_show: list["Sensor", list["Sensor"]], # noqa F821 + sensors_to_show: list["Sensor" | list["Sensor"] | dict[str, "Sensor"]], # noqa F821 event_starts_after: datetime | None = None, event_ends_before: datetime | None = None, + combine_legend: bool = True, **override_chart_specs: dict, ): # Determine the shared data resolution @@ -424,19 +508,18 @@ def chart_for_multiple_sensors( ] } - # Set up field definition for sensor descriptions - sensor_field_definition = FIELD_DEFINITIONS["sensor_description"].copy() - sensor_field_definition["scale"] = dict( - domain=[sensor.to_dict()["description"] for sensor in all_shown_sensors] - ) - sensors_specs = [] - for s in sensors_to_show: + for entry in sensors_to_show: + title = entry.get("title") + sensors = entry.get("sensors") # List the sensors that go into one row - if isinstance(s, list): - row_sensors: list["Sensor"] = s # noqa F821 - else: - row_sensors: list["Sensor"] = [s] # noqa F821 + row_sensors: list["Sensor"] = sensors # noqa F821 + + # Set up field definition for sensor descriptions + sensor_field_definition = FIELD_DEFINITIONS["sensor_description"].copy() + sensor_field_definition["scale"] = dict( + domain=[sensor.to_dict()["description"] for sensor in row_sensors] + ) # Derive the unit that should be shown unit = determine_shared_unit(row_sensors) @@ -490,6 +573,7 @@ def chart_for_multiple_sensors( event_start_field_definition, event_value_field_definition, sensor_field_definition, + combine_legend=combine_legend, ) ] @@ -519,14 +603,7 @@ def chart_for_multiple_sensors( # Layer the lines, rectangles and circles within one row, and filter by which sensors are represented in the row sensor_specs = { - "title": join_words_into_a_list( - [ - f"{capitalize(sensor.name)}" - for sensor in row_sensors - # the sensor type is already shown as the y-axis title (avoid redundant info) - if sensor.name != sensor.sensor_type - ] - ), + "title": f"{capitalize(title)}" if title else None, "transform": [ { "filter": { @@ -550,14 +627,15 @@ def chart_for_multiple_sensors( "as": "source_name_and_id", }, ], - spacing=100, - bounds="flush", ) chart_specs["config"] = { "view": {"continuousWidth": 800, "continuousHeight": 150}, "autosize": {"type": "fit-x", "contains": "padding"}, } - chart_specs["resolve"] = {"scale": {"x": "shared"}} + if combine_legend is True: + chart_specs["resolve"] = {"scale": {"x": "shared"}} + else: + chart_specs["resolve"] = {"scale": {"color": "independent"}} for k, v in override_chart_specs.items(): chart_specs[k] = v return chart_specs @@ -596,6 +674,7 @@ def create_line_layer( event_start_field_definition: dict, event_value_field_definition: dict, sensor_field_definition: dict, + combine_legend: bool, ): event_resolutions = list(set([sensor.event_resolution for sensor in sensors])) assert all(res == timedelta(0) for res in event_resolutions) or all( @@ -605,15 +684,26 @@ def create_line_layer( line_layer = { "mark": { "type": "line", - "interpolate": "step-after" - if event_resolution != timedelta(0) - else "linear", + "interpolate": ( + "step-after" if event_resolution != timedelta(0) else "linear" + ), "clip": True, }, "encoding": { "x": event_start_field_definition, "y": event_value_field_definition, - "color": sensor_field_definition, + "color": ( + sensor_field_definition + if combine_legend + else { + **sensor_field_definition, + "legend": { + "orient": "right", + "columns": 1, + "direction": "vertical", + }, + } + ), "strokeDash": { "scale": { # Distinguish forecasters and schedulers by line stroke @@ -628,6 +718,9 @@ def create_line_layer( }, "detail": [FIELD_DEFINITIONS["source"]], }, + "selection": { + "scroll": {"type": "interval", "bind": "scales", "encodings": ["x"]} + }, } return line_layer diff --git a/flexmeasures/data/models/data_sources.py b/flexmeasures/data/models/data_sources.py index 21f12621d..2fb31bc66 100644 --- a/flexmeasures/data/models/data_sources.py +++ b/flexmeasures/data/models/data_sources.py @@ -344,11 +344,17 @@ def to_dict(self) -> dict: model_incl_version = self.model if self.model else "" if self.model and self.version: model_incl_version += f" (v{self.version})" + if "forecast" in self.type.lower(): + _type = "forecaster" # e.g. 'forecaster' or 'forecasting script' + elif "schedul" in self.type.lower(): # e.g. 'scheduler' or 'scheduling script' + _type = "scheduler" + else: + _type = "other" return dict( id=self.id, name=self.name, model=model_incl_version, - type=self.type if self.type in ("forecaster", "scheduler") else "other", + type=_type, description=self.description, ) diff --git a/flexmeasures/data/models/forecasting/model_spec_factory.py b/flexmeasures/data/models/forecasting/model_spec_factory.py index a2b19863e..bce0b259f 100644 --- a/flexmeasures/data/models/forecasting/model_spec_factory.py +++ b/flexmeasures/data/models/forecasting/model_spec_factory.py @@ -90,7 +90,8 @@ def _load_series(self) -> pd.Series: def check_data(self, df: pd.DataFrame): """Raise error if data is empty or contains nan values. - Here, other than in load_series, we can show the query, which is quite helpful.""" + Here, other than in load_series, we can show the query, which is quite helpful. + """ if df.empty: raise MissingData( "No values found in database for the requested %s data. It's no use to continue I'm afraid." @@ -120,8 +121,9 @@ def create_initial_model_specs( # noqa: C901 transform_to_normal: bool = True, use_regressors: bool = True, # If false, do not create regressor specs use_periodicity: bool = True, # If false, do not create lags given the asset's periodicity - custom_model_params: dict - | None = None, # overwrite model params, most useful for tests or experiments + custom_model_params: ( + dict | None + ) = None, # overwrite model params, most useful for tests or experiments time_series_class: type | None = TimedBelief, ) -> ModelSpecs: """ @@ -221,9 +223,9 @@ def _parameterise_forecasting_by_asset_and_asset_type( params["resolution"] = sensor.event_resolution if transform_to_normal: - params[ - "outcome_var_transformation" - ] = get_normalization_transformation_from_sensor_attributes(sensor) + params["outcome_var_transformation"] = ( + get_normalization_transformation_from_sensor_attributes(sensor) + ) return params diff --git a/flexmeasures/data/models/forecasting/model_specs/naive.py b/flexmeasures/data/models/forecasting/model_specs/naive.py index 3bc285128..e572cb408 100644 --- a/flexmeasures/data/models/forecasting/model_specs/naive.py +++ b/flexmeasures/data/models/forecasting/model_specs/naive.py @@ -24,7 +24,8 @@ class Naive(OLS): """Naive prediction model for a single input feature that simply throws back the given feature. - Under the hood, it uses linear regression by ordinary least squares, trained with points (0,0) and (1,1).""" + Under the hood, it uses linear regression by ordinary least squares, trained with points (0,0) and (1,1). + """ def __init__(self, *args, **kwargs): super().__init__([0, 1], [0, 1]) diff --git a/flexmeasures/data/models/generic_assets.py b/flexmeasures/data/models/generic_assets.py index 3ba9330d4..4373a2d9e 100644 --- a/flexmeasures/data/models/generic_assets.py +++ b/flexmeasures/data/models/generic_assets.py @@ -148,9 +148,11 @@ def __acl__(self): """ return { "create-children": f"account:{self.account_id}", - "read": self.owner.__acl__()["read"] - if self.account_id is not None - else EVERY_LOGGED_IN_USER, + "read": ( + self.owner.__acl__()["read"] + if self.account_id is not None + else EVERY_LOGGED_IN_USER + ), "update": f"account:{self.account_id}", "delete": (f"account:{self.account_id}", "role:account-admin"), } @@ -347,13 +349,9 @@ def search_annotations( self, annotations_after: datetime | None = None, annotations_before: datetime | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, annotation_type: str = None, include_account_annotations: bool = False, as_frame: bool = False, @@ -390,13 +388,9 @@ def count_annotations( annotations_after: datetime | None = None, annotation_ends_before: datetime | None = None, # deprecated annotations_before: datetime | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, annotation_type: str = None, ) -> int: """Count the number of annotations assigned to this asset.""" @@ -435,13 +429,10 @@ def chart( event_ends_before: datetime | None = None, beliefs_after: datetime | None = None, beliefs_before: datetime | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + combine_legend: bool = True, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, include_data: bool = False, dataset_name: str | None = None, resolution: str | timedelta | None = None, @@ -454,6 +445,7 @@ def chart( :param event_ends_before: only return beliefs about events that end before this datetime (inclusive) :param beliefs_after: only return beliefs formed after this datetime (inclusive) :param beliefs_before: only return beliefs formed before this datetime (inclusive) + :param combine_legend: show a combined legend of all plots below the chart :param source: search only beliefs by this source (pass the DataSource, or its name or id) or list of sources :param include_data: if True, include data in the chart, or if False, exclude data :param dataset_name: optionally name the dataset used in the chart (the default name is sensor_) @@ -475,6 +467,7 @@ def chart( chart_type, sensors_to_show=self.sensors_to_show, dataset_name=dataset_name, + combine_legend=combine_legend, **kwargs, ) @@ -507,13 +500,9 @@ def search_beliefs( beliefs_before: datetime | None = None, horizons_at_least: timedelta | None = None, horizons_at_most: timedelta | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, most_recent_beliefs_only: bool = True, most_recent_events_only: bool = False, as_json: bool = False, @@ -569,13 +558,17 @@ def search_beliefs( bdf["belief_horizon"] = bdf.belief_horizons.to_numpy() df = simplify_index( bdf, - index_levels_to_columns=["source"] - if most_recent_beliefs_only - else ["belief_time", "source"], + index_levels_to_columns=( + ["source"] + if most_recent_beliefs_only + else ["belief_time", "source"] + ), ).set_index( - ["source"] - if most_recent_beliefs_only - else ["belief_time", "source"], + ( + ["source"] + if most_recent_beliefs_only + else ["belief_time", "source"] + ), append=True, ) df["sensor"] = sensor # or some JSONifiable representation @@ -585,45 +578,80 @@ def search_beliefs( else: df = simplify_index( BeliefsDataFrame(), - index_levels_to_columns=["source"] - if most_recent_beliefs_only - else ["belief_time", "source"], + index_levels_to_columns=( + ["source"] + if most_recent_beliefs_only + else ["belief_time", "source"] + ), ).set_index( - ["source"] - if most_recent_beliefs_only - else ["belief_time", "source"], + ( + ["source"] + if most_recent_beliefs_only + else ["belief_time", "source"] + ), append=True, ) df["sensor"] = {} # ensure the same columns as a non-empty frame df = df.reset_index() - df["source"] = df["source"].apply(lambda x: x.to_dict()) - df["sensor"] = df["sensor"].apply(lambda x: x.to_dict()) + + try: + df["source"] = df["source"].apply(lambda x: x.to_dict()) + df["sensor"] = df["sensor"].apply(lambda x: x.to_dict()) + except Exception as e: + print(str(e)) + print(df["sensor"]) + print(df["source"]) + df["source"] = df["source"].ffill().apply(lambda x: x.to_dict()) + df["sensor"] = df["sensor"].ffill().apply(lambda x: x.to_dict()) + return df.to_json(orient="records") return bdf_dict @property - def sensors_to_show(self) -> list["Sensor" | list["Sensor"]]: # noqa F821 - """Sensors to show, as defined by the sensors_to_show attribute. + def sensors_to_show( + self, + ) -> list[dict[str, "Sensor"]]: # noqa F821 + """ + Sensors to show, as defined by the sensors_to_show attribute. + + Sensors to show are defined as a list of sensor IDs, which are set by the "sensors_to_show" field in the asset's "attributes" column. + Valid sensors either belong to the asset itself, to other assets in the same account, or to public assets. + In play mode, sensors from different accounts can be added. - Sensors to show are defined as a list of sensor ids, which - is set by the "sensors_to_show" field of the asset's "attributes" column. - Valid sensors either belong to the asset itself, to other assets in the same account, - or to public assets. In play mode, sensors from different accounts can be added. + Sensor IDs can be nested to denote that sensors should be 'shown together', for example, layered rather than vertically concatenated. + Additionally, each row of sensors can be accompanied by a title. + If no title is provided, `"title": None` will be assigned in the returned dictionary. - Sensor ids can be nested to denote that sensors should be 'shown together', - for example, layered rather than vertically concatenated. - How to interpret 'shown together' is technically left up to the function returning chart specs, - as are any restrictions regarding what sensors can be shown together, such as: - - whether they should share the same unit - - whether they should share the same name - - whether they should belong to different assets + How to interpret 'shown together' is technically left up to the function returning chart specifications, as are any restrictions regarding which sensors can be shown together, such as: + - Whether they should share the same unit + - Whether they should share the same name + - Whether they should belong to different assets - For example, this denotes showing sensors 42 and 44 together: + For example, this input denotes showing sensors 42 and 44 together: sensors_to_show = [40, 35, 41, [42, 44], 43, 45] - In case the field is missing, defaults to two of the asset's sensors, - which will be shown together (e.g. sharing the same y-axis) in case they share the same unit. + And this input denotes showing sensors 42 and 44 together with a custom title: + + sensors_to_show = [ + {"title": "Title 1", "sensor": 40}, + {"title": "Title 2", "sensors": [41, 42]}, + [43, 44], 45, 46 + ] + + In both cases, the returned format will contain sensor objects mapped to their respective sensor IDs, as follows: + + [ + {"title": "Title 1", "sensor": }, + {"title": "Title 2", "sensors": [, ]}, + {"title": None, "sensors": [, ]}, + {"title": None, "sensor": }, + {"title": None, "sensor": } + ] + + In case the `sensors_to_show` field is missing, it defaults to two of the asset's sensors. These will be shown together (e.g., sharing the same y-axis) if they share the same unit; otherwise, they will be shown separately. + + Sensors are validated to ensure they are accessible by the user. If certain sensors are inaccessible, they will be excluded from the result, and a warning will be logged. The function only returns sensors that the user has permission to view. """ if not self.has_attribute("sensors_to_show"): sensors_to_show = self.sensors[:2] @@ -632,9 +660,22 @@ def sensors_to_show(self) -> list["Sensor" | list["Sensor"]]: # noqa F821 and sensors_to_show[0].unit == sensors_to_show[1].unit ): # Sensors are shown together (e.g. they can share the same y-axis) - return [sensors_to_show] + return [{"title": None, "sensors": sensors_to_show}] # Otherwise, show separately - return sensors_to_show + return [{"title": None, "sensors": [sensor]} for sensor in sensors_to_show] + + sensor_ids_to_show = self.get_attribute("sensors_to_show") + # Import the schema for validation + from flexmeasures.data.schemas.generic_assets import SensorsToShowSchema + + sensors_to_show_schema = SensorsToShowSchema() + + # Deserialize the sensor_ids_to_show using SensorsToShowSchema + standardized_sensors_to_show = sensors_to_show_schema.deserialize( + sensor_ids_to_show + ) + + sensor_id_allowlist = SensorsToShowSchema.flatten(standardized_sensors_to_show) # Only allow showing sensors from assets owned by the user's organization, # except in play mode, where any sensor may be shown @@ -646,13 +687,12 @@ def sensors_to_show(self) -> list["Sensor" | list["Sensor"]]: # noqa F821 from flexmeasures.data.services.sensors import get_sensors - sensor_ids_to_show = self.get_attribute("sensors_to_show") accessible_sensor_map = { sensor.id: sensor for sensor in get_sensors( account=accounts, include_public_assets=True, - sensor_id_allowlist=flatten_unique(sensor_ids_to_show), + sensor_id_allowlist=sensor_id_allowlist, ) } @@ -660,24 +700,21 @@ def sensors_to_show(self) -> list["Sensor" | list["Sensor"]]: # noqa F821 sensors_to_show = [] missed_sensor_ids = [] - # we make sure to build in the order given by the sensors_to_show attribute, and with the same nesting - for s in sensor_ids_to_show: - if isinstance(s, list): - inaccessible = [sid for sid in s if sid not in accessible_sensor_map] - missed_sensor_ids.extend(inaccessible) - if len(inaccessible) < len(s): - sensors_to_show.append( - [ - accessible_sensor_map[sensor_id] - for sensor_id in s - if sensor_id in accessible_sensor_map - ] - ) - else: - if s not in accessible_sensor_map: - missed_sensor_ids.append(s) - else: - sensors_to_show.append(accessible_sensor_map[s]) + for entry in standardized_sensors_to_show: + + title = entry.get("title") + sensors = entry.get("sensors") + + accessible_sensors = [ + accessible_sensor_map.get(sid) + for sid in sensors + if sid in accessible_sensor_map + ] + inaccessible = [sid for sid in sensors if sid not in accessible_sensor_map] + missed_sensor_ids.extend(inaccessible) + if accessible_sensors: + sensors_to_show.append({"title": title, "sensors": accessible_sensors}) + if missed_sensor_ids: current_app.logger.warning( f"Cannot include sensor(s) {missed_sensor_ids} in sensors_to_show on asset {self}, as it is not accessible to user {current_user}." diff --git a/flexmeasures/data/models/legacy_migration_utils.py b/flexmeasures/data/models/legacy_migration_utils.py index 926155997..e0d355726 100644 --- a/flexmeasures/data/models/legacy_migration_utils.py +++ b/flexmeasures/data/models/legacy_migration_utils.py @@ -2,6 +2,7 @@ This module is part of our data model migration (see https://github.com/SeitaBV/flexmeasures/projects/9). It will become obsolete when Assets, Markets and WeatherSensors can no longer be initialized. """ + from __future__ import annotations from datetime import datetime @@ -26,9 +27,11 @@ def copy_old_sensor_attributes( a: getattr(old_sensor_type, a) for a in old_sensor_type_attributes } new_model_attributes_from_old_sensor = { - a: getattr(old_sensor, a) - if not isinstance(getattr(old_sensor, a), datetime) - else getattr(old_sensor, a).isoformat() + a: ( + getattr(old_sensor, a) + if not isinstance(getattr(old_sensor, a), datetime) + else getattr(old_sensor, a).isoformat() + ) for a in old_sensor_attributes } return dict( diff --git a/flexmeasures/data/models/parsing_utils.py b/flexmeasures/data/models/parsing_utils.py index f36914559..df58f70f0 100644 --- a/flexmeasures/data/models/parsing_utils.py +++ b/flexmeasures/data/models/parsing_utils.py @@ -10,13 +10,15 @@ def parse_source_arg( - source: DataSource - | int - | str - | Sequence[DataSource] - | Sequence[int] - | Sequence[str] - | None, + source: ( + DataSource + | int + | str + | Sequence[DataSource] + | Sequence[int] + | Sequence[str] + | None + ), ) -> list[DataSource] | None: """Parse the "source" argument by looking up DataSources corresponding to any given ids or names. diff --git a/flexmeasures/data/models/planning/linear_optimization.py b/flexmeasures/data/models/planning/linear_optimization.py index 783f26d91..008ce2242 100644 --- a/flexmeasures/data/models/planning/linear_optimization.py +++ b/flexmeasures/data/models/planning/linear_optimization.py @@ -37,10 +37,10 @@ def device_scheduler( # noqa C901 device_upwards_price: list[pd.Series], device_future_reward: list[float], initial_stock: float = 0, - ems_flow_relaxed: bool = False, - device_stock_relaxed: bool = False, - ems_flow_relaxation_cost: float = 1e9, # TODO: compute this value based on input data - stock_relaxation_cost: float = 20000, + # TODO: compute the ems_flow_relaxation_cost value based on input data + ems_flow_relaxation_cost: float | None = None, + stock_relaxation_cost: float | None = None, + ems_flow_soft_derivative_cost: float | None = None, ) -> tuple[list[pd.Series], float, SolverResults, ConcreteModel]: """This generic device scheduler is able to handle an EMS with multiple devices, with various types of constraints on the EMS level and on the device level, @@ -63,6 +63,8 @@ def device_scheduler( # noqa C901 EMS constraints are on an EMS level. Handled constraints (listed by column name): derivative max: maximum flow derivative min: minimum flow + derivative soft max: aggregate power that range from derivaive soft max to derivative max are penalized by ems_flow_soft_derivative_cost + derivative soft min: aggregate power that range from derivaive soft min to derivative min are penalized by ems_flow_soft_derivative_cost Commitments are on an EMS level. Parameter explanations: commitment_quantities: amounts of flow specified in commitments (both previously ordered and newly requested) - e.g. in MW or boxes/h @@ -77,6 +79,10 @@ def device_scheduler( # noqa C901 DataFrame. Later we could pass in a MultiIndex DataFrame directly. """ + is_device_stock_relaxed = stock_relaxation_cost is not None + is_ems_flow_relaxed = ems_flow_relaxation_cost is not None + is_ems_flow_soft_derivative_active = ems_flow_soft_derivative_cost is not None + model = ConcreteModel() # If the EMS has no devices, don't bother @@ -218,6 +224,34 @@ def ems_derivative_min_select(m, j): else: return v + def ems_derivative_soft_min_select(m, j): + v = ems_constraints["derivative min"].iloc[j] + + if ( + "derivative soft min" in ems_constraints.columns + and is_ems_flow_soft_derivative_active + ): + v = np.nanmax([ems_constraints["derivative soft min"].iloc[j], v]) + + if np.isnan(v): + return -infinity + else: + return v + + def ems_derivative_soft_max_select(m, j): + v = ems_constraints["derivative max"].iloc[j] + + if ( + "derivative soft max" in ems_constraints.columns + and is_ems_flow_soft_derivative_active + ): + v = np.nanmin([ems_constraints["derivative soft max"].iloc[j], v]) + + if np.isnan(v): + return infinity + else: + return v + def device_efficiency(m, d, j): """Assume perfect efficiency if no efficiency information is available.""" try: @@ -273,6 +307,15 @@ def device_stock_delta(m, d, j): ) model.ems_derivative_max = Param(model.j, initialize=ems_derivative_max_select) model.ems_derivative_min = Param(model.j, initialize=ems_derivative_min_select) + + if is_ems_flow_soft_derivative_active: + model.ems_derivative_soft_min = Param( + model.j, initialize=ems_derivative_soft_min_select + ) + model.ems_derivative_soft_max = Param( + model.j, initialize=ems_derivative_soft_max_select + ) + model.device_efficiency = Param(model.d, model.j, initialize=device_efficiency) model.device_derivative_down_efficiency = Param( model.d, model.j, initialize=device_derivative_down_efficiency @@ -305,6 +348,10 @@ def device_stock_delta(m, d, j): model.ems_power_slack_upper = Var(domain=NonNegativeReals, initialize=0) model.ems_power_slack_lower = Var(domain=NonNegativeReals, initialize=0) + if is_ems_flow_soft_derivative_active: + model.ems_power_margin_lower = Var(domain=NonNegativeReals, initialize=0) + model.ems_power_margin_upper = Var(domain=NonNegativeReals, initialize=0) + def _get_stock_change(m, d, j): """Determine final stock change of device d until time j.""" stock_changes = [ @@ -330,7 +377,7 @@ def device_lower_bounds(m, d, j): and apply storage efficiencies to stock levels from one datetime to the next.""" final_stock_change = _get_stock_change(m, d, j) stock_slack = 0 - if device_stock_relaxed: + if is_device_stock_relaxed: stock_slack += m.device_stock_slack_lower[d, j] return ( m.device_min[d, j] - stock_slack, @@ -344,7 +391,7 @@ def device_upper_bounds(m, d, j): and apply storage efficiencies to stock levels from one datetime to the next.""" final_stock_change = _get_stock_change(m, d, j) stock_slack = 0 - if device_stock_relaxed: + if is_device_stock_relaxed: stock_slack += m.device_stock_slack_upper[d, j] return ( None, @@ -385,7 +432,7 @@ def device_down_derivative_sign(m, d, j): def ems_derivative_lower_bound(m, j): - if ems_flow_relaxed: + if is_ems_flow_relaxed: return ( m.ems_derivative_min[j], sum(m.ems_power[:, j]) + m.ems_power_slack_lower, @@ -394,8 +441,36 @@ def ems_derivative_lower_bound(m, j): else: return m.ems_derivative_min[j], sum(m.ems_power[:, j]), None + def ems_derivative_soft_margin_lower(m, j): + return ( + 0, + m.ems_power_margin_lower, + m.ems_derivative_soft_min[j] - m.ems_derivative_min[j], + ) + + def ems_derivative_soft_margin_upper(m, j): + return ( + 0, + m.ems_power_margin_upper, + m.ems_derivative_max[j] - m.ems_derivative_soft_max[j], + ) + + def ems_derivative_soft_lower_bound(m, j): + return ( + m.ems_derivative_soft_min[j] - m.ems_power_margin_lower, + sum(m.ems_power[:, j]), + None, + ) + + def ems_derivative_soft_upper_bound(m, j): + return ( + None, + sum(m.ems_power[:, j]), + m.ems_derivative_soft_max[j] + m.ems_power_margin_upper, + ) + def ems_derivative_upper_bound(m, j): - if ems_flow_relaxed: + if is_ems_flow_relaxed: return ( None, sum(m.ems_power[:, j]) - m.ems_power_slack_upper, @@ -444,9 +519,24 @@ def device_derivative_equalities(m, d, j): model.device_power_down_sign = Constraint( model.d, model.j, rule=device_down_derivative_sign ) - model.ems_power_upper_bound = Constraint(model.j, rule=ems_derivative_upper_bound) + model.ems_power_upper_bounds = Constraint(model.j, rule=ems_derivative_upper_bound) model.ems_power_lower_bounds = Constraint(model.j, rule=ems_derivative_lower_bound) + if is_ems_flow_soft_derivative_active: + model.ems_power_soft_upper_bounds = Constraint( + model.j, rule=ems_derivative_soft_upper_bound + ) + model.ems_power_soft_lower_bounds = Constraint( + model.j, rule=ems_derivative_soft_lower_bound + ) + + model.ems_derivative_soft_margin_lower = Constraint( + model.j, rule=ems_derivative_soft_margin_lower + ) + model.ems_derivative_soft_margin_upper = Constraint( + model.j, rule=ems_derivative_soft_margin_upper + ) + model.ems_power_commitment_equalities = Constraint( model.j, rule=ems_flow_commitment_equalities ) @@ -471,15 +561,20 @@ def cost_function(m): final_stock_change = _get_stock_change(m, d, m.j[-1]) costs -= final_stock_change * m.device_future_rewards_price[d] - if ems_flow_relaxed: + if is_ems_flow_relaxed: costs += m.ems_power_slack_upper * ems_flow_relaxation_cost costs += m.ems_power_slack_lower * ems_flow_relaxation_cost - if device_stock_relaxed: + + if is_device_stock_relaxed: for j in m.j: for d in m.d: costs += m.device_stock_slack_upper[d, j] * stock_relaxation_cost costs += m.device_stock_slack_lower[d, j] * stock_relaxation_cost + if is_ems_flow_soft_derivative_active: + costs += m.ems_power_margin_lower * ems_flow_soft_derivative_cost + costs += m.ems_power_margin_upper * ems_flow_soft_derivative_cost + return costs model.costs = Objective(rule=cost_function, sense=minimize) diff --git a/flexmeasures/data/models/planning/process.py b/flexmeasures/data/models/planning/process.py index cd97a0b3d..e21ece15b 100644 --- a/flexmeasures/data/models/planning/process.py +++ b/flexmeasures/data/models/planning/process.py @@ -32,7 +32,7 @@ def compute(self) -> pd.Series | None: This scheduler supports three types of `process_types`: - INFLEXIBLE: this process needs to be scheduled as soon as possible. - - BREAKABLE: this process can be divisible in smaller consumption periods. + - BREAKABLE: this process can be broken up into smaller segments with some idle time in between. - SHIFTABLE: this process can start at any time within the specified time window. The resulting schedule provides the power flow at each time period. diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index a08f94a3c..ca40d911b 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -27,7 +27,7 @@ from flexmeasures.data.schemas.scheduling import FlexContextSchema from flexmeasures.utils.time_utils import get_max_planning_horizon from flexmeasures.utils.coding_utils import deprecated -from flexmeasures.utils.unit_utils import ur +from flexmeasures.utils.unit_utils import ur, convert_units class MetaStorageScheduler(Scheduler): @@ -99,6 +99,8 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 self.flex_context.get("curtailable_device_sensors", []) # or self.sensor.generic_asset.get_inflexible_device_sensors() # todo: support setting as asset attribute? ) + consumption_price = self.flex_context.get("consumption_price") + production_price = self.flex_context.get("production_price") inflexible_device_sensors = ( self.flex_context.get("inflexible_device_sensors") or self.sensor.generic_asset.get_inflexible_device_sensors() @@ -121,7 +123,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 power_capacity_in_mw = ur.Quantity(f"{power_capacity_in_mw} MW") power_capacity_in_mw = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=power_capacity_in_mw, + variable_quantity=power_capacity_in_mw, actuator=sensor, unit="MW", query_window=(start, end), @@ -130,22 +132,54 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) # Check for known prices or price forecasts, trimming planning window accordingly - up_deviation_prices, (start, end) = get_prices( - (start, end), - resolution, - beliefs_before=belief_time, - price_sensor=consumption_price_sensor, - sensor=sensor, - allow_trimmed_query_window=False, - ) - down_deviation_prices, (start, end) = get_prices( - (start, end), - resolution, - beliefs_before=belief_time, - price_sensor=production_price_sensor, - sensor=sensor, - allow_trimmed_query_window=False, - ) + if consumption_price is not None: + up_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=consumption_price, + actuator=sensor, + unit=( + consumption_price.unit + if isinstance(consumption_price, Sensor) + else str(consumption_price.units) + ), + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fallback_attribute="market_id", + fill_sides=True, + ).to_frame() + else: + up_deviation_prices, (start, end) = get_prices( + (start, end), + resolution, + beliefs_before=belief_time, + price_sensor=consumption_price_sensor, + sensor=sensor, + allow_trimmed_query_window=False, + ) + if production_price is not None: + down_deviation_prices = get_continuous_series_sensor_or_quantity( + variable_quantity=production_price, + actuator=sensor, + unit=( + production_price.unit + if isinstance(production_price, Sensor) + else str(production_price.units) + ), + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fallback_attribute="market_id", + fill_sides=True, + ).to_frame() + else: + down_deviation_prices, (start, end) = get_prices( + (start, end), + resolution, + beliefs_before=belief_time, + price_sensor=production_price_sensor, + sensor=sensor, + allow_trimmed_query_window=False, + ) start = pd.Timestamp(start).tz_convert("UTC") end = pd.Timestamp(end).tz_convert("UTC") @@ -192,7 +226,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 device_future_reward = [0 for d in range(D)] # Add a tiny expected future reward for having more state of charge at the end of the planning window - device_future_reward[0] = 10**-3 + device_future_reward[0] = self.flex_model.get("future_reward", 0.0) # Set up device constraints: only one scheduled flexible device for this EMS (at index 0), plus the forecasted inflexible devices (at indices 1 to n). device_constraints = [ @@ -251,7 +285,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # fetch SOC constraints from sensors if isinstance(soc_targets, Sensor): soc_targets = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=soc_targets, + variable_quantity=soc_targets, actuator=sensor, unit="MWh", query_window=(start, end), @@ -262,7 +296,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) if isinstance(soc_minima, Sensor): soc_minima = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=soc_minima, + variable_quantity=soc_minima, actuator=sensor, unit="MWh", query_window=(start, end), @@ -273,7 +307,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) if isinstance(soc_maxima, Sensor): soc_maxima = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=soc_maxima, + variable_quantity=soc_maxima, actuator=sensor, unit="MWh", query_window=(start, end), @@ -304,29 +338,29 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 device_constraints[0]["derivative min"] = ( -1 ) * get_continuous_series_sensor_or_quantity( - quantity_or_sensor=production_capacity, + variable_quantity=production_capacity, actuator=sensor, unit="MW", query_window=(start, end), resolution=resolution, beliefs_before=belief_time, - fallback_attribute="production_capacity", + fallback_attribute="production-capacity", max_value=power_capacity_in_mw, ) if sensor.get_attribute("is_strictly_non_negative"): device_constraints[0]["derivative max"] = 0 else: - device_constraints[0][ - "derivative max" - ] = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=consumption_capacity, - actuator=sensor, - unit="MW", - query_window=(start, end), - resolution=resolution, - beliefs_before=belief_time, - fallback_attribute="consumption_capacity", - max_value=power_capacity_in_mw, + device_constraints[0]["derivative max"] = ( + get_continuous_series_sensor_or_quantity( + variable_quantity=consumption_capacity, + actuator=sensor, + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + fallback_attribute="consumption-capacity", + max_value=power_capacity_in_mw, + ) ) soc_gain = self.flex_model.get("soc_gain", []) @@ -337,7 +371,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 for is_usage, soc_delta in zip([False, True], [soc_gain, soc_usage]): for component in soc_delta: stock_delta_series = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=component, + variable_quantity=component, actuator=sensor, unit="MW", query_window=(start, end), @@ -363,7 +397,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 # Apply round-trip efficiency evenly to charging and discharging charging_efficiency = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=self.flex_model.get("charging_efficiency"), + variable_quantity=self.flex_model.get("charging_efficiency"), actuator=sensor, unit="dimensionless", query_window=(start, end), @@ -372,7 +406,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 fallback_attribute="charging-efficiency", ).fillna(1) discharging_efficiency = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=self.flex_model.get("discharging_efficiency"), + variable_quantity=self.flex_model.get("discharging_efficiency"), actuator=sensor, unit="dimensionless", query_window=(start, end), @@ -401,7 +435,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ): device_constraints[0]["efficiency"] = ( get_continuous_series_sensor_or_quantity( - quantity_or_sensor=storage_efficiency, + variable_quantity=storage_efficiency, actuator=sensor, unit="dimensionless", query_window=(start, end), @@ -440,7 +474,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) ems_power_capacity_in_mw = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=self.flex_context.get("ems_power_capacity_in_mw"), + variable_quantity=self.flex_context.get("ems_power_capacity_in_mw"), actuator=sensor.generic_asset, unit="MW", query_window=(start, end), @@ -450,7 +484,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) ems_constraints["derivative max"] = get_continuous_series_sensor_or_quantity( - quantity_or_sensor=self.flex_context.get("ems_consumption_capacity_in_mw"), + variable_quantity=self.flex_context.get("ems_consumption_capacity_in_mw"), actuator=sensor.generic_asset, unit="MW", query_window=(start, end), @@ -462,7 +496,7 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ems_constraints["derivative min"] = ( -1 ) * get_continuous_series_sensor_or_quantity( - quantity_or_sensor=self.flex_context.get("ems_production_capacity_in_mw"), + variable_quantity=self.flex_context.get("ems_production_capacity_in_mw"), actuator=sensor.generic_asset, unit="MW", query_window=(start, end), @@ -472,6 +506,33 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 max_value=ems_power_capacity_in_mw, ) + if "ems_soft_consumption_capacity_in_mw" in self.flex_context: + ems_constraints["derivative soft max"] = ( + get_continuous_series_sensor_or_quantity( + quantity_or_sensor=self.flex_context.get( + "ems_soft_consumption_capacity_in_mw" + ), + actuator=sensor.generic_asset, + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + ) + ) + if "ems_soft_production_capacity_in_mw" in self.flex_context: + ems_constraints["derivative soft min"] = ( + -1 + ) * get_continuous_series_sensor_or_quantity( + quantity_or_sensor=self.flex_context.get( + "ems_soft_production_capacity_in_mw" + ), + actuator=sensor.generic_asset, + unit="MW", + query_window=(start, end), + resolution=resolution, + beliefs_before=belief_time, + ) + return ( sensor, start, @@ -491,17 +552,9 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 def persist_flex_model(self): """Store new soc info as GenericAsset attributes""" self.sensor.generic_asset.set_attribute("soc_datetime", self.start.isoformat()) - soc_unit = self.flex_model.get("soc_unit") - if soc_unit == "kWh": - self.sensor.generic_asset.set_attribute( - "soc_in_mwh", self.flex_model["soc_at_start"] / 1000 - ) - elif soc_unit == "MWh": - self.sensor.generic_asset.set_attribute( - "soc_in_mwh", self.flex_model["soc_at_start"] - ) - else: - raise NotImplementedError(f"Unsupported SoC unit '{soc_unit}'.") + self.sensor.generic_asset.set_attribute( + "soc_in_mwh", self.flex_model["soc_at_start"] + ) def deserialize_flex_config(self): """ @@ -535,18 +588,14 @@ def deserialize_flex_config(self): ) else: self.flex_model["soc-at-start"] = 0 - # soc-unit - if "soc-unit" not in self.flex_model or self.flex_model["soc-unit"] is None: - if self.sensor.unit in ("MWh", "kWh"): - self.flex_model["soc-unit"] = self.sensor.unit - elif self.sensor.unit in ("MW", "kW"): - self.flex_model["soc-unit"] = self.sensor.unit + "h" self.ensure_soc_min_max() - # Now it's time to check if our flex configurations holds up to schemas + # Now it's time to check if our flex configuration holds up to schemas self.flex_model = StorageFlexModelSchema( - start=self.start, sensor=self.sensor + start=self.start, + sensor=self.sensor, + default_soc_unit=self.flex_model.get("soc-unit"), ).load(self.flex_model) self.flex_context = FlexContextSchema().load(self.flex_context) @@ -669,6 +718,7 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: storage_schedule = fallback_charging_policy( sensor, device_constraints[0], start, end, resolution ) + storage_schedule = convert_units(storage_schedule, "MW", sensor.unit) # Round schedule if self.round_to_decimals: @@ -726,13 +776,20 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: device_upwards_price=device_upwards_price, device_future_reward=device_future_reward, initial_stock=soc_at_start * (timedelta(hours=1) / resolution), - ems_flow_relaxed=self.flex_model.get("relaxed", False), + ems_flow_relaxation_cost=self.flex_context.get( + "ems_power_limit_relaxation_cost", None + ), + ems_flow_soft_derivative_cost=self.flex_context.get( + "ems_soft_power_limit_relaxation_cost", None + ), + stock_relaxation_cost=self.flex_model.get("soc_deviation_penalty", None), ) if scheduler_results.solver.termination_condition == "infeasible": raise InfeasibleProblemException() # Obtain the storage schedule from all device schedules within the EMS storage_schedule = ems_schedule[0] + storage_schedule = convert_units(storage_schedule, "MW", sensor.unit) # Round schedule if self.round_to_decimals: diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index 1d2bdd53d..715e221e0 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -26,7 +26,7 @@ integrate_time_series, ) from flexmeasures.tests.utils import get_test_sensor -from flexmeasures.utils.unit_utils import convert_units +from flexmeasures.utils.unit_utils import convert_units, ur TOLERANCE = 0.00001 @@ -86,11 +86,11 @@ def test_battery_solver_day_1( resolution, flex_model={"soc-at-start": soc_at_start}, flex_context={ - "inflexible-device-sensors": [ - s.id for s in add_inflexible_device_forecasts.keys() - ] - if use_inflexible_device - else [], + "inflexible-device-sensors": ( + [s.id for s in add_inflexible_device_forecasts.keys()] + if use_inflexible_device + else [] + ), "site-power-capacity": "2 MW", }, ) @@ -620,9 +620,9 @@ def test_building_solver_day_2( capacity["battery consumption headroom"] = capacity["consumption headroom"].clip( upper=battery.get_attribute("capacity_in_mw") ) - capacity[ - "schedule" - ] = schedule.values # consumption is positive, production is negative + capacity["schedule"] = ( + schedule.values + ) # consumption is positive, production is negative with pd.option_context( "display.max_rows", None, "display.max_columns", None, "display.width", 2000 ): @@ -680,6 +680,7 @@ def test_soc_bounds_timeseries(db, add_battery_assets): resolution = timedelta(hours=1) # soc parameters + soc_unit = "MWh" soc_at_start = battery.get_attribute("soc_in_mwh") soc_min = 0.5 soc_max = 4.5 @@ -703,6 +704,7 @@ def compute_schedule(flex_model): return soc_schedule flex_model = { + "soc-unit": soc_unit, "soc-at-start": soc_at_start, "soc-min": soc_min, "soc-max": soc_max, @@ -721,6 +723,7 @@ def compute_schedule(flex_model): soc_targets = [{"datetime": "2015-01-02T19:00:00+01:00", "value": 2.0}] flex_model = { + "soc-unit": soc_unit, "soc-at-start": soc_at_start, "soc-min": soc_min, "soc-max": soc_max, @@ -1022,10 +1025,14 @@ def compute_schedule(flex_model): compute_schedule(flex_model) -def get_sensors_from_db(db, battery_assets, battery_name="Test battery"): +def get_sensors_from_db( + db, battery_assets, battery_name="Test battery", power_sensor_name="power" +): # get the sensors from the database epex_da = get_test_sensor(db) - battery = battery_assets[battery_name].sensors[0] + battery = [ + s for s in battery_assets[battery_name].sensors if s.name == power_sensor_name + ][0] assert battery.get_attribute("market_id") == epex_da.id return epex_da, battery @@ -1732,10 +1739,10 @@ def test_battery_stock_delta_sensor( "gain,usage,expected_delta", [ (["1 MW"], ["1MW"], 0), # delta stock is 0 (1 MW - 1 MW) - (["0.5 MW", "0.5 MW"], [], 1), # 1 MW stock gain + (["0.5 MW", "0.5 MW"], None, 1), # 1 MW stock gain (["100 kW"], None, 0.1), # 100 MW stock gain - (None, ["100 kW"], -0.1), # 100 kW stock loss - ([], [], None), # no gain defined -> no gain or loss happens + (None, ["100 kW"], -0.1), # 100 kW stock usage + (None, None, None), # no gain/usage defined -> no gain or usage happens ], ) def test_battery_stock_delta_quantity( @@ -1744,7 +1751,7 @@ def test_battery_stock_delta_quantity( """ Test the stock gain field when a constant value is provided. - We expect a constant gain/loss to happen in every time period equal to the energy + We expect a constant gain/usage to happen in every time period equal to the energy value provided. """ _, battery = get_sensors_from_db(db, add_battery_assets) @@ -2053,3 +2060,194 @@ def compute_schedule(flex_model): # this yields the same results as with the SOC targets # because soc-maxima = soc-minima = soc-targets assert all(abs(soc[9:].values - values[:-1]) < 1e-5) + + +@pytest.mark.parametrize("unit", [None, "MWh", "kWh"]) +@pytest.mark.parametrize("soc_unit", ["kWh", "MWh"]) +@pytest.mark.parametrize("power_sensor_name", ["power", "power (kW)"]) +def test_battery_storage_different_units( + add_battery_assets, + db, + power_sensor_name, + soc_unit, + unit, +): + """ + Test scheduling a 1 MWh battery for 2h with a low -> high price transition with + different units for the soc-min, soc-max, soc-at-start and power sensor. + """ + + soc_min = ur.Quantity("100 kWh") + soc_max = ur.Quantity("1 MWh") + soc_at_start = ur.Quantity("100 kWh") + + if unit is not None: + soc_min = str(soc_min.to(unit)) + soc_max = str(soc_max.to(unit)) + soc_at_start = str(soc_at_start.to(unit)) + else: + soc_min = soc_min.to(soc_unit).magnitude + soc_max = soc_max.to(soc_unit).magnitude + soc_at_start = soc_at_start.to(soc_unit).magnitude + + epex_da, battery = get_sensors_from_db( + db, + add_battery_assets, + battery_name="Test battery", + power_sensor_name=power_sensor_name, + ) + tz = pytz.timezone("Europe/Amsterdam") + + # transition from cheap to expensive (90 -> 100) + start = tz.localize(datetime(2015, 1, 2, 14, 0, 0)) + end = tz.localize(datetime(2015, 1, 2, 16, 0, 0)) + resolution = timedelta(minutes=15) + + flex_model = { + "soc-min": soc_min, + "soc-max": soc_max, + "soc-at-start": soc_at_start, + "soc-unit": soc_unit, + "roundtrip-efficiency": 1, + "storage-efficiency": 1, + "power-capacity": "1 MW", + } + + scheduler: Scheduler = StorageScheduler( + battery, + start, + end, + resolution, + flex_model=flex_model, + flex_context={ + "site-power-capacity": "1 MW", + }, + ) + schedule = scheduler.compute() + + if power_sensor_name == "power (kW)": + schedule /= 1000 + + # Check if constraints were met + if isinstance(soc_at_start, str): + soc_at_start = ur.Quantity(soc_at_start).to("MWh").magnitude + elif isinstance(soc_at_start, float) or isinstance(soc_at_start, int): + soc_at_start = soc_at_start * convert_units(1, soc_unit, "MWh") + check_constraints(battery, schedule, soc_at_start) + + # charge fully in the cheap price period (100 kWh -> 1000kWh) + assert schedule[:4].sum() * 0.25 == 0.9 + + # discharge fully in the expensive price period (1000 kWh -> 100 kWh) + assert schedule[4:].sum() * 0.25 == -0.9 + + +@pytest.mark.parametrize( + "ts_field, ts_specs", + [ + # The battery only has time to charge up to 950 kWh halfway + ( + "power-capacity", + [ + { + "start": "2015-01-02T14:00+01", + "end": "2015-01-02T16:00+01", + "value": "850 kW", + } + ], + ), + # Same, but the event time is specified with a duration instead of an end time + ( + "power-capacity", + [ + { + "start": "2015-01-02T14:00+01", + "duration": "PT2H", + "value": "850 kW", + } + ], + ), + # Can only charge up to 950 kWh halfway + ( + "soc-maxima", + [ + { + "datetime": "2015-01-02T15:00+01", + "value": "950 kWh", + } + ], + ), + # Must end up at a minimum of 200 kWh, for which it is cheapest to charge to 950 and then to discharge to 200 + ( + "soc-minima", + [ + { + "datetime": "2015-01-02T16:00+01", + "value": "200 kWh", + } + ], + ), + ], +) +def test_battery_storage_with_time_series_in_flex_model( + add_battery_assets, + db, + ts_field, + ts_specs, +): + """ + Test scheduling a 1 MWh battery for 2h with a low -> high price transition with + a time series used for the various flex-model fields. + """ + + soc_min = "100 kWh" + soc_max = "1 MWh" + soc_at_start = "100 kWh" + + epex_da, battery = get_sensors_from_db( + db, + add_battery_assets, + battery_name="Test battery", + power_sensor_name="power", + ) + tz = pytz.timezone("Europe/Amsterdam") + + # transition from cheap to expensive (90 -> 100) + start = tz.localize(datetime(2015, 1, 2, 14, 0, 0)) + end = tz.localize(datetime(2015, 1, 2, 16, 0, 0)) + resolution = timedelta(minutes=15) + + flex_model = { + "soc-min": soc_min, + "soc-max": soc_max, + "soc-at-start": soc_at_start, + "roundtrip-efficiency": 1, + "storage-efficiency": 1, + "power-capacity": "1 MW", + } + flex_model[ts_field] = ts_specs + + scheduler: Scheduler = StorageScheduler( + battery, + start, + end, + resolution, + flex_model=flex_model, + flex_context={ + "site-power-capacity": "1 MW", + }, + ) + schedule = scheduler.compute() + + # Check if constraints were met + soc_at_start = ur.Quantity(soc_at_start).to("MWh").magnitude + check_constraints(battery, schedule, soc_at_start) + + # charge 850 kWh in the cheap price period (100 kWh -> 950kWh) + assert schedule[:4].sum() * 0.25 == pytest.approx(0.85) + + # discharge fully or to what's needed in the expensive price period (950 kWh -> 100 or 200 kWh) + if ts_field == "soc-minima": + assert schedule[4:].sum() * 0.25 == pytest.approx(-0.75) + else: + assert schedule[4:].sum() * 0.25 == pytest.approx(-0.85) diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index f0ae3f3c4..ef3654bc1 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -18,6 +18,7 @@ from flexmeasures import Asset from flexmeasures.data.queries.utils import simplify_index +from flexmeasures.utils.flexmeasures_inflection import capitalize, pluralize from flexmeasures.utils.unit_utils import ur, convert_units from pint.errors import UndefinedUnitError, DimensionalityError @@ -136,41 +137,78 @@ def get_prices( raise UnknownPricesException( f"Prices unknown for planning window. (sensor {price_sensor.id})" ) - elif ( - nan_prices.any() - or pd.Timestamp(price_df.index[0]).tz_convert("UTC") + else: + price_df = extend_to_edges( + df=price_df, + query_window=query_window, + resolution=resolution, + sensor=price_sensor, + allow_trimmed_query_window=allow_trimmed_query_window, + ) + return price_df, query_window + + +def extend_to_edges( + df: pd.DataFrame | pd.Series, + query_window: tuple[datetime, datetime], + resolution: timedelta, + kind_of_values: str = "price", + sensor: Sensor = None, + allow_trimmed_query_window: bool = False, +): + """Values are extended to the edges of the query window. + + - The first available value serves as a naive backcasts. + - The last available value serves as a naive forecast. + """ + if df.isnull().all(): + current_app.logger.warning("No prices in planning window. Assuming 0 prices.") + df = df.fillna(1) + + nan_values = df.isnull().values + if ( + nan_values.any() + or pd.Timestamp(df.index[0]).tz_convert("UTC") != pd.Timestamp(query_window[0]).tz_convert("UTC") - or pd.Timestamp(price_df.index[-1]).tz_convert("UTC") + resolution + or pd.Timestamp(df.index[-1]).tz_convert("UTC") + resolution != pd.Timestamp(query_window[-1]).tz_convert("UTC") ): if allow_trimmed_query_window: - first_event_start = price_df.first_valid_index() - last_event_end = price_df.last_valid_index() + resolution + first_event_start = df.first_valid_index() + last_event_end = df.last_valid_index() + resolution + sensor_info = ( + f" (sensor {sensor.id})" if sensor and hasattr(sensor, "id") else "" + ) + current_app.logger.warning( - f"Prices partially unknown for planning window (sensor {price_sensor.id}). " + f"Prices partially unknown for planning window ({sensor_info}). " f"Trimming planning window (from {query_window[0]} until {query_window[-1]}) to {first_event_start} until {last_event_end}." ) - query_window = (first_event_start, last_event_end) else: + sensor_info = ( + f" (sensor {df.sensor.id})" + if hasattr(df, "sensor") and hasattr(df.sensor, "id") + else "" + ) current_app.logger.warning( - f"Prices partially unknown for planning window (sensor {price_sensor.id}). " - f"Assuming the first price is valid from the start of the planning window ({query_window[0]}), " - f"and the last price is valid until the end of the planning window ({query_window[-1]})." + f"{capitalize(pluralize(kind_of_values))} partially unknown for planning window ({sensor_info}). " + f"Assuming the first {kind_of_values} is valid from the start of the planning window ({query_window[0]}), " + f"and the last {kind_of_values} is valid until the end of the planning window ({query_window[-1]})." ) index = initialize_index( start=query_window[0], end=query_window[1], resolution=resolution, ) - price_df = price_df.reindex(index) - # or to also forward fill intermediate NaN values, use: price_df = price_df.ffill().bfill() - price_df[: price_df.first_valid_index()] = price_df[ - price_df.index == price_df.first_valid_index() - ].values[0] - price_df[price_df.last_valid_index() :] = price_df[ - price_df.index == price_df.last_valid_index() + df = df.reindex(index) + # or to also forward fill intermediate NaN values, use: df = df.ffill().bfill() + df[: df.first_valid_index()] = df[ + df.index == df.first_valid_index() ].values[0] - return price_df, query_window + df[df.last_valid_index() :] = df[df.index == df.last_valid_index()].values[ + 0 + ] + return df def get_power_values( @@ -309,7 +347,10 @@ def get_quantity_from_attribute( :return: The retrieved quantity or the provided default. """ # Get the default value from the entity attribute - value: str | float | int = entity.get_attribute(attribute, np.nan) + if isinstance(entity, Asset): + value: str | float | int = entity.get_attribute(attribute, np.nan) + else: + value: str | float | int = entity.generic_asset.get_attribute(attribute, np.nan) # Try to convert it to a quantity in the desired unit try: @@ -327,19 +368,23 @@ def get_quantity_from_attribute( def get_series_from_quantity_or_sensor( - quantity_or_sensor: Sensor | ur.Quantity, + variable_quantity: Sensor | list[dict] | ur.Quantity, unit: ur.Quantity | str, query_window: tuple[datetime, datetime], resolution: timedelta, beliefs_before: datetime | None = None, as_instantaneous_events: bool = True, boundary_policy: str | None = None, + fill_sides: bool = False, ) -> pd.Series: """ Get a time series given a quantity or sensor defined on a time window. - :param quantity_or_sensor: A pint Quantity or timely-beliefs Sensor, measuring e.g. power capacity - or efficiency. + :param variable_quantity: Variable quantity measuring e.g. power capacity or efficiency. + One of the following types: + - a timely-beliefs Sensor recording the data + - a list of dictionaries representing a time series specification + - a pint Quantity representing a fixed quantity :param unit: Unit of the output data. :param query_window: Tuple representing the start and end of the requested data. :param resolution: Time resolution of the requested data. @@ -347,21 +392,26 @@ def get_series_from_quantity_or_sensor( at that time. :param as_instantaneous_events: Optionally, convert to instantaneous events, in which case the passed resolution is interpreted as the desired frequency of the data. + :param boundary_policy: When upsampling to instantaneous events, + take the 'max', 'min' or 'first' value at event boundaries. + :param fill_sides If True, values are extended to the edges of the query window: + - The first available value serves as a naive backcast. + - The last available value serves as a naive forecast. :return: Pandas Series with the requested time series data. """ start, end = query_window index = initialize_index(start=start, end=end, resolution=resolution) - if isinstance(quantity_or_sensor, ur.Quantity): - if np.isnan(quantity_or_sensor.magnitude): + if isinstance(variable_quantity, ur.Quantity): + if np.isnan(variable_quantity.magnitude): magnitude = np.nan else: - magnitude = quantity_or_sensor.to(unit).magnitude - time_series = pd.Series(magnitude, index=index) - elif isinstance(quantity_or_sensor, Sensor): + magnitude = variable_quantity.to(unit).magnitude + time_series = pd.Series(magnitude, index=index, name="event_value") + elif isinstance(variable_quantity, Sensor): bdf: tb.BeliefsDataFrame = TimedBelief.search( - quantity_or_sensor, + variable_quantity, event_starts_after=query_window[0], event_ends_before=query_window[1], resolution=resolution, @@ -373,18 +423,30 @@ def get_series_from_quantity_or_sensor( if as_instantaneous_events: bdf = bdf.resample_events(timedelta(0), boundary_policy=boundary_policy) time_series = simplify_index(bdf).reindex(index).squeeze() - time_series = convert_units(time_series, quantity_or_sensor.unit, unit) + time_series = convert_units(time_series, variable_quantity.unit, unit) + elif isinstance(variable_quantity, list): + time_series = pd.Series(np.nan, index=index) + for event in variable_quantity: + value = event["value"] + start = event["start"] + end = event["end"] + time_series[start : end - resolution] = value else: raise TypeError( - f"quantity_or_sensor {quantity_or_sensor} should be a pint Quantity or timely-beliefs Sensor" + f"quantity_or_sensor {variable_quantity} should be a pint Quantity or timely-beliefs Sensor" + ) + + if fill_sides: + time_series = extend_to_edges( + df=time_series, query_window=query_window, resolution=resolution ) return time_series def get_continuous_series_sensor_or_quantity( - quantity_or_sensor: Sensor | ur.Quantity | None, + variable_quantity: Sensor | list[dict] | ur.Quantity | None, actuator: Sensor | Asset, unit: ur.Quantity | str, query_window: tuple[datetime, datetime], @@ -394,11 +456,12 @@ def get_continuous_series_sensor_or_quantity( max_value: float | int | pd.Series = np.nan, as_instantaneous_events: bool = False, boundary_policy: str | None = None, + fill_sides: bool = False, ) -> pd.Series: - """Creates a time series from a quantity or sensor within a specified window, + """Creates a time series from a sensor, time series specification, or quantity within a specified window, falling back to a given `fallback_attribute` and making sure no values exceed `max_value`. - :param quantity_or_sensor: The quantity or sensor containing the data. + :param variable_quantity: A sensor recording the data, a time series specification or a fixed quantity. :param actuator: The actuator from which relevant defaults are retrieved. :param unit: The desired unit of the data. :param query_window: The time window (start, end) to query the data. @@ -408,23 +471,27 @@ def get_continuous_series_sensor_or_quantity( :param max_value: Maximum value (also replacing NaN values). :param as_instantaneous_events: optionally, convert to instantaneous events, in which case the passed resolution is interpreted as the desired frequency of the data. + :param fill_sides If True, values are extended to the edges of the query window: + - The first available value serves as a naive backcast. + - The last available value serves as a naive forecast. :returns: time series data with missing values handled based on the chosen method. """ - if quantity_or_sensor is None: - quantity_or_sensor = get_quantity_from_attribute( + if variable_quantity is None: + variable_quantity = get_quantity_from_attribute( entity=actuator, attribute=fallback_attribute, unit=unit, ) time_series = get_series_from_quantity_or_sensor( - quantity_or_sensor=quantity_or_sensor, + variable_quantity=variable_quantity, unit=unit, query_window=query_window, resolution=resolution, beliefs_before=beliefs_before, as_instantaneous_events=as_instantaneous_events, boundary_policy=boundary_policy, + fill_sides=fill_sides, ) # Apply upper limit diff --git a/flexmeasures/data/models/reporting/aggregator.py b/flexmeasures/data/models/reporting/aggregator.py index d0e1a7474..3b275a13b 100644 --- a/flexmeasures/data/models/reporting/aggregator.py +++ b/flexmeasures/data/models/reporting/aggregator.py @@ -5,6 +5,7 @@ import pandas as pd +from flexmeasures.data.models.data_sources import keep_latest_version from flexmeasures.data.models.reporting import Reporter from flexmeasures.data.schemas.reporting.aggregation import ( AggregatorConfigSchema, @@ -34,6 +35,7 @@ def _compute_report( output: list[dict[str, Any]], resolution: timedelta | None = None, belief_time: datetime | None = None, + use_latest_version_only: bool = True, ) -> list[dict[str, Any]]: """ This method merges all the BeliefDataFrames into a single one, dropping @@ -58,6 +60,17 @@ def _compute_report( source = input_description.get("source") source = input_description.get("sources", source) + if use_latest_version_only and source is None: + source = sensor.search_data_sources( + event_ends_after=start, + event_starts_before=end, + source_types=input_description.pop("source_types", None), + exclude_source_types=input_description.pop( + "exclude_source_types", None + ), + ) + if len(source) > 0: + source = keep_latest_version(source) df = sensor.search_beliefs( event_starts_after=start, diff --git a/flexmeasures/data/models/reporting/pandas_reporter.py b/flexmeasures/data/models/reporting/pandas_reporter.py index 30edb12cd..71e848978 100644 --- a/flexmeasures/data/models/reporting/pandas_reporter.py +++ b/flexmeasures/data/models/reporting/pandas_reporter.py @@ -83,9 +83,9 @@ def fetch_data( source = sensor.search_data_sources( event_ends_after=start, event_starts_before=end, - source_types=_input_search_parameters.get("source_types"), - exclude_source_types=_input_search_parameters.get( - "exclude_source_types" + source_types=_input_search_parameters.pop("source_types", None), + exclude_source_types=_input_search_parameters.pop( + "exclude_source_types", None ), ) if len(source) > 0: @@ -113,7 +113,7 @@ def fetch_data( event_resolution=sensor.event_resolution, ) if droplevels: - # dropping belief_time, source and cummulative_probability columns + # dropping belief_time, source and cumulative_probability columns bdf = bdf.droplevel([1, 2, 3]) assert ( bdf.index.is_unique diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py index 9345fbad4..2b0f947a0 100644 --- a/flexmeasures/data/models/time_series.py +++ b/flexmeasures/data/models/time_series.py @@ -37,7 +37,7 @@ to_annotation_frame, ) from flexmeasures.data.models.charts import chart_type_to_chart_specs -from flexmeasures.data.models.data_sources import DataSource +from flexmeasures.data.models.data_sources import DataSource, keep_latest_version from flexmeasures.data.models.generic_assets import GenericAsset from flexmeasures.data.models.validation_utils import check_required_attributes from flexmeasures.data.queries.sensors import query_sensors_by_proximity @@ -201,13 +201,9 @@ def check_required_attributes( def latest_state( self, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, ) -> tb.BeliefsDataFrame: """Search the most recent event for this sensor, and return the most recent ex-post belief. @@ -227,13 +223,9 @@ def search_annotations( annotations_after: datetime_type | None = None, annotation_ends_before: datetime_type | None = None, # deprecated annotations_before: datetime_type | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, include_asset_annotations: bool = False, include_account_annotations: bool = False, as_frame: bool = False, @@ -307,16 +299,12 @@ def search_beliefs( beliefs_before: datetime_type | None = None, horizons_at_least: timedelta | None = None, horizons_at_most: timedelta | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, most_recent_beliefs_only: bool = True, most_recent_events_only: bool = False, - most_recent_only: bool = None, # deprecated + most_recent_only: bool = False, one_deterministic_belief_per_event: bool = False, one_deterministic_belief_per_event_per_source: bool = False, as_json: bool = False, @@ -332,23 +320,16 @@ def search_beliefs( :param beliefs_before: only return beliefs formed before this datetime (inclusive) :param horizons_at_least: only return beliefs with a belief horizon equal or greater than this timedelta (for example, use timedelta(0) to get ante knowledge time beliefs) :param horizons_at_most: only return beliefs with a belief horizon equal or less than this timedelta (for example, use timedelta(0) to get post knowledge time beliefs) - :param source: search only beliefs by this source (pass the DataSource, or its name or id) or list of sources - :param most_recent_beliefs_only: only return the most recent beliefs for each event from each source (minimum belief horizon) - :param most_recent_events_only: only return (post knowledge time) beliefs for the most recent event (maximum event start) + :param source: search only beliefs by this source (pass the DataSource, or its name or id) or list of sources. Without this set and a most recent parameter used (see below), the results can be of any source. + :param most_recent_beliefs_only: only return the most recent beliefs for each event from each source (minimum belief horizon). Defaults to True. + :param most_recent_events_only: only return (post knowledge time) beliefs for the most recent event (maximum event start). Defaults to False. + :param most_recent_only: only return a single belief, the most recent from the most recent event. Fastest method if you only need one. Defaults to False. To use, also set most_recent_beliefs_only=False. Use with care when data uses cumulative probability (more than one belief per event_start and horizon). :param one_deterministic_belief_per_event: only return a single value per event (no probabilistic distribution and only 1 source) :param one_deterministic_belief_per_event_per_source: only return a single value per event per source (no probabilistic distribution) :param as_json: return beliefs in JSON format (e.g. for use in charts) rather than as BeliefsDataFrame :param resolution: optionally set the resolution of data being displayed :returns: BeliefsDataFrame or JSON string (if as_json is True) """ - # todo: deprecate the 'most_recent_only' argument in favor of 'most_recent_beliefs_only' (announced v0.8.0) - most_recent_beliefs_only = tb_utils.replace_deprecated_argument( - "most_recent_only", - most_recent_only, - "most_recent_beliefs_only", - most_recent_beliefs_only, - required_argument=False, - ) bdf = TimedBelief.search( sensors=self, event_starts_after=event_starts_after, @@ -360,6 +341,7 @@ def search_beliefs( source=source, most_recent_beliefs_only=most_recent_beliefs_only, most_recent_events_only=most_recent_events_only, + most_recent_only=most_recent_only, one_deterministic_belief_per_event=one_deterministic_belief_per_event, one_deterministic_belief_per_event_per_source=one_deterministic_belief_per_event_per_source, resolution=resolution, @@ -379,13 +361,9 @@ def chart( event_ends_before: datetime_type | None = None, beliefs_after: datetime_type | None = None, beliefs_before: datetime_type | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, most_recent_beliefs_only: bool = True, include_data: bool = False, include_sensor_annotations: bool = False, @@ -656,19 +634,15 @@ def search( beliefs_before: datetime_type | None = None, horizons_at_least: timedelta | None = None, horizons_at_most: timedelta | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, user_source_ids: int | list[int] | None = None, source_types: list[str] | None = None, exclude_source_types: list[str] | None = None, most_recent_beliefs_only: bool = True, most_recent_events_only: bool = False, - most_recent_only: bool = None, # deprecated + most_recent_only: bool = False, one_deterministic_belief_per_event: bool = False, one_deterministic_belief_per_event_per_source: bool = False, resolution: str | timedelta = None, @@ -689,8 +663,9 @@ def search( :param user_source_ids: Optional list of user source ids to query only specific user sources :param source_types: Optional list of source type names to query only specific source types * :param exclude_source_types: Optional list of source type names to exclude specific source types * - :param most_recent_beliefs_only: only return the most recent beliefs for each event from each source (minimum belief horizon) + :param most_recent_beliefs_only: only return the most recent beliefs for each event from each source (minimum belief horizon). Defaults to True. :param most_recent_events_only: only return (post knowledge time) beliefs for the most recent event (maximum event start) + :param most_recent_only: only return a single belief, the most recent from the most recent event. Fastest method if you only need one. :param one_deterministic_belief_per_event: only return a single value per event (no probabilistic distribution and only 1 source) :param one_deterministic_belief_per_event_per_source: only return a single value per event per source (no probabilistic distribution) :param resolution: Optional timedelta or pandas freqstr used to resample the results ** @@ -710,14 +685,6 @@ def search( "sensors", sensors, ) - # todo: deprecate the 'most_recent_only' argument in favor of 'most_recent_beliefs_only' (announced v0.8.0) - most_recent_beliefs_only = tb_utils.replace_deprecated_argument( - "most_recent_only", - most_recent_only, - "most_recent_beliefs_only", - most_recent_beliefs_only, - required_argument=False, - ) # convert to list sensors = [sensors] if not isinstance(sensors, list) else sensors @@ -752,6 +719,7 @@ def search( source=parsed_sources, most_recent_beliefs_only=most_recent_beliefs_only, most_recent_events_only=most_recent_events_only, + most_recent_only=most_recent_only, custom_filter_criteria=source_criteria, custom_join_targets=custom_join_targets, ) @@ -765,11 +733,21 @@ def search( # Fast track, no need to loop over beliefs pass else: - bdf = ( - bdf.for_each_belief(get_median_belief) - .groupby(level=["event_start"], group_keys=False) - .apply(lambda x: x.head(1)) - ) + source = keep_latest_version(bdf.lineage.sources) + if len(source) <= 1: + indices = list(bdf.index.names) + bdf = bdf.reset_index() + bdf = bdf[bdf["source"] == source[0]] + bdf = bdf.set_index(indices) + else: + current_app.logger.warn( + f"Different data sources found on sensor {bdf.sensor.name} (ID={bdf.sensor.id}). Computing their average values is a slow operation." + ) + bdf = ( + bdf.for_each_belief(get_median_belief) + .groupby(level=["event_start"], group_keys=False) + .apply(lambda x: x.head(1)) + ) elif one_deterministic_belief_per_event_per_source: if len(bdf) == 0 or bdf.lineage.probabilistic_depth == 1: # Fast track, no need to loop over beliefs diff --git a/flexmeasures/data/models/user.py b/flexmeasures/data/models/user.py index 8704681b5..32bac6e28 100644 --- a/flexmeasures/data/models/user.py +++ b/flexmeasures/data/models/user.py @@ -5,7 +5,7 @@ from flask_security import UserMixin, RoleMixin import pandas as pd -from sqlalchemy import select +from sqlalchemy import select, func from sqlalchemy.orm import relationship, backref from sqlalchemy import Boolean, DateTime, Column, Integer, String, ForeignKey from sqlalchemy.ext.hybrid import hybrid_property @@ -62,6 +62,9 @@ class Account(db.Model, AuthModelMixin): secondary="roles_accounts", backref=backref("accounts", lazy="dynamic"), ) + primary_color = Column(String(7), default=None) + secondary_color = Column(String(7), default=None) + logo_url = Column(String(255), default=None) annotations = db.relationship( "Annotation", secondary="annotations_accounts", @@ -120,13 +123,9 @@ def search_annotations( annotations_after: datetime | None = None, annotation_ends_before: datetime | None = None, # deprecated annotations_before: datetime | None = None, - source: DataSource - | list[DataSource] - | int - | list[int] - | str - | list[str] - | None = None, + source: ( + DataSource | list[DataSource] | int | list[int] | str | list[str] | None + ) = None, as_frame: bool = False, ) -> list[Annotation] | pd.DataFrame: """Return annotations assigned to this account. @@ -178,6 +177,20 @@ def search_annotations( return to_annotation_frame(annotations) if as_frame else annotations + @property + def number_of_assets(self): + from flexmeasures.data.models.generic_assets import GenericAsset + + return db.session.execute( + select(func.count()).where(GenericAsset.account_id == self.id) + ).scalar_one_or_none() + + @property + def number_of_users(self): + return db.session.execute( + select(func.count()).where(User.account_id == self.id) + ).scalar_one_or_none() + class RolesUsers(db.Model): __tablename__ = "roles_users" diff --git a/flexmeasures/data/queries/generic_assets.py b/flexmeasures/data/queries/generic_assets.py index 083313bf9..181a398d2 100644 --- a/flexmeasures/data/queries/generic_assets.py +++ b/flexmeasures/data/queries/generic_assets.py @@ -3,7 +3,8 @@ from itertools import groupby from flask_login import current_user -from sqlalchemy import select, Select +from sqlalchemy import select, Select, or_, and_, union_all +from sqlalchemy.orm import aliased from flexmeasures.data import db from flexmeasures.auth.policy import user_has_admin_access @@ -148,3 +149,39 @@ def get_asset_group_queries( asset_queries.update(get_location_queries()) return asset_queries + + +def query_assets_by_search_terms( + search_terms: list[str] | None, + filter_statement: bool = True, +) -> Select: + select_statement = select(GenericAsset) + if search_terms is not None: + # Search terms in the search filter should either come back in the asset name or account name + private_select_statement = select_statement.join( + Account, Account.id == GenericAsset.account_id + ) + private_filter_statement = filter_statement & and_( + *( + or_( + GenericAsset.name.ilike(f"%{term}%"), + Account.name.ilike(f"%{term}%"), + ) + for term in search_terms + ) + ) + public_select_statement = select_statement + public_filter_statement = ( + filter_statement + & GenericAsset.account_id.is_(None) + & and_(GenericAsset.name.ilike(f"%{term}%") for term in search_terms) + ) + subquery = union_all( + private_select_statement.where(private_filter_statement), + public_select_statement.where(public_filter_statement), + ).subquery() + asset_alias = aliased(GenericAsset, subquery) + query = select(asset_alias).order_by(asset_alias.id) + else: + query = select_statement.where(filter_statement) + return query diff --git a/flexmeasures/data/queries/sensors.py b/flexmeasures/data/queries/sensors.py index 3336f65a6..ea0194c17 100644 --- a/flexmeasures/data/queries/sensors.py +++ b/flexmeasures/data/queries/sensors.py @@ -1,6 +1,6 @@ from __future__ import annotations -from sqlalchemy.sql import Select, select +from sqlalchemy import select, Select from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType from flexmeasures.data.queries.utils import potentially_limit_assets_query_to_account diff --git a/flexmeasures/data/queries/users.py b/flexmeasures/data/queries/users.py new file mode 100644 index 000000000..84b0e04f6 --- /dev/null +++ b/flexmeasures/data/queries/users.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from sqlalchemy import select, Select, or_, and_ + +from flexmeasures.data.models.user import User as UserModel, Account + + +def query_users_by_search_terms( + search_terms: list[str] | None, + filter_statement: bool = True, +) -> Select: + select_statement = select(UserModel) + if search_terms is not None: + filter_statement = filter_statement & and_( + *( + or_( + UserModel.email.ilike(f"%{term}%"), + UserModel.username.ilike(f"%{term}%"), + UserModel.account.has(Account.name.ilike(f"%{term}%")), + ) + for term in search_terms + ) + ) + query = select_statement.where(filter_statement).order_by(UserModel.id) + else: + query = select_statement.where(filter_statement).order_by(UserModel.id) + return query diff --git a/flexmeasures/data/schemas/__init__.py b/flexmeasures/data/schemas/__init__.py index 1df7317b4..8f56be0ab 100644 --- a/flexmeasures/data/schemas/__init__.py +++ b/flexmeasures/data/schemas/__init__.py @@ -5,7 +5,7 @@ from .account import AccountIdField from .generic_assets import GenericAssetIdField as AssetIdField from .locations import LatitudeField, LongitudeField -from .sensors import SensorIdField, QuantityOrSensor +from .sensors import SensorIdField, VariableQuantityField from .sources import DataSourceIdField as SourceIdField from .times import ( AwareDateTimeField, diff --git a/flexmeasures/data/schemas/account.py b/flexmeasures/data/schemas/account.py index a1c25ce2c..3f34f84e6 100644 --- a/flexmeasures/data/schemas/account.py +++ b/flexmeasures/data/schemas/account.py @@ -1,6 +1,6 @@ from flask.cli import with_appcontext from flexmeasures.data import ma -from marshmallow import fields +from marshmallow import fields, validates from flexmeasures.data import db from flexmeasures.data.models.user import ( @@ -8,6 +8,7 @@ AccountRole as AccountRoleModel, ) from flexmeasures.data.schemas.utils import FMValidationError, MarshmallowClickMixin +from flexmeasures.utils.validation_utils import validate_color_hex, validate_url class AccountRoleSchema(ma.SQLAlchemySchema): @@ -29,9 +30,33 @@ class Meta: id = ma.auto_field(dump_only=True) name = ma.auto_field(required=True) + primary_color = ma.auto_field(required=False) + secondary_color = ma.auto_field(required=False) + logo_url = ma.auto_field(required=False) account_roles = fields.Nested("AccountRoleSchema", exclude=("accounts",), many=True) consultancy_account_id = ma.auto_field() + @validates("primary_color") + def validate_primary_color(self, value): + try: + validate_color_hex(value) + except ValueError as e: + raise FMValidationError(str(e)) + + @validates("secondary_color") + def validate_secondary_color(self, value): + try: + validate_color_hex(value) + except ValueError as e: + raise FMValidationError(str(e)) + + @validates("logo_url") + def validate_logo_url(self, value): + try: + validate_url(value) + except ValueError as e: + raise FMValidationError(str(e)) + class AccountIdField(fields.Int, MarshmallowClickMixin): """Field that deserializes to an Account and serializes back to an integer.""" diff --git a/flexmeasures/data/schemas/generic_assets.py b/flexmeasures/data/schemas/generic_assets.py index 82435c24b..fba44fe2b 100644 --- a/flexmeasures/data/schemas/generic_assets.py +++ b/flexmeasures/data/schemas/generic_assets.py @@ -17,7 +17,6 @@ ) from flexmeasures.auth.policy import user_has_admin_access from flexmeasures.cli import is_running as running_as_cli -from flexmeasures.utils.coding_utils import flatten_unique class JSON(fields.Field): @@ -31,6 +30,124 @@ def _serialize(self, value, attr, data, **kwargs) -> str: return json.dumps(value) +class SensorsToShowSchema(fields.Field): + """ + Schema for validating and deserializing the `sensors_to_show` attribute of a GenericAsset. + + The `sensors_to_show` attribute defines which sensors should be displayed for a particular asset. + It supports various input formats, which are standardized into a list of dictionaries, each containing + a `title` (optional) and a `sensors` list. The valid input formats include: + + - A single sensor ID (int): `42` -> `{"title": None, "sensors": [42]}` + - A list of sensor IDs (list of ints): `[42, 43]` -> `{"title": None, "sensors": [42, 43]}` + - A dictionary with a title and sensor: `{"title": "Temperature", "sensor": 42}` -> `{"title": "Temperature", "sensors": [42]}` + - A dictionary with a title and sensors: `{"title": "Pressure", "sensors": [42, 43]}` + + Validation ensures that: + - The input is either a list, integer, or dictionary. + - If the input is a dictionary, it must contain either `sensor` (int) or `sensors` (list of ints). + - All sensor IDs must be valid integers. + + Example Input: + - `[{"title": "Test", "sensors": [1, 2]}, {"title": None, "sensors": [3, 4]}, 5]` + + Example Output (Standardized): + - `[{"title": "Test", "sensors": [1, 2]}, {"title": None, "sensors": [3, 4]}, {"title": None, "sensors": [5]}]` + """ + + def deserialize(self, value, **kwargs) -> list: + """ + Validate and deserialize the input value. + """ + try: + # Parse JSON if input is a string + if isinstance(value, str): + value = json.loads(value) + + # Ensure value is a list + if not isinstance(value, list): + raise ValidationError("sensors_to_show should be a list.") + + # Standardize each item in the list + return [self._standardize_item(item) for item in value] + except json.JSONDecodeError: + raise ValidationError("Invalid JSON string.") + + def _standardize_item(self, item) -> dict: + """ + Standardize different input formats to a consistent dictionary format. + """ + if isinstance(item, int): + return {"title": None, "sensors": [item]} + elif isinstance(item, list): + if not all(isinstance(sensor_id, int) for sensor_id in item): + raise ValidationError( + "All elements in a list within 'sensors_to_show' must be integers." + ) + return {"title": None, "sensors": item} + elif isinstance(item, dict): + if "title" not in item: + raise ValidationError("Dictionary must contain a 'title' key.") + else: + title = item["title"] + if not isinstance(title, str) and title is not None: + raise ValidationError("'title' value must be a string.") + + if "sensor" in item: + sensor = item["sensor"] + if not isinstance(sensor, int): + raise ValidationError("'sensor' value must be an integer.") + return {"title": title, "sensors": [sensor]} + elif "sensors" in item: + sensors = item["sensors"] + if not isinstance(sensors, list) or not all( + isinstance(sensor_id, int) for sensor_id in sensors + ): + raise ValidationError("'sensors' value must be a list of integers.") + return {"title": title, "sensors": sensors} + else: + raise ValidationError( + "Dictionary must contain either 'sensor' or 'sensors' key." + ) + else: + raise ValidationError( + "Invalid item type in 'sensors_to_show'. Expected int, list, or dict." + ) + + @classmethod + def flatten(cls, nested_list) -> list[int]: + """ + Flatten a nested list of sensors or sensor dictionaries into a unique list of sensor IDs. + + This method processes the following formats, for each of the entries of the nested list: + - A list of sensor IDs: `[1, 2, 3]` + - A list of dictionaries where each dictionary contains a `sensors` list or a `sensor` key: + `[{"title": "Temperature", "sensors": [1, 2]}, {"title": "Pressure", "sensor": 3}]` + - Mixed formats: `[{"title": "Temperature", "sensors": [1, 2]}, {"title": "Pressure", "sensor": 3}, 4, 5, 1]` + + It extracts all sensor IDs, removes duplicates, and returns a flattened list of unique sensor IDs. + + Args: + nested_list (list): A list containing sensor IDs, or dictionaries with `sensors` or `sensor` keys. + + Returns: + list: A unique list of sensor IDs. + """ + + all_objects = [] + for s in nested_list: + if isinstance(s, list): + all_objects.extend(s) + elif isinstance(s, dict): + if "sensors" in s: + all_objects.extend(s["sensors"]) + if "sensor" in s: + all_objects.append(s["sensor"]) + else: + all_objects.append(s) + return list(dict.fromkeys(all_objects).keys()) + + class GenericAssetSchema(ma.SQLAlchemySchema): """ GenericAsset schema, with validations. @@ -39,12 +156,22 @@ class GenericAssetSchema(ma.SQLAlchemySchema): id = ma.auto_field(dump_only=True) name = fields.Str(required=True) account_id = ma.auto_field() + owner = ma.Nested("AccountSchema", dump_only=True, only=("id", "name")) latitude = LatitudeField(allow_none=True) longitude = LongitudeField(allow_none=True) generic_asset_type_id = fields.Integer(required=True) + generic_asset_type = ma.Nested( + "GenericAssetTypeSchema", dump_only=True, only=("id", "name") + ) attributes = JSON(required=False) parent_asset_id = fields.Int(required=False, allow_none=True) - child_assets = ma.Nested("GenericAssetSchema", many=True, dumb_only=True) + child_assets = ma.Nested( + "GenericAssetSchema", + many=True, + dump_only=True, + only=("id", "name", "account_id", "generic_asset_type"), + ) + sensors = ma.Nested("SensorSchema", many=True, dump_only=True, only=("id", "name")) production_price_sensor_id = fields.Int(required=False, allow_none=True) consumption_price_sensor_id = fields.Int(required=False, allow_none=True) inflexible_device_sensor_ids = fields.List( @@ -112,27 +239,19 @@ def validate_account(self, account_id: int | None): def validate_attributes(self, attributes: dict): sensors_to_show = attributes.get("sensors_to_show", []) - # Check type - if not isinstance(sensors_to_show, list): - raise ValidationError("sensors_to_show should be a list.") - for sensor_listing in sensors_to_show: - if not isinstance(sensor_listing, (int, list)): - raise ValidationError( - "sensors_to_show should only contain sensor IDs (integers) or lists thereof." - ) - if isinstance(sensor_listing, list): - for sensor_id in sensor_listing: - if not isinstance(sensor_id, int): - raise ValidationError( - "sensors_to_show should only contain sensor IDs (integers) or lists thereof." - ) - - # Check whether IDs represent accessible sensors - from flexmeasures.data.schemas import SensorIdField - - sensor_ids = flatten_unique(sensors_to_show) - for sensor_id in sensor_ids: - SensorIdField().deserialize(sensor_id) + if sensors_to_show: + + # Use SensorsToShowSchema to validate and deserialize sensors_to_show + sensors_to_show_schema = SensorsToShowSchema() + + standardized_sensors = sensors_to_show_schema.deserialize(sensors_to_show) + unique_sensor_ids = SensorsToShowSchema.flatten(standardized_sensors) + + # Check whether IDs represent accessible sensors + from flexmeasures.data.schemas import SensorIdField + + for sensor_id in unique_sensor_ids: + SensorIdField().deserialize(sensor_id) class GenericAssetTypeSchema(ma.SQLAlchemySchema): diff --git a/flexmeasures/data/schemas/reporting/pandas_reporter.py b/flexmeasures/data/schemas/reporting/pandas_reporter.py index c0b5e3374..198f5d219 100644 --- a/flexmeasures/data/schemas/reporting/pandas_reporter.py +++ b/flexmeasures/data/schemas/reporting/pandas_reporter.py @@ -8,7 +8,11 @@ ) from flexmeasures.data.schemas.io import RequiredInput, RequiredOutput -from timely_beliefs import BeliefsDataFrame +from timely_beliefs import BeliefsDataFrame, BeliefsSeries + + +from pandas.core.resample import Resampler +from pandas.core.groupby.grouper import Grouper class PandasMethodCall(Schema): @@ -22,31 +26,70 @@ class PandasMethodCall(Schema): @validates_schema def validate_method_call(self, data, **kwargs): + """Validates the method name and its arguments against a set of base classes. - method = data["method"] - method_callable = getattr( - BeliefsDataFrame, method, None - ) # what if the object which is applied to is not a BeliefsDataFrame... - - if not callable(method_callable): - raise ValidationError( - f"method {method} is not a valid BeliefsDataFrame method." - ) - - method_signature = signature(method_callable) + This validation ensures that the provided method exists in one of the + specified base classes (`BeliefsSeries`, `BeliefsDataFrame`, `Resampler`, `Grouper`) + and that the provided arguments (`args` and `kwargs`) are valid for the method's + signature. - try: - args = data.get("args", []).copy() - _kwargs = data.get("kwargs", {}).copy() + Args: + data (dict): A dictionary containing the method name (`method`) and optionally + the method arguments (`args` as a list and `kwargs` as a dictionary). + **kwargs: Additional keyword arguments passed by the validation framework. - args.insert(0, BeliefsDataFrame) + Raises: + ValidationError: If the method is not callable in any of the base classes or + if the provided arguments do not match the method signature. + """ - method_signature.bind(*args, **_kwargs) - except TypeError: + method = data["method"] + is_callable = [] + bad_arguments = True + + # Iterate through the base classes to validate the method + for base_class in [BeliefsSeries, BeliefsDataFrame, Resampler, Grouper]: + + # Check if the method exists in the base class + method_callable = getattr(base_class, method, None) + if method_callable is None: + # Method does not exist in this base class + is_callable.append(False) + continue + + # Check if the found method is callable + is_callable.append(callable(method_callable)) + + # Retrieve the method's signature for argument validation + method_signature = signature(method_callable) + + try: + # Copy `args` and `kwargs` to avoid modifying the input data + args = data.get("args", []).copy() + _kwargs = data.get("kwargs", {}).copy() + + # Insert the base class as the first argument to the method (self/cls context) + args.insert(0, BeliefsDataFrame) + + # Bind the arguments to the method's signature for validation + method_signature.bind(*args, **_kwargs) + bad_arguments = False # Arguments are valid if binding succeeds + except TypeError: + # If binding raises a TypeError, the arguments are invalid + pass + + # Raise an error if all arguments are invalid across all base classes + if bad_arguments: raise ValidationError( f"Bad arguments or keyword arguments for method {method}" ) + # Raise an error if the method is not callable in any of the base classes + if not any(is_callable): + raise ValidationError( + f"Method {method} is not a valid BeliefsSeries, BeliefsDataFrame, Resampler or Grouper method." + ) + class PandasReporterConfigSchema(ReporterConfigSchema): """ diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 4199e5f53..cb685d648 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -1,6 +1,9 @@ -from marshmallow import Schema, fields, validate +from marshmallow import Schema, fields, validate, validates_schema, ValidationError -from flexmeasures.data.schemas.sensors import QuantityOrSensor, SensorIdField +from flexmeasures.data.schemas.sensors import ( + VariableQuantityField, + SensorIdField, +) class FlexContextSchema(Schema): @@ -8,29 +11,80 @@ class FlexContextSchema(Schema): This schema lists fields that can be used to describe sensors in the optimised portfolio """ - ems_power_capacity_in_mw = QuantityOrSensor( + ems_power_capacity_in_mw = VariableQuantityField( "MW", required=False, data_key="site-power-capacity", validate=validate.Range(min=0), ) - ems_production_capacity_in_mw = QuantityOrSensor( + ems_production_capacity_in_mw = VariableQuantityField( "MW", required=False, data_key="site-production-capacity", validate=validate.Range(min=0), ) - ems_consumption_capacity_in_mw = QuantityOrSensor( + ems_consumption_capacity_in_mw = VariableQuantityField( "MW", required=False, data_key="site-consumption-capacity", validate=validate.Range(min=0), ) + + ems_soft_production_capacity_in_mw = VariableQuantityField( + "MW", + required=False, + data_key="site-soft-production-capacity", + validate=validate.Range(min=0), + ) + ems_soft_consumption_capacity_in_mw = VariableQuantityField( + "MW", + required=False, + data_key="site-soft-consumption-capacity", + validate=validate.Range(min=0), + ) + + ems_power_limit_relaxation_cost = fields.Float( + data_key="power-limit-deviation-cost", required=False, default=None + ) + ems_soft_power_limit_relaxation_cost = fields.Float( + data_key="soft-power-limit-deviation-cost", required=False, default=None + ) + consumption_price_sensor = SensorIdField(data_key="consumption-price-sensor") production_price_sensor = SensorIdField(data_key="production-price-sensor") curtailable_device_sensors = fields.List( SensorIdField(), data_key="curtailable-device-sensors" ) + + # todo: deprecated since flexmeasures==0.23 + consumption_price_sensor = SensorIdField(data_key="consumption-price-sensor") + production_price_sensor = SensorIdField(data_key="production-price-sensor") + + consumption_price = VariableQuantityField( + "/MWh", + required=False, + data_key="consumption-price", + return_magnitude=False, + ) + production_price = VariableQuantityField( + "/MWh", + required=False, + data_key="production-price", + return_magnitude=False, + ) + inflexible_device_sensors = fields.List( SensorIdField(), data_key="inflexible-device-sensors" ) + + @validates_schema + def check_prices(self, data: dict, **kwargs): + """Check whether the flex-context contains at most 1 consumption price and at most 1 production price field.""" + if "consumption_price_sensor" in data and "consumption_price" in data: + raise ValidationError( + "Must pass either consumption-price or consumption-price-sensor." + ) + if "production_price_sensor" in data and "production_price" in data: + raise ValidationError( + "Must pass either production-price or production-price-sensor." + ) diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index 69ec74e7b..44a123cf3 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -15,7 +15,7 @@ from flexmeasures.data.models.time_series import Sensor from flexmeasures.data.schemas.units import QuantityField -from flexmeasures.data.schemas.sensors import QuantityOrSensor, TimeSeriesOrSensor +from flexmeasures.data.schemas.sensors import VariableQuantityField from flexmeasures.utils.unit_utils import ur @@ -54,36 +54,62 @@ class StorageFlexModelSchema(Schema): You can use StorageScheduler.deserialize_flex_config to get that filled in. """ - soc_at_start = fields.Float(required=True, data_key="soc-at-start") + soc_at_start = QuantityField( + required=True, + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ + return_magnitude=True, + data_key="soc-at-start", + ) - soc_min = fields.Float(validate=validate.Range(min=0), data_key="soc-min") - soc_max = fields.Float(data_key="soc-max") + soc_min = QuantityField( + validate=validate.Range( + min=0 + ), # change to min=ur.Quantity("0 MWh") in case return_magnitude=False + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ + return_magnitude=True, + data_key="soc-min", + ) + soc_max = QuantityField( + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ + return_magnitude=True, + data_key="soc-max", + ) - power_capacity_in_mw = QuantityOrSensor( + power_capacity_in_mw = VariableQuantityField( "MW", required=False, data_key="power-capacity" ) - consumption_capacity = QuantityOrSensor( + consumption_capacity = VariableQuantityField( "MW", data_key="consumption-capacity", required=False ) - production_capacity = QuantityOrSensor( + production_capacity = VariableQuantityField( "MW", data_key="production-capacity", required=False ) - # Timezone placeholder for the soc_maxima, soc_minima and soc_targets fields are overridden in __init__ - soc_maxima = TimeSeriesOrSensor( - unit="MWh", timezone="placeholder", data_key="soc-maxima" + # Timezone placeholders for the soc_maxima, soc_minima and soc_targets fields are overridden in __init__ + soc_maxima = VariableQuantityField( + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ + timezone="placeholder", + data_key="soc-maxima", ) - soc_minima = TimeSeriesOrSensor( - unit="MWh", + soc_minima = VariableQuantityField( + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ timezone="placeholder", data_key="soc-minima", value_validator=validate.Range(min=0), ) - soc_targets = TimeSeriesOrSensor( - unit="MWh", timezone="placeholder", data_key="soc-targets" + soc_targets = VariableQuantityField( + to_unit="MWh", + default_src_unit="dimensionless", # placeholder, overridden in __init__ + timezone="placeholder", + data_key="soc-targets", ) soc_unit = fields.Str( @@ -94,12 +120,13 @@ class StorageFlexModelSchema(Schema): ] ), data_key="soc-unit", - ) # todo: allow unit to be set per field, using QuantityField("%", validate=validate.Range(min=0, max=1)) + required=False, + ) - charging_efficiency = QuantityOrSensor( + charging_efficiency = VariableQuantityField( "%", data_key="charging-efficiency", required=False ) - discharging_efficiency = QuantityOrSensor( + discharging_efficiency = VariableQuantityField( "%", data_key="discharging-efficiency", required=False ) @@ -107,37 +134,74 @@ class StorageFlexModelSchema(Schema): data_key="roundtrip-efficiency", required=False ) - storage_efficiency = QuantityOrSensor( + storage_efficiency = VariableQuantityField( "%", default_src_unit="dimensionless", data_key="storage-efficiency" ) prefer_charging_sooner = fields.Bool(data_key="prefer-charging-sooner") - soc_gain = fields.List(QuantityOrSensor("MW"), data_key="soc-gain", required=False) + soc_gain = fields.List( + VariableQuantityField("MW"), + data_key="soc-gain", + required=False, + validate=validate.Length(min=1), + ) soc_usage = fields.List( - QuantityOrSensor("MW"), data_key="soc-usage", required=False + VariableQuantityField("MW"), + data_key="soc-usage", + required=False, + validate=validate.Length(min=1), ) - relaxed = fields.Bool(data_key="relaxed", required=False, default=False) + future_reward = fields.Float(data_key="future-reward", required=False, default=None) + soc_deviation_penalty = fields.Float( + data_key="soc-deviation-penalty", required=False, default=None + ) - def __init__(self, start: datetime, sensor: Sensor, *args, **kwargs): + def __init__( + self, + start: datetime, + sensor: Sensor, + *args, + default_soc_unit: str | None = None, + **kwargs, + ): """Pass the schedule's start, so we can use it to validate soc-target datetimes.""" self.start = start self.sensor = sensor - self.soc_maxima = TimeSeriesOrSensor( - unit="MWh", timezone=sensor.timezone, data_key="soc-maxima" + + # guess default soc-unit + if default_soc_unit is None: + if self.sensor.unit in ("MWh", "kWh"): + default_soc_unit = self.sensor.unit + elif self.sensor.unit in ("MW", "kW"): + default_soc_unit = self.sensor.unit + "h" + + self.soc_maxima = VariableQuantityField( + to_unit="MWh", + default_src_unit=default_soc_unit, + timezone=sensor.timezone, + data_key="soc-maxima", ) - self.soc_minima = TimeSeriesOrSensor( - unit="MWh", + self.soc_minima = VariableQuantityField( + to_unit="MWh", + default_src_unit=default_soc_unit, timezone=sensor.timezone, data_key="soc-minima", value_validator=validate.Range(min=0), ) - self.soc_targets = TimeSeriesOrSensor( - unit="MWh", timezone=sensor.timezone, data_key="soc-targets" + self.soc_targets = VariableQuantityField( + to_unit="MWh", + default_src_unit=default_soc_unit, + timezone=sensor.timezone, + data_key="soc-targets", ) super().__init__(*args, **kwargs) + if default_soc_unit is not None: + for field in self.fields.keys(): + if field.startswith("soc_"): + setattr(self.fields[field], "default_src_unit", default_soc_unit) @validates_schema def check_whether_targets_exceed_max_planning_horizon(self, data: dict, **kwargs): @@ -188,33 +252,8 @@ def check_redundant_efficiencies(self, data: dict, **kwargs): @post_load def post_load_sequence(self, data: dict, **kwargs) -> dict: """Perform some checks and corrections after we loaded.""" - # currently we only handle MWh internally - # TODO: review when we moved away from capacity having to be described in MWh - if data.get("soc_unit") == "kWh": - data["soc_at_start"] /= 1000.0 - if data.get("soc_min") is not None: - data["soc_min"] /= 1000.0 - if data.get("soc_max") is not None: - data["soc_max"] /= 1000.0 - if ( - not isinstance(data.get("soc_targets"), Sensor) - and data.get("soc_targets") is not None - ): - for target in data["soc_targets"]: - target["value"] /= 1000.0 - if ( - not isinstance(data.get("soc_minima"), Sensor) - and data.get("soc_minima") is not None - ): - for minimum in data["soc_minima"]: - minimum["value"] /= 1000.0 - if ( - not isinstance(data.get("soc_maxima"), Sensor) - and data.get("soc_maxima") is not None - ): - for maximum in data["soc_maxima"]: - maximum["value"] /= 1000.0 - data["soc_unit"] = "MWh" + # currently we only handle MWh internally, and the conversion to MWh happened during deserialization + data["soc_unit"] = "MWh" # Convert efficiency to dimensionless (to the (0,1] range) if data.get("roundtrip_efficiency") is not None: diff --git a/flexmeasures/data/schemas/sensors.py b/flexmeasures/data/schemas/sensors.py index 0c093b6e6..640337139 100644 --- a/flexmeasures/data/schemas/sensors.py +++ b/flexmeasures/data/schemas/sensors.py @@ -1,4 +1,8 @@ from __future__ import annotations + +import numbers + +from flask import current_app from marshmallow import ( Schema, fields, @@ -8,7 +12,6 @@ validates_schema, ) from marshmallow.validate import Validator -from pint import DimensionalityError import json import re @@ -22,9 +25,15 @@ FMValidationError, MarshmallowClickMixin, with_appcontext_if_needed, + convert_to_quantity, +) +from flexmeasures.utils.unit_utils import ( + is_valid_unit, + ur, + units_are_convertible, ) -from flexmeasures.utils.unit_utils import is_valid_unit, ur, units_are_convertible from flexmeasures.data.schemas.times import DurationField, AwareDateTimeField +from flexmeasures.data.schemas.units import QuantityField class JSON(fields.Field): @@ -39,7 +48,12 @@ def _serialize(self, value, attr, data, **kwargs) -> str: class TimedEventSchema(Schema): - value = fields.Float(required=True) + value = QuantityField( + required=True, + to_unit="dimensionless", # placeholder, overridden in __init__ + default_src_unit="dimensionless", # placeholder, overridden in __init__ + return_magnitude=True, # placeholder, overridden in __init__ + ) datetime = AwareDateTimeField(required=False) start = AwareDateTimeField(required=False) end = AwareDateTimeField(required=False) @@ -49,6 +63,9 @@ def __init__( self, timezone: str | None = None, value_validator: Validator | None = None, + to_unit: str | None = None, + default_src_unit: str | None = None, + return_magnitude: bool = True, *args, **kwargs, ): @@ -59,6 +76,16 @@ def __init__( self.timezone = timezone self.value_validator = value_validator super().__init__(*args, **kwargs) + if to_unit is not None: + if to_unit.startswith("/"): + if len(to_unit) < 2: + raise ValueError( + f"Variable `to_unit='{to_unit}'` must define a denominator." + ) + setattr(self.fields["value"], "to_unit", to_unit) + if default_src_unit is not None: + setattr(self.fields["value"], "default_src_unit", default_src_unit) + setattr(self.fields["value"], "return_magnitude", return_magnitude) @validates("value") def validate_value(self, _value): @@ -204,71 +231,129 @@ def _serialize(self, sensor: Sensor, attr, data, **kwargs) -> int: return sensor.id -class QuantityOrSensor(MarshmallowClickMixin, fields.Field): +class VariableQuantityField(MarshmallowClickMixin, fields.Field): def __init__( - self, to_unit: str, default_src_unit: str | None = None, *args, **kwargs + self, + to_unit, + *args, + default_src_unit: str | None = None, + return_magnitude: bool = False, + timezone: str | None = None, + value_validator: Validator | None = None, + **kwargs, ): - """Field for validating, serializing and deserializing a Quantity or a Sensor. + """Field for validating, serializing and deserializing a variable quantity. + + A variable quantity can be represented by a sensor, time series or fixed quantity. + + # todo: Sensor should perhaps deserialize already to sensor data NB any validators passed are only applied to Quantities. For example, validate=validate.Range(min=0) will raise a ValidationError in case of negative quantities, but will let pass any sensor that has recorded negative values. - :param to_unit: unit in which the sensor or quantity should be convertible to - :param default_src_unit: what unit to use in case of getting a numeric value + :param to_unit: Unit to which the sensor, time series or quantity should be convertible. + - Sensors are checked for convertibility, but the original sensor is returned, + so its values are not yet converted. + - Time series and quantities are already converted to the given unit. + - Units starting with '/' (e.g. '/MWh') lead to accepting any value, which will be + converted to the given unit. For example, + a quantity of 1 EUR/kWh with to_unit='/MWh' is deserialized to 1000 EUR/MWh. + :param default_src_unit: What unit to use in case of getting a numeric value. + Does not apply to time series or sensors. + :param return_magnitude: In case of getting a time series, whether the result should include + the magnitude of each quantity, or each Quantity object itself. + :param timezone: Only used in case a time series is specified and one of the *timed events* + in the time series uses a nominal duration, such as "P1D". """ - _validate = kwargs.pop("validate", None) super().__init__(*args, **kwargs) if _validate is not None: # Insert validation into self.validators so that multiple errors can be stored. validator = RepurposeValidatorToIgnoreSensors(_validate) self.validators.insert(0, validator) - self.to_unit = ur.Quantity(to_unit) + self.timezone = timezone + self.value_validator = value_validator + if to_unit.startswith("/") and len(to_unit) < 2: + raise ValueError( + f"Variable `to_unit='{to_unit}'` must define a denominator." + ) + self.to_unit = to_unit self.default_src_unit = default_src_unit + self.return_magnitude = return_magnitude @with_appcontext_if_needed() def _deserialize( - self, value: str | dict[str, int], attr, obj, **kwargs - ) -> ur.Quantity | Sensor: + self, value: dict[str, int] | list[dict] | str, attr, obj, **kwargs + ) -> Sensor | list[dict] | ur.Quantity: + if isinstance(value, dict): - if "sensor" not in value: - raise FMValidationError( - "Dictionary provided but `sensor` key not found." - ) - sensor = SensorIdField(unit=self.to_unit)._deserialize( - value["sensor"], None, None + return self._deserialize_dict(value) + elif isinstance(value, list): + return self._deserialize_list(value) + elif isinstance(value, str): + return self._deserialize_str(value) + elif isinstance(value, numbers.Real) and self.default_src_unit is not None: + return self._deserialize_numeric(value, attr, obj, **kwargs) + else: + raise FMValidationError( + f"Unsupported value type. `{type(value)}` was provided but only dict, list and str are supported." ) - return sensor + def _deserialize_dict(self, value: dict[str, int]) -> Sensor: + """Deserialize a sensor reference to a Sensor.""" + if "sensor" not in value: + raise FMValidationError("Dictionary provided but `sensor` key not found.") + sensor = SensorIdField( + unit=self.to_unit if not self.to_unit.startswith("/") else None + ).deserialize(value["sensor"], None, None) + return sensor - elif isinstance(value, str): - try: - return ur.Quantity(value).to(self.to_unit) - except DimensionalityError as e: - raise FMValidationError( - f"Cannot convert value `{value}` to '{self.to_unit}'" - ) from e - else: - if self.default_src_unit is not None: - return self._deserialize( - f"{value} {self.default_src_unit}", attr, obj, **kwargs + def _deserialize_list(self, value: list[dict]) -> list[dict]: + """Deserialize a time series to a list of timed events.""" + if self.return_magnitude is True: + current_app.logger.warning( + "Deserialized time series will include Quantity objects in the future. Set `return_magnitude=False` to trigger the new behaviour." + ) + field = fields.List( + fields.Nested( + TimedEventSchema( + timezone=self.timezone, + value_validator=self.value_validator, + to_unit=self.to_unit, + default_src_unit=self.default_src_unit, + return_magnitude=self.return_magnitude, ) - - raise FMValidationError( - f"Unsupported value type. `{type(value)}` was provided but only dict and str are supported." ) + ) + return field._deserialize(value, None, None) + + def _deserialize_str(self, value: str) -> ur.Quantity: + """Deserialize a string to a Quantity.""" + return convert_to_quantity(value=value, to_unit=self.to_unit) + + def _deserialize_numeric( + self, value: numbers.Real, attr, obj, **kwargs + ) -> ur.Quantity: + """Try to deserialize a numeric value to a Quantity, using the default_src_unit.""" + return self._deserialize( + f"{value} {self.default_src_unit}", attr, obj, **kwargs + ) def _serialize( - self, value: ur.Quantity | dict[str, Sensor], attr, data, **kwargs + self, value: Sensor | pd.Series | ur.Quantity, attr, data, **kwargs ) -> str | dict[str, int]: - if isinstance(value, ur.Quantity): - return str(value.to(self.to_unit)) - elif isinstance(value, Sensor): + if isinstance(value, Sensor): return dict(sensor=value.id) + elif isinstance(value, pd.Series): + raise NotImplementedError( + "Serialization of a time series from a Pandas Series is not implemented yet." + ) + elif isinstance(value, ur.Quantity): + return str(value.to(self.to_unit)) else: raise FMValidationError( - "Serialized Quantity Or Sensor needs to be of type int, float or Sensor" + "Serialized quantity, sensor or time series needs to be of type int, float, Sensor or pandas.Series." ) def convert(self, value, param, ctx, **kwargs): @@ -286,57 +371,6 @@ def convert(self, value, param, ctx, **kwargs): return super().convert(_value, param, ctx, **kwargs) -class TimeSeriesOrSensor(MarshmallowClickMixin, fields.Field): - def __init__( - self, - unit, - *args, - timezone: str | None = None, - value_validator: Validator | None = None, - **kwargs, - ): - """ - The timezone is only used in case a time series is specified and one - of the *timed events* in the time series uses a nominal duration, such as "P1D". - """ - super().__init__(*args, **kwargs) - self.timezone = timezone - self.value_validator = value_validator - self.unit = ur.Quantity(unit) - - @with_appcontext_if_needed() - def _deserialize( - self, value: str | dict[str, int], attr, obj, **kwargs - ) -> list[dict] | Sensor: - - if isinstance(value, dict): - if "sensor" not in value: - raise FMValidationError( - "Dictionary provided but `sensor` key not found." - ) - - sensor = SensorIdField(unit=self.unit)._deserialize( - value["sensor"], None, None - ) - - return sensor - - elif isinstance(value, list): - field = fields.List( - fields.Nested( - TimedEventSchema( - timezone=self.timezone, value_validator=self.value_validator - ) - ) - ) - - return field._deserialize(value, None, None) - else: - raise FMValidationError( - f"Unsupported value type. `{type(value)}` was provided but only dict and list are supported." - ) - - class RepurposeValidatorToIgnoreSensors(validate.Validator): """Validator that executes another validator (the one you initialize it with) only on non-Sensor values.""" @@ -348,3 +382,21 @@ def __call__(self, value): if not isinstance(value, Sensor): self.original_validator(value) return value + + +class QuantityOrSensor(VariableQuantityField): + def __init__(self, *args, **kwargs): + """Deprecated class. Use `VariableQuantityField` instead.""" + current_app.logger.warning( + "Class `TimeSeriesOrSensor` is deprecated. Use `VariableQuantityField` instead." + ) + super().__init__(return_magnitude=False, *args, **kwargs) + + +class TimeSeriesOrSensor(VariableQuantityField): + def __init__(self, *args, **kwargs): + """Deprecated class. Use `VariableQuantityField` instead.""" + current_app.logger.warning( + "Class `TimeSeriesOrSensor` is deprecated. Use `VariableQuantityField` instead." + ) + super().__init__(return_magnitude=True, *args, **kwargs) diff --git a/flexmeasures/data/schemas/tests/test_latitude_longitude.py b/flexmeasures/data/schemas/tests/test_latitude_longitude.py index 46b48a89f..69fb7aefd 100644 --- a/flexmeasures/data/schemas/tests/test_latitude_longitude.py +++ b/flexmeasures/data/schemas/tests/test_latitude_longitude.py @@ -1,7 +1,8 @@ import pytest +from marshmallow.exceptions import ValidationError + from flexmeasures.data.schemas.locations import LatitudeField, LongitudeField -from flexmeasures.data.schemas.utils import ValidationError @pytest.mark.parametrize( diff --git a/flexmeasures/data/schemas/tests/test_reporting.py b/flexmeasures/data/schemas/tests/test_reporting.py index 079adb9de..efa1ab2c5 100644 --- a/flexmeasures/data/schemas/tests/test_reporting.py +++ b/flexmeasures/data/schemas/tests/test_reporting.py @@ -60,7 +60,7 @@ {"df_output": "output1", "df_input": "sensor_1", "method": "copy"}, {"method": "copy"}, {"df_output": "final_output", "method": "resample", "args": ["1h"]}, - {"method": "sum"}, + {"method": "asfreq"}, ], }, True, diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 2d18e85ef..27e554c1a 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -204,7 +204,7 @@ def test_efficiency_pair( def load_schema(): flex_model = { "storage-efficiency": 1, - "soc-at-start": 0, + "soc-at-start": "0 MWh", } for f in fields: flex_model[f] = "90%" diff --git a/flexmeasures/data/schemas/tests/test_sensor.py b/flexmeasures/data/schemas/tests/test_sensor.py index 3891b7b17..44b22b2bf 100644 --- a/flexmeasures/data/schemas/tests/test_sensor.py +++ b/flexmeasures/data/schemas/tests/test_sensor.py @@ -1,42 +1,92 @@ import pytest from flexmeasures import Sensor -from flexmeasures.data.schemas.sensors import QuantityOrSensor +from flexmeasures.data.schemas.sensors import ( + QuantityOrSensor, + VariableQuantityField, +) from flexmeasures.utils.unit_utils import ur from marshmallow import ValidationError @pytest.mark.parametrize( - "sensor_id, src_quantity, dst_unit, fails", + "src_quantity, dst_unit, fails, exp_dst_quantity", [ # deserialize a sensor - (1, None, "MWh", False), - (1, None, "kWh", False), - (1, None, "kW", False), - (1, None, "EUR", True), - (2, None, "EUR/kWh", False), - (2, None, "EUR", True), + ({"sensor": 1}, "MWh", False, None), + ({"sensor": 1}, "kWh", False, None), + ({"sensor": 1}, "kW", False, None), + ({"sensor": 1}, "EUR", True, None), + ({"sensor": 1}, "/h", False, None), # convertable to MWh²/h + ({"sensor": 2}, "EUR/kWh", False, None), + ({"sensor": 2}, "EUR", True, None), # deserialize a quantity - (None, "1MWh", "MWh", False), - (None, "1 MWh", "kWh", False), - (None, "1 MWh", "kW", True), - (None, "100 EUR/MWh", "EUR/kWh", False), - (None, "100 EUR/MWh", "EUR", True), + ("1MWh", "MWh", False, "1 MWh"), + ("1 MWh", "kWh", False, "1000.0 kWh"), + ("1 MWh", "kW", True, None), + ("100 EUR/MWh", "EUR/kWh", False, "0.1 EUR/kWh"), + ("100 EUR/MWh", "EUR", True, None), + ("1 EUR/kWh", "/MWh", False, "1.0 kEUR/MWh"), + ("50%", "/MWh", False, "500.0 kWh/MWh"), + ("/", "/MWh", True, None), + ("/", "MWh", True, None), + ("10 batteries", "MWh", True, None), + # deserialize a time series specification + ( + [{"start": "2024-08-17T11:00+02", "duration": "PT1H", "value": "2 MWh"}], + "kWh", + False, + "2000.0 kWh", + ), + ( + [ + { + "start": "2024-08-17T11:00+02", + "duration": "PT1H", + "value": "829.4 Wh/kWh", + } + ], + "/MWh", + False, + "829.4 kWh/MWh", + ), + ( + [ + { + "start": "2024-08-17T11:00+02", + "duration": "PT1H", + "value": "914.7 EUR/kWh", + } + ], + "/MWh", + False, + "914.7 kEUR/MWh", + ), + # todo: uncomment after to_preferred gets rid of mEUR + # ( + # [{"start": "2024-08-17T11:00+02", "duration": "PT1H", "value": "120.8 EUR/MWh"}], + # "/kWh", + # False, + # "0.1208 EUR/kWh", + # ), ], ) def test_quantity_or_sensor_deserialize( - setup_dummy_sensors, sensor_id, src_quantity, dst_unit, fails + setup_dummy_sensors, src_quantity, dst_unit, fails, exp_dst_quantity ): - schema = QuantityOrSensor(to_unit=dst_unit) + schema = VariableQuantityField(to_unit=dst_unit, return_magnitude=False) try: - if sensor_id is None: - schema.deserialize(src_quantity) - else: - schema.deserialize({"sensor": sensor_id}) + dst_quantity = schema.deserialize(src_quantity) + if isinstance(src_quantity, ur.Quantity): + assert dst_quantity == ur.Quantity(exp_dst_quantity) + assert str(dst_quantity) == exp_dst_quantity + elif isinstance(src_quantity, list): + assert dst_quantity[0]["value"] == ur.Quantity(exp_dst_quantity) + assert str(dst_quantity[0]["value"]) == exp_dst_quantity assert not fails - except ValidationError: - assert fails + except ValidationError as e: + assert fails, e @pytest.mark.parametrize( @@ -92,5 +142,34 @@ def test_quantity_or_sensor_field( assert val == db.session.get(Sensor, sensor_id) assert not fails - except Exception: - assert fails + except Exception as e: + assert fails, e + + +@pytest.mark.parametrize( + "input_param, dst_unit, fails", + [ + # deserialize a quantity + ([{"value": 1, "datetime": "2024-07-21T00:15+07"}], "MWh", False), + ([{"value": "1", "datetime": "2024-07-21T00:15+07"}], "MWh", True), + ([{"value": "1MWh", "datetime": "2024-07-21T00:15+07"}], "MWh", False), + ([{"value": "1000 kWh", "datetime": "2024-07-21T00:15+07"}], "MWh", False), + ([{"value": "1 MW", "datetime": "2024-07-21T00:15+07"}], "MWh", True), + ], +) +def test_time_series_field(input_param, dst_unit, fails, db): + + field = VariableQuantityField( + to_unit=dst_unit, + default_src_unit="MWh", + return_magnitude=False, + ) + + try: + val = field.convert(input_param, None, None) + assert val[0]["value"].units == ur.Unit(dst_unit) + assert val[0]["value"].magnitude == 1 + + assert not fails + except Exception as e: + assert fails, e diff --git a/flexmeasures/data/schemas/units.py b/flexmeasures/data/schemas/units.py index 99803ca45..a2d6fbc1b 100644 --- a/flexmeasures/data/schemas/units.py +++ b/flexmeasures/data/schemas/units.py @@ -2,7 +2,7 @@ from marshmallow import fields, validate, ValidationError -from flexmeasures.data.schemas.utils import MarshmallowClickMixin +from flexmeasures.data.schemas.utils import MarshmallowClickMixin, convert_to_quantity from flexmeasures.utils.unit_utils import is_valid_unit, ur @@ -34,17 +34,55 @@ class QuantityField(MarshmallowClickMixin, fields.Str): """ - def __init__(self, to_unit: str, *args, **kwargs): + def __init__( + self, + to_unit: str, + *args, + default_src_unit: str | None = None, + return_magnitude: bool = False, + **kwargs, + ): super().__init__(*args, **kwargs) # Insert validation into self.validators so that multiple errors can be stored. validator = QuantityValidator() self.validators.insert(0, validator) - self.to_unit = ur.Quantity(to_unit) + if to_unit.startswith("/") and len(to_unit) < 2: + raise ValueError( + f"Variable `to_unit='{to_unit}'` must define a denominator." + ) + self.to_unit = to_unit + self.default_src_unit = default_src_unit + self.return_magnitude = return_magnitude - def _deserialize(self, value, attr, obj, **kwargs) -> ur.Quantity: + def _deserialize( + self, + value, + attr, + obj, + return_magnitude: bool | None = None, + **kwargs, + ) -> ur.Quantity: """Turn a quantity describing string into a Quantity.""" - return ur.Quantity(value).to(self.to_unit) + if return_magnitude is None: + return_magnitude = self.return_magnitude + if isinstance(value, str): + q = convert_to_quantity(value=value, to_unit=self.to_unit) + elif self.default_src_unit is not None: + q = self._deserialize( + f"{value} {self.default_src_unit}", + attr, + obj, + **kwargs, + return_magnitude=False, + ) + else: + q = self._deserialize( + f"{value}", attr, obj, **kwargs, return_magnitude=False + ) + if return_magnitude: + return q.magnitude + return q def _serialize(self, value, attr, data, **kwargs): """Turn a Quantity into a string in scientific format.""" - return "{:~P}".format(value.to(self.to_unit)) + return "{:~P}".format(value.to(ur.Quantity(self.to_unit))) diff --git a/flexmeasures/data/schemas/utils.py b/flexmeasures/data/schemas/utils.py index 7ea293c31..28186a385 100644 --- a/flexmeasures/data/schemas/utils.py +++ b/flexmeasures/data/schemas/utils.py @@ -2,7 +2,9 @@ import marshmallow as ma from click import get_current_context from flask.cli import with_appcontext as with_cli_appcontext -from marshmallow import ValidationError +from pint import DefinitionSyntaxError, DimensionalityError, UndefinedUnitError + +from flexmeasures.utils.unit_utils import to_preferred, ur class MarshmallowClickMixin(click.ParamType): @@ -20,7 +22,7 @@ def convert(self, value, param, ctx, **kwargs): raise click.exceptions.BadParameter(e, ctx=ctx, param=param) -class FMValidationError(ValidationError): +class FMValidationError(ma.exceptions.ValidationError): """ Custom validation error class. It differs from the classic validation error by having two @@ -47,3 +49,27 @@ def decorator(f): return f return decorator + + +def convert_to_quantity(value: str, to_unit: str) -> ur.Quantity: + """Convert value to quantity in the given unit. + + :param value: Value to convert. + :param to_unit: Unit to convert to. If the unit starts with a '/', + the value can have any unit, and the unit is used as the denominator. + :returns: Quantity in the desired unit. + """ + if to_unit.startswith("/") and len(to_unit) < 2: + raise ValueError(f"Variable `to_unit='{to_unit}'` must define a denominator.") + try: + if to_unit.startswith("/"): + return to_preferred( + ur.Quantity(value) * ur.Quantity(to_unit[1:]) + ) / ur.Quantity(to_unit[1:]) + return ur.Quantity(value).to(ur.Quantity(to_unit)) + except DimensionalityError as e: + raise FMValidationError(f"Cannot convert value `{value}` to '{to_unit}'") from e + except (AssertionError, DefinitionSyntaxError, UndefinedUnitError) as e: + raise FMValidationError( + f"Cannot convert value '{value}' to a valid quantity. {e}" + ) diff --git a/flexmeasures/data/scripts/data_gen.py b/flexmeasures/data/scripts/data_gen.py index ebf1d451f..27f9ddc6b 100644 --- a/flexmeasures/data/scripts/data_gen.py +++ b/flexmeasures/data/scripts/data_gen.py @@ -1,6 +1,7 @@ """ Populate the database with data we know or read in. """ + from __future__ import annotations from pathlib import Path diff --git a/flexmeasures/data/services/asset_grouping.py b/flexmeasures/data/services/asset_grouping.py index c1688183e..c6b02dee4 100644 --- a/flexmeasures/data/services/asset_grouping.py +++ b/flexmeasures/data/services/asset_grouping.py @@ -2,6 +2,7 @@ Convenience functions and class for accessing generic assets in groups. For example, group by asset type or by location. """ + from __future__ import annotations import inflect diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py index b3043a734..cbbd77368 100644 --- a/flexmeasures/data/services/scheduling.py +++ b/flexmeasures/data/services/scheduling.py @@ -16,6 +16,7 @@ from flask import current_app import click from rq import get_current_job, Callback +from rq.exceptions import InvalidJobOperation from rq.job import Job import timely_beliefs as tb import pandas as pd @@ -226,7 +227,10 @@ def create_scheduling_job( job.save_meta() # in case the function enqueues it - job_status = job.get_status(refresh=True) + try: + job_status = job.get_status(refresh=True) + except InvalidJobOperation: + job_status = None # with job_status=None, we ensure that only fresh new jobs are enqueued (in the contrary they should be requeued) if enqueue and not job_status: diff --git a/flexmeasures/data/services/sensors.py b/flexmeasures/data/services/sensors.py index e07a5fd41..a3172c1dc 100644 --- a/flexmeasures/data/services/sensors.py +++ b/flexmeasures/data/services/sensors.py @@ -4,8 +4,11 @@ import hashlib from datetime import datetime, timedelta from flask import current_app +from functools import lru_cache from isodate import duration_isoformat +import time from timely_beliefs import BeliefsDataFrame +import pandas as pd from humanize.time import naturaldelta @@ -16,6 +19,7 @@ from flexmeasures.data import db from flexmeasures import Sensor, Account, Asset +from flexmeasures.data.models.data_sources import DataSource from flexmeasures.data.models.generic_assets import GenericAsset from flexmeasures.data.schemas.reporting import StatusSchema from flexmeasures.utils.time_utils import server_now @@ -35,12 +39,12 @@ def get_sensors( :param sensor_name_allowlist: optionally, allow only sensors whose name is in this list """ sensor_query = sa.select(Sensor) - if account is None: - account_ids = [] - elif isinstance(account, list): - account_ids = [account.id for account in account] + if isinstance(account, list): + accounts = account else: - account_ids = [account.id] + accounts: list = [account] if account else [] + account_ids: list = [acc.id for acc in accounts] + sensor_query = sensor_query.join( GenericAsset, GenericAsset.id == Sensor.generic_asset_id ).filter(Sensor.generic_asset_id == GenericAsset.id) @@ -57,14 +61,19 @@ def get_sensors( sensor_query = sensor_query.filter(Sensor.id.in_(sensor_id_allowlist)) if sensor_name_allowlist: sensor_query = sensor_query.filter(Sensor.name.in_(sensor_name_allowlist)) + return db.session.scalars(sensor_query).all() def _get_sensor_bdf(sensor: Sensor, staleness_search: dict) -> BeliefsDataFrame | None: - """Get bdf for a given sensor with given search parameters.""" + """ + Retrieve the BeliefsDataFrame for a given sensor using the specified search parameters. The 'most_recent_only' + parameter is set to True, which ensures that only the most recent belief from the most recent event is returned. + """ bdf = TimedBelief.search( sensors=sensor, - most_recent_events_only=True, + most_recent_beliefs_only=False, + most_recent_only=True, **staleness_search, ) if bdf.empty: @@ -328,3 +337,93 @@ def build_asset_jobs_data( ) return jobs_data + + +@lru_cache() +def _get_sensor_stats(sensor: Sensor, ttl_hash=None) -> dict: + # Subquery for filtered aggregates + subquery_for_filtered_aggregates = ( + sa.select( + TimedBelief.source_id, + sa.func.max(TimedBelief.event_value).label("max_event_value"), + sa.func.avg(TimedBelief.event_value).label("avg_event_value"), + sa.func.sum(TimedBelief.event_value).label("sum_event_value"), + sa.func.min(TimedBelief.event_value).label("min_event_value"), + ) + .filter(TimedBelief.event_value != float("NaN")) + .filter(TimedBelief.sensor_id == sensor.id) + .group_by(TimedBelief.source_id) + .subquery() + ) + + raw_stats = db.session.execute( + sa.select( + DataSource.name, + sa.func.min(TimedBelief.event_start).label("min_event_start"), + sa.func.max(TimedBelief.event_start).label("max_event_start"), + subquery_for_filtered_aggregates.c.min_event_value, + subquery_for_filtered_aggregates.c.max_event_value, + subquery_for_filtered_aggregates.c.avg_event_value, + subquery_for_filtered_aggregates.c.sum_event_value, + sa.func.count(TimedBelief.event_value).label("count_event_value"), + ) + .select_from(TimedBelief) + .join(DataSource, DataSource.id == TimedBelief.source_id) + .join( + subquery_for_filtered_aggregates, + subquery_for_filtered_aggregates.c.source_id == TimedBelief.source_id, + ) + .filter(TimedBelief.sensor_id == sensor.id) + .group_by( + DataSource.name, + subquery_for_filtered_aggregates.c.min_event_value, + subquery_for_filtered_aggregates.c.max_event_value, + subquery_for_filtered_aggregates.c.avg_event_value, + subquery_for_filtered_aggregates.c.sum_event_value, + ) + ).fetchall() + + stats = dict() + for row in raw_stats: + ( + data_source, + min_event_start, + max_event_start, + min_value, + max_value, + mean_value, + sum_values, + count_values, + ) = row + first_event_start = ( + pd.Timestamp(min_event_start).tz_convert(sensor.timezone).isoformat() + ) + last_event_end = ( + pd.Timestamp(max_event_start + sensor.event_resolution) + .tz_convert(sensor.timezone) + .isoformat() + ) + stats[data_source] = { + "First event start": first_event_start, + "Last event end": last_event_end, + "Min value": min_value, + "Max value": max_value, + "Mean value": mean_value, + "Sum over values": sum_values, + "Number of values": count_values, + } + return stats + + +def _get_ttl_hash(seconds=120) -> int: + """Returns the same value within "seconds" time period + Is needed to make LRU cache a TTL one + (lru_cache is used when call arguments are the same, + here we ensure that call arguments are the same in "seconds" period of time). + """ + return round(time.time() / seconds) + + +def get_sensor_stats(sensor: Sensor) -> dict: + """Get stats for a sensor""" + return _get_sensor_stats(sensor, ttl_hash=_get_ttl_hash()) diff --git a/flexmeasures/data/services/time_series.py b/flexmeasures/data/services/time_series.py index c32482ee0..c86cd6a6b 100644 --- a/flexmeasures/data/services/time_series.py +++ b/flexmeasures/data/services/time_series.py @@ -7,7 +7,6 @@ from flask import current_app import pandas as pd import timely_beliefs as tb -from timely_beliefs.beliefs import utils as belief_utils from flexmeasures.data.queries.utils import simplify_index @@ -69,7 +68,7 @@ def drop_unchanged_beliefs(bdf: tb.BeliefsDataFrame) -> tb.BeliefsDataFrame: ex_ante_bdf = bdf[bdf.belief_horizons > timedelta(0)] ex_post_bdf = bdf[bdf.belief_horizons <= timedelta(0)] if not ex_ante_bdf.empty and not ex_post_bdf.empty: - # We treat each part separately to avoid the ex-post knowledge would be lost + # We treat each part separately to avoid that ex-post knowledge would be lost ex_ante_bdf = drop_unchanged_beliefs(ex_ante_bdf) ex_post_bdf = drop_unchanged_beliefs(ex_post_bdf) bdf = pd.concat([ex_ante_bdf, ex_post_bdf]) @@ -111,18 +110,14 @@ def _drop_unchanged_beliefs_compared_to_db( else: # Look up only ex-post beliefs (horizon <= 0) kwargs = dict(horizons_at_most=timedelta(0)) - previous_beliefs_in_db = bdf.sensor.search_beliefs( + previous_most_recent_beliefs_in_db = bdf.sensor.search_beliefs( event_starts_after=bdf.event_starts[0], event_ends_before=bdf.event_ends[-1], beliefs_before=bdf.lineage.belief_times[0], # unique belief time source=bdf.lineage.sources[0], # unique source - most_recent_beliefs_only=False, + most_recent_beliefs_only=True, **kwargs, ) - # todo: delete next line and set most_recent_beliefs_only=True when this is resolved: https://github.com/SeitaBV/timely-beliefs/pull/117 - previous_most_recent_beliefs_in_db = belief_utils.select_most_recent_belief( - previous_beliefs_in_db - ) compare_fields = ["event_start", "source", "cumulative_probability", "event_value"] a = bdf.reset_index().set_index(compare_fields) diff --git a/flexmeasures/data/services/users.py b/flexmeasures/data/services/users.py index d483eba1d..c00e1f8e5 100644 --- a/flexmeasures/data/services/users.py +++ b/flexmeasures/data/services/users.py @@ -35,43 +35,6 @@ def get_user(id: str) -> User: return user -def get_users( - account_name: str | None = None, - role_name: str | None = None, - account_role_name: str | None = None, - only_active: bool = True, -) -> list[User]: - """Return a list of User objects. - The role_name parameter allows to filter by role. - Set only_active to False if you also want non-active users. - """ - user_query = select(User) - - if account_name is not None: - account = db.session.execute( - select(Account).filter_by(name=account_name) - ).scalar_one_or_none() - if not account: - raise NotFound(f"There is no account named {account_name}!") - user_query = user_query.filter_by(account=account) - - if only_active: - user_query = user_query.filter(User.active.is_(True)) - - if role_name is not None: - role = db.session.execute( - select(Role).filter_by(name=role_name) - ).scalar_one_or_none() - if role: - user_query = user_query.filter(User.flexmeasures_roles.contains(role)) - - users = db.session.scalars(user_query).all() - if account_role_name is not None: - users = [u for u in users if u.account.has_role(account_role_name)] - - return users - - def find_user_by_email(user_email: str, keep_in_session: bool = True) -> User: user_datastore = SQLAlchemySessionUserDatastore(db.session, User, Role) user = user_datastore.find_user(email=user_email) diff --git a/flexmeasures/data/tests/test_SensorsToShowSchema.py b/flexmeasures/data/tests/test_SensorsToShowSchema.py new file mode 100644 index 000000000..c5eb9919e --- /dev/null +++ b/flexmeasures/data/tests/test_SensorsToShowSchema.py @@ -0,0 +1,125 @@ +import pytest +from marshmallow import ValidationError + +from flexmeasures.data.schemas.generic_assets import SensorsToShowSchema + + +def test_single_sensor_id(): + schema = SensorsToShowSchema() + input_value = [42] + expected_output = [{"title": None, "sensors": [42]}] + assert schema.deserialize(input_value) == expected_output + + +def test_list_of_sensor_ids(): + schema = SensorsToShowSchema() + input_value = [42, 43] + expected_output = [ + {"title": None, "sensors": [42]}, + {"title": None, "sensors": [43]}, + ] + assert schema.deserialize(input_value) == expected_output + + +def test_dict_with_title_and_single_sensor(): + schema = SensorsToShowSchema() + input_value = [{"title": "Temperature", "sensor": 42}] + expected_output = [{"title": "Temperature", "sensors": [42]}] + assert schema.deserialize(input_value) == expected_output + + +def test_dict_with_title_and_multiple_sensors(): + schema = SensorsToShowSchema() + input_value = [{"title": "Pressure", "sensors": [42, 43]}] + expected_output = [{"title": "Pressure", "sensors": [42, 43]}] + assert schema.deserialize(input_value) == expected_output + + +def test_invalid_sensor_string_input(): + schema = SensorsToShowSchema() + with pytest.raises( + ValidationError, + match="Invalid item type in 'sensors_to_show'. Expected int, list, or dict.", + ): + schema.deserialize(["invalid_string"]) + + +def test_invalid_sensor_in_list(): + schema = SensorsToShowSchema() + input_value = [{"title": "Test", "sensors": [42, "invalid"]}] + with pytest.raises( + ValidationError, match="'sensors' value must be a list of integers." + ): + schema.deserialize(input_value) + + +def test_invalid_sensor_dict_without_sensors_key(): + schema = SensorsToShowSchema() + input_value = [{"title": "Test", "something_else": 42}] + with pytest.raises( + ValidationError, + match="Dictionary must contain either 'sensor' or 'sensors' key.", + ): + schema.deserialize(input_value) + + +def test_mixed_valid_inputs(): + schema = SensorsToShowSchema() + input_value = [ + {"title": "Test", "sensors": [1, 2]}, + {"title": None, "sensors": [3, 4]}, + 5, + ] + expected_output = [ + {"title": "Test", "sensors": [1, 2]}, + {"title": None, "sensors": [3, 4]}, + {"title": None, "sensors": [5]}, + ] + assert schema.deserialize(input_value) == expected_output + + +def test_string_json_input(): + schema = SensorsToShowSchema() + input_value = ( + '[{"title": "Test", "sensors": [1, 2]}, {"title": "Test2", "sensors": [3]}]' + ) + expected_output = [ + {"title": "Test", "sensors": [1, 2]}, + {"title": "Test2", "sensors": [3]}, + ] + assert schema.deserialize(input_value) == expected_output + + +# New test cases for misspelled or missing title and mixed sensor/sensors formats + + +def test_dict_missing_title_key(): + schema = SensorsToShowSchema() + input_value = [{"sensor": 42}] + with pytest.raises(ValidationError, match="Dictionary must contain a 'title' key."): + schema.deserialize(input_value) + + +def test_dict_misspelled_title_key(): + schema = SensorsToShowSchema() + input_value = [{"titel": "Temperature", "sensor": 42}] # Misspelled 'title' + with pytest.raises(ValidationError, match="Dictionary must contain a 'title' key."): + schema.deserialize(input_value) + + +def test_dict_with_sensor_as_list(): + schema = SensorsToShowSchema() + input_value = [{"title": "Temperature", "sensor": [42]}] + with pytest.raises(ValidationError, match="'sensor' value must be an integer."): + schema.deserialize(input_value) + + +def test_dict_with_sensors_as_int(): + schema = SensorsToShowSchema() + input_value = [ + {"title": "Temperature", "sensors": 42} + ] # 'sensors' should be a list, not an int + with pytest.raises( + ValidationError, match="'sensors' value must be a list of integers." + ): + schema.deserialize(input_value) diff --git a/flexmeasures/data/tests/test_scheduling_repeated_jobs.py b/flexmeasures/data/tests/test_scheduling_repeated_jobs.py index 31acab44b..61bf29105 100644 --- a/flexmeasures/data/tests/test_scheduling_repeated_jobs.py +++ b/flexmeasures/data/tests/test_scheduling_repeated_jobs.py @@ -214,7 +214,7 @@ def test_scheduling_multiple_triggers( soc_targets = [dict(datetime=target_datetime.isoformat(), value=target_soc)] job = create_scheduling_job( - sensor=charging_station, + asset_or_sensor=charging_station, start=start, end=end, belief_time=start, diff --git a/flexmeasures/data/tests/test_scheduling_repeated_jobs_fresh_db.py b/flexmeasures/data/tests/test_scheduling_repeated_jobs_fresh_db.py index 8852e5e64..98e7533c0 100644 --- a/flexmeasures/data/tests/test_scheduling_repeated_jobs_fresh_db.py +++ b/flexmeasures/data/tests/test_scheduling_repeated_jobs_fresh_db.py @@ -65,7 +65,7 @@ def test_requeue_failing_job( for _ in range(2): job = create_scheduling_job( - sensor=charging_station, + asset_or_sensor=charging_station, start=start, end=end, resolution=resolution, diff --git a/flexmeasures/data/transactional.py b/flexmeasures/data/transactional.py index 315f3e5b3..9cf003298 100644 --- a/flexmeasures/data/transactional.py +++ b/flexmeasures/data/transactional.py @@ -2,6 +2,7 @@ These, and only these, functions should help you with treating your own code in the context of one database transaction. Which makes our lives easier. """ + from __future__ import annotations import sys @@ -84,7 +85,8 @@ def task_with_status_report(task_function, task_name: str | None = None): still needs to add to the session). If the task wants to commit partial results, and at the same time report that some things did not run well, it can raise a PartialTaskCompletionException and we recommend to use save-points (db.session.being_nested) to - do partial rollbacks (see https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html#using-savepoint).""" + do partial rollbacks (see https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html#using-savepoint). + """ task_name_to_report = ( task_name # store this closure var somewhere else before we might assign to it diff --git a/flexmeasures/data/utils.py b/flexmeasures/data/utils.py index 70ac77804..b07ce83f5 100644 --- a/flexmeasures/data/utils.py +++ b/flexmeasures/data/utils.py @@ -58,7 +58,7 @@ def get_data_source( def save_to_db( data: BeliefsDataFrame | BeliefsSeries | list[BeliefsDataFrame | BeliefsSeries], - bulk_save_objects: bool = False, + bulk_save_objects: bool = True, save_changed_beliefs_only: bool = True, ) -> str: """Save the timed beliefs to the database. diff --git a/flexmeasures/tests/utils.py b/flexmeasures/tests/utils.py index 5b303d46e..f999e2811 100644 --- a/flexmeasures/tests/utils.py +++ b/flexmeasures/tests/utils.py @@ -1,6 +1,6 @@ from __future__ import annotations -from sqlalchemy import select +from sqlalchemy import select, event from flexmeasures.data.models.time_series import Sensor @@ -10,3 +10,21 @@ def get_test_sensor(db) -> Sensor | None: select(Sensor).filter_by(name="epex_da") ).scalar_one_or_none() return sensor + + +class QueryCounter(object): + """Context manager to count SQLALchemy queries.""" + + def __init__(self, connection): + self.connection = connection.engine + self.count = 0 + + def __enter__(self): + event.listen(self.connection, "before_cursor_execute", self.callback) + return self + + def __exit__(self, *args, **kwargs): + event.remove(self.connection, "before_cursor_execute", self.callback) + + def callback(self, *args, **kwargs): + self.count += 1 diff --git a/flexmeasures/ui/__init__.py b/flexmeasures/ui/__init__.py index b065aed17..6a8007a35 100644 --- a/flexmeasures/ui/__init__.py +++ b/flexmeasures/ui/__init__.py @@ -23,6 +23,7 @@ from flexmeasures.utils.time_utils import ( localized_datetime_str, naturalized_datetime_str, + to_utc_timestamp, ) from flexmeasures.utils.app_utils import ( parse_config_entry_by_account_roles, @@ -46,6 +47,7 @@ def register_at(app: Flask): from flexmeasures.ui.crud.users import UserCrudUI from flexmeasures.ui.crud.accounts import AccountCrudUI from flexmeasures.ui.views.sensors import SensorUI + from flexmeasures.ui.utils.color_defaults import get_color_settings AssetCrudUI.register(app) UserCrudUI.register(app) @@ -60,6 +62,11 @@ def register_at(app: Flask): register_rq_dashboard(app) + # Injects Flexmeasures default colors into all templates + @app.context_processor + def inject_global_vars(): + return get_color_settings(None) + @app.route("/favicon.ico") def favicon(): return send_from_directory( @@ -111,10 +118,11 @@ def basic_admin_auth(): return # Logged-in users can view queues on the demo server, but only admins can view them on other servers - if app.config.get("FLEXMEASURES_MODE", "") == "demo": - rq_dashboard.blueprint.before_request(basic_auth) - else: - rq_dashboard.blueprint.before_request(basic_admin_auth) + if app.config.get("FLEXMEASURES_ENV") != "documentation": + if app.config.get("FLEXMEASURES_MODE", "") == "demo": + rq_dashboard.blueprint.before_request(basic_auth) + else: + rq_dashboard.blueprint.before_request(basic_admin_auth) # To set template variables, use set_global_template_variables in app.py app.register_blueprint(rq_dashboard.blueprint, url_prefix="/tasks") @@ -129,13 +137,14 @@ def add_jinja_filters(app): ) # Allow expression statements (e.g. for modifying lists) app.jinja_env.filters["localized_datetime"] = localized_datetime_str app.jinja_env.filters["naturalized_datetime"] = naturalized_datetime_str + app.jinja_env.filters["to_utc_timestamp"] = to_utc_timestamp app.jinja_env.filters["naturalized_timedelta"] = naturaldelta app.jinja_env.filters["capitalize"] = capitalize app.jinja_env.filters["pluralize"] = pluralize app.jinja_env.filters["parameterize"] = parameterize app.jinja_env.filters["isnull"] = pd.isnull - app.jinja_env.filters["hide_nan_if_desired"] = ( - lambda x: "" + app.jinja_env.filters["hide_nan_if_desired"] = lambda x: ( + "" if x in ("nan", "nan%", "NAN") and current_app.config.get("FLEXMEASURES_HIDE_NAN_IN_UI", False) else x @@ -143,12 +152,12 @@ def add_jinja_filters(app): app.jinja_env.filters["asset_icon"] = asset_icon_name app.jinja_env.filters["username"] = username app.jinja_env.filters["accountname"] = accountname - app.jinja_env.filters[ - "parse_config_entry_by_account_roles" - ] = parse_config_entry_by_account_roles - app.jinja_env.filters[ - "find_first_applicable_config_entry" - ] = find_first_applicable_config_entry + app.jinja_env.filters["parse_config_entry_by_account_roles"] = ( + parse_config_entry_by_account_roles + ) + app.jinja_env.filters["find_first_applicable_config_entry"] = ( + find_first_applicable_config_entry + ) def add_jinja_variables(app): diff --git a/flexmeasures/ui/crud/accounts.py b/flexmeasures/ui/crud/accounts.py index 98259ac9a..c9b2d8836 100644 --- a/flexmeasures/ui/crud/accounts.py +++ b/flexmeasures/ui/crud/accounts.py @@ -1,13 +1,17 @@ from __future__ import annotations from sqlalchemy import select +from werkzeug.exceptions import Forbidden, Unauthorized from flask import request, url_for from flask_classful import FlaskView from flask_security import login_required +from flask_security.core import current_user + +from flexmeasures.auth.policy import user_has_admin_access, check_access + from flexmeasures.ui.crud.api_wrapper import InternalApi from flexmeasures.ui.utils.view_utils import render_flexmeasures_template -from flexmeasures.ui.crud.assets import get_assets_by_account -from flexmeasures.ui.crud.users import get_users_by_account +from flexmeasures.data.models.audit_log import AuditLog from flexmeasures.data.models.user import Account from flexmeasures.data import db @@ -34,46 +38,47 @@ class AccountCrudUI(FlaskView): @login_required def index(self): """/accounts""" - accounts = get_accounts() - for account in accounts: - account["asset_count"] = len(get_assets_by_account(account["id"])) - account["user_count"] = len(get_users_by_account(account["id"])) return render_flexmeasures_template( "crud/accounts.html", - accounts=accounts, ) @login_required def get(self, account_id: str): """/accounts/""" include_inactive = request.args.get("include_inactive", "0") != "0" - account = get_account(account_id) - if account["consultancy_account_id"]: + account = db.session.execute(select(Account).filter_by(id=account_id)).scalar() + if account.consultancy_account_id: consultancy_account = db.session.execute( - select(Account).filter_by(id=account["consultancy_account_id"]) + select(Account).filter_by(id=account.consultancy_account_id) ).scalar_one_or_none() if consultancy_account: - account["consultancy_account_name"] = consultancy_account.name - assets = get_assets_by_account(account_id) - assets += get_assets_by_account(account_id=None) - users = get_users_by_account(account_id, include_inactive=include_inactive) + account.consultancy_account.name = consultancy_account.name + accounts = get_accounts() if user_has_admin_access(current_user, "read") else [] + + user_can_view_account_auditlog = True + try: + check_access(AuditLog.account_table_acl(account), "read") + except (Forbidden, Unauthorized): + user_can_view_account_auditlog = False + return render_flexmeasures_template( "crud/account.html", account=account, - assets=assets, - users=users, + accounts=accounts, include_inactive=include_inactive, + can_view_account_auditlog=user_can_view_account_auditlog, ) @login_required def auditlog(self, account_id: str): """/accounts/auditlog/""" - account = get_account(account_id) + account = db.session.execute(select(Account).filter_by(id=account_id)).scalar() audit_log_response = InternalApi().get( url_for("AccountAPI:auditlog", id=account_id) ) audit_logs_response = audit_log_response.json() + return render_flexmeasures_template( "crud/account_audit_log.html", audit_logs=audit_logs_response, diff --git a/flexmeasures/ui/crud/assets/utils.py b/flexmeasures/ui/crud/assets/utils.py index 2864b57c0..9056558ce 100644 --- a/flexmeasures/ui/crud/assets/utils.py +++ b/flexmeasures/ui/crud/assets/utils.py @@ -104,15 +104,24 @@ def expunge_asset(): if make_obj: children = asset_data.pop("child_assets", []) + asset_data.pop("sensors", []) + asset_data.pop("owner", None) + asset_type = asset_data.pop("generic_asset_type", {}) + asset = GenericAsset( **{ **asset_data, **{"attributes": json.loads(asset_data.get("attributes", "{}"))}, } ) # TODO: use schema? - asset.generic_asset_type = db.session.get( - GenericAssetType, asset.generic_asset_type_id - ) + if "generic_asset_type_id" in asset_data: + asset.generic_asset_type = db.session.get( + GenericAssetType, asset_data["generic_asset_type_id"] + ) + else: + asset.generic_asset_type = db.session.get( + GenericAssetType, asset_type.get("id", None) + ) expunge_asset() asset.owner = db.session.get(Account, asset_data["account_id"]) expunge_asset() @@ -132,7 +141,9 @@ def expunge_asset(): child_assets = [] for child in children: - child.pop("child_assets") + if "child_assets" in child: + # not deeper than one level + child.pop("child_assets") child_asset = process_internal_api_response(child, child["id"], True) child_assets.append(child_asset) asset.child_assets = child_assets diff --git a/flexmeasures/ui/crud/assets/views.py b/flexmeasures/ui/crud/assets/views.py index 011b9c8e8..c4453d7d8 100644 --- a/flexmeasures/ui/crud/assets/views.py +++ b/flexmeasures/ui/crud/assets/views.py @@ -1,9 +1,8 @@ +from __future__ import annotations from flask import url_for, current_app, request from flask_classful import FlaskView, route from flask_security import login_required, current_user from webargs.flaskparser import use_kwargs -from sqlalchemy import select -from flexmeasures.auth.policy import user_has_admin_access from flexmeasures.data import db from flexmeasures.auth.error_handling import unauthorized_handler @@ -13,6 +12,7 @@ GenericAsset, get_center_location_of_assets, ) +from flexmeasures.ui.utils.view_utils import ICON_MAPPING from flexmeasures.data.models.user import Account from flexmeasures.ui.utils.view_utils import render_flexmeasures_template from flexmeasures.ui.crud.api_wrapper import InternalApi @@ -21,7 +21,6 @@ process_internal_api_response, user_can_create_assets, user_can_delete, - get_assets_by_account, ) from flexmeasures.data.services.sensors import ( build_sensor_status_data, @@ -50,23 +49,15 @@ class AssetCrudUI(FlaskView): trailing_slash = False @login_required - def index(self, msg=""): + def index(self, msg="", **kwargs): """GET from /assets List the user's assets. For admins, list across all accounts. """ - assets = [] - - if user_has_admin_access(current_user, "read"): - for account in db.session.scalars(select(Account)).all(): - assets += get_assets_by_account(account.id) - assets += get_assets_by_account(account_id=None) - else: - assets = get_assets_by_account(current_user.account_id) return render_flexmeasures_template( "crud/assets.html", - assets=assets, + asset_icon_map=ICON_MAPPING, message=msg, user_can_create_assets=user_can_create_assets(), ) diff --git a/flexmeasures/ui/crud/users.py b/flexmeasures/ui/crud/users.py index acc3c720b..282d97afa 100644 --- a/flexmeasures/ui/crud/users.py +++ b/flexmeasures/ui/crud/users.py @@ -4,16 +4,18 @@ from flask import request, url_for from flask_classful import FlaskView -from flask_login import current_user +from flask_security.core import current_user from flask_security import login_required from flask_wtf import FlaskForm from wtforms import StringField, FloatField, DateTimeField, BooleanField from wtforms.validators import DataRequired +from werkzeug.exceptions import Forbidden, Unauthorized from sqlalchemy import select -from flexmeasures.auth.policy import ADMIN_READER_ROLE, ADMIN_ROLE +from flexmeasures.auth.policy import ADMIN_READER_ROLE, ADMIN_ROLE, check_access from flexmeasures.auth.decorators import roles_required, roles_accepted from flexmeasures.data import db +from flexmeasures.data.models.audit_log import AuditLog from flexmeasures.data.models.user import User, Role, Account from flexmeasures.data.services.users import ( get_user, @@ -39,8 +41,17 @@ class UserForm(FlaskForm): def render_user(user: User | None, asset_count: int = 0, msg: str | None = None): user_form = UserForm() user_form.process(obj=user) + + user_view_user_auditlog = True + try: + check_access(AuditLog.user_table_acl(current_user), "read") + except (Forbidden, Unauthorized): + user_view_user_auditlog = False + return render_flexmeasures_template( "crud/user.html", + can_view_user_auditlog=user_view_user_auditlog, + logged_in_user=current_user, user=user, user_form=user_form, asset_count=asset_count, @@ -75,20 +86,14 @@ def process_internal_api_response( return user_data -def get_users_by_account( - account_id: int | str, include_inactive: bool = False -) -> list[User]: +def get_all_users(include_inactive: bool = False) -> list[User]: get_users_response = InternalApi().get( url_for( "UserAPI:index", - account_id=account_id, include_inactive=include_inactive, ) ) - users = [ - process_internal_api_response(user, make_obj=True) - for user in get_users_response.json() - ] + users = [user for user in get_users_response.json()] return users @@ -100,17 +105,8 @@ class UserCrudUI(FlaskView): def index(self): """/users""" include_inactive = request.args.get("include_inactive", "0") != "0" - users = [] - if current_user.has_role(ADMIN_ROLE) or current_user.has_role( - ADMIN_READER_ROLE - ): - accounts = db.session.scalars(select(Account)).all() - else: - accounts = [current_user.account] - for account in accounts: - users += get_users_by_account(account.id, include_inactive) return render_flexmeasures_template( - "crud/users.html", users=users, include_inactive=include_inactive + "crud/users.html", include_inactive=include_inactive ) @login_required diff --git a/flexmeasures/ui/static/css/flexmeasures.css b/flexmeasures/ui/static/css/flexmeasures.css index a96dd358f..84d6b84a0 100644 --- a/flexmeasures/ui/static/css/flexmeasures.css +++ b/flexmeasures/ui/static/css/flexmeasures.css @@ -1,37 +1,7 @@ /* Styling for the FlexMeasures Platform */ - -:root { - /* colors by name */ - --primary-color: #1a3443; - --primary-border-color: #102029; - --primary-hover-color: #152b38; - --primary-transparent: rgba(26, 52, 67, .2); - --secondary-color: #f1a122; - --secondary-hover-color: #ff9d00; - --secondary-transparent: rgba(241, 161, 34, .2); - --white: #FFF; - --black: #000; - --light-gray: #eeeeee; - --gray: #bbb; - --red: #c21431; - --green: #14c231; - - /* colors by function */ - --nav-default-color: var(--white); - --nav-default-background-color: var(--primary-color); - --nav-hover-color: var(--secondary-hover-color); - --nav-hover-background-color: var(--primary-hover-color); - --nav-open-color: var(--secondary-color); - --nav-open-background-color: var(--primary-color); - --nav-current-color: var(--black); - --nav-current-background-color: var(--secondary-color); - --nav-current-hover-color: var(--black); - --nav-current-hover-background-color: var(--secondary-hover-color); - --create-color: var(--green); - --delete-color: var(--red); -} +/* /* devanagari */ @font-face { @@ -348,6 +318,7 @@ p.error { .dropdown-menu>li>a:not(.active) { color: var(--nav-default-color); background-color: var(--nav-default-background-color); + text-align: center; } .dropdown-menu>li>a.active { color: var(--nav-hover-color); @@ -450,30 +421,6 @@ input:checked + .slider:before { /* ---- Carousel ---- */ -.carousel-container { - width: 500px; - height: 500px; - overflow: hidden; - position: relative; - padding:0; - border-radius: 25px; -} -.carousel { - width: 500px; - position: absolute; - left: 50%; - top: 0; - margin-left: -250px; -} -.carousel .item img { - display: block; - margin: 0 auto; - border-radius: 25px; -} -.carousel-control.left, .carousel-control.right { - background:transparent; -} - .carousel-inner img { width: 100%; } @@ -1043,6 +990,19 @@ body .dataTables_wrapper .dataTables_paginate .paginate_button.current:hover { .sidepanel label { color: var(--nav-default-color) !important; } + .leftside-dropdown { + margin-left: -95px !important; + color: var(--nav-default-color); + background: var(--nav-default-background-color); + width: calc(var(--litepicker-day-width) * 6); + padding: 13px 10px; + min-height: 35px; // line-height + padding-top + padding-bottom + border-top-left-radius: 0px; + border-bottom-left-radius: 0px; + border-top-right-radius: 15px; + border-bottom-right-radius: 15px; + text-align: right; + } .sidepanel-container { z-index: 20; position: relative; @@ -1054,7 +1014,7 @@ body .dataTables_wrapper .dataTables_paginate .paginate_button.current:hover { width: calc(var(--litepicker-day-width) * 9); margin: 15px; padding: 20px 15px; - min-height: 60px; // line-height + padding-top + padding-bottom + min-height: 60px; /* line-height + padding-top + padding-bottom */ transition: .3s; -webkit-transition: .3s; -moz-transition: .3s; @@ -1357,33 +1317,37 @@ body .dataTables_wrapper .dataTables_paginate .paginate_button.current:hover { float: none !important; } -.header-action-button>div button { +.header-action-button>div button, +.header-action-button>div label { font-size: 15px; padding: 15px 30px; - border: none; line-height: 16px; - outline: none !important; - box-shadow: none; - border-radius: 0px; + border-radius: 0px !important; text-align: center; width: 100%; - background: var(--primary-color); - border-top-left-radius: 0; - border-top-right-radius: 0; - border-bottom-left-radius: 0; - border-bottom-right-radius: 0; - border-right: 1px solid var(--primary-border-color) !important; - -webkit-border-radius: 0px; - -moz-border-radius: 0px; - -ms-border-radius: 0px; - -o-border-radius: 0px; + border-top-left-radius: 0 !important; + border-top-right-radius: 0 !important; + border-bottom-left-radius: 0 !important; + border-bottom-right-radius: 0 !important; + -webkit-border-radius: 0px !important; + -moz-border-radius: 0px !important; + -ms-border-radius: 0px !important; + -o-border-radius: 0px !important; transition: .4s; -webkit-transition: .4s; -moz-transition: .4s; -ms-transition: .4s; -o-transition: .4s; } -.rq-style.header-action-button>div button { +.header-action-button>div button { + border: none; + outline: none !important; + box-shadow: none; + background: var(--primary-color); + border-right: 1px solid var(--primary-border-color) !important; +} +.rq-style.header-action-button>div button, +.rq-style.header-action-button>div label { padding: 15px 25px; } @@ -1391,15 +1355,19 @@ body .dataTables_wrapper .dataTables_paginate .paginate_button.current:hover { background: var(--primary-hover-color) !important; } -.header-action-button>div:first-child button { - border-top-left-radius: 15px; - border-bottom-left-radius: 15px; +.header-action-button>div:first-child button, +.header-action-button>div:first-child label { + border-top-left-radius: 15px !important; + border-bottom-left-radius: 15px !important; } +.header-action-button>div:last-child button, +.header-action-button>div:last-child label { + border-top-right-radius: 15px !important; + border-bottom-right-radius: 15px !important; +} .header-action-button>div:last-child button { border-right: none; - border-top-right-radius: 15px; - border-bottom-right-radius: 15px; } .create-button:hover { @@ -1417,7 +1385,8 @@ body .dataTables_wrapper .dataTables_paginate .paginate_button.current:hover { width: 100%; } - .header-action-button>div button { + .header-action-button>div button, + .header-action-button>div label { margin-bottom: 15px; border-radius: 15px !important; -webkit-border-radius: 15px !important; @@ -1949,4 +1918,3 @@ body.touched [title]:hover:after { .table-responsive{ padding: 20px; } - diff --git a/flexmeasures/ui/static/images/flexmeasures-preview.jpg b/flexmeasures/ui/static/images/flexmeasures-preview.jpg index e1e19a646..83bcb66b7 100644 Binary files a/flexmeasures/ui/static/images/flexmeasures-preview.jpg and b/flexmeasures/ui/static/images/flexmeasures-preview.jpg differ diff --git a/flexmeasures/ui/static/js/flexmeasures.js b/flexmeasures/ui/static/js/flexmeasures.js index 8bad98c24..021f19f4e 100644 --- a/flexmeasures/ui/static/js/flexmeasures.js +++ b/flexmeasures/ui/static/js/flexmeasures.js @@ -65,8 +65,13 @@ function clickableTable(element, urlColumn) { if (euclidean <= radiusLimit) { var columnIndex = table.column(':contains(' + urlColumn + ')').index(); + var data = table.row(this).data(); - var url = data[columnIndex]; + if(Array.isArray(data)){ + var url = data[columnIndex]; + } else{ + var url = data["url"]; + } handleClick(event, url); } } diff --git a/flexmeasures/ui/templates/admin/logged_in_user.html b/flexmeasures/ui/templates/admin/logged_in_user.html index 22c195f4f..a3ebccf40 100644 --- a/flexmeasures/ui/templates/admin/logged_in_user.html +++ b/flexmeasures/ui/templates/admin/logged_in_user.html @@ -14,7 +14,7 @@
- +
@@ -24,7 +24,7 @@

User Overview

Logged-in user: {{ logged_in_user.username }}
- +
@@ -32,7 +32,7 @@

User Overview

- + {% if account_roles %} @@ -76,17 +76,21 @@

User Overview

+ {% if can_view_account_auditlog %}
- + {% endif %} + {% if can_view_user_auditlog %}
- + {% endif %}
-{% endblock %} +{% endblock %} \ No newline at end of file diff --git a/flexmeasures/ui/templates/admin/login_user.html b/flexmeasures/ui/templates/admin/login_user.html index 524c62419..9c9d43169 100644 --- a/flexmeasures/ui/templates/admin/login_user.html +++ b/flexmeasures/ui/templates/admin/login_user.html @@ -10,94 +10,105 @@ {% block divs %}
-
+
+ {% block platform_title %} +

The FlexMeasures Platform

+ {% endblock platform_title %} {% include "security/_messages.html" %}
-
Email address
Account{{ logged_in_user.account.name }}{{ logged_in_user.account.name }}
- - - - - - - - - - {% if account.consultancy_account_name %} - - - - - {% endif %} - -
- ID - - {{ account.id }} -
- Roles - - {{ account.account_roles | map(attribute='name') | join(", ") }} -
- Consultancy - - {{ account.consultancy_account_name }} -
-
- -
-

Users

- -
-
- +
+
+
+
+ Edit Account +
+
+ +
+
+

Edit {{ account.name }}

+ Owned by account: {{ account.name }} (ID: {{ account.id + }}) + +
+
+
+ +
- -
- - - - - - - - - - - - - - - {% for user in users %} - - - - - - - - - - - {% endfor %} - -
UsernameEmailRolesTimezoneLast LoginLast SeenActiveURL
- {{ user.username }} - - {{ user.email }} - {% for role in user.flexmeasures_roles %} - {{ role.name }}{{ "," if not loop.last }} - {% endfor %} - - {{ user.timezone }} - - {{ user.last_login_at | naturalized_datetime}} - - {{ user.last_seen_at | naturalized_datetime}} - - {{ user.active }} - - /users/{{ user.id }} -
-
-
-
-

Assets

-
- - - - - - - - - - - - - - - {% for asset in assets %} - - - - - - - - - - + + +
+
+ + + +
+
+ +
+
+ + + +
+
+ +
+
+ + + +
+
+ + {% if user_is_admin %} +
+
+ +
-
NameLocationAsset IDAccountSensorsStatus - {% if user_can_create_assets %} -
- -
- {% endif %} - 		URL
- {{ asset.name }} - - {% if asset.latitude and asset.longitude %} - LAT: {{ "{:,.4f}".format( asset.latitude ) }} LONG: - {{ "{:,.4f}".format( asset.longitude ) }} - {% endif %} - - {{ asset.id }} - - {% if asset.owner %} - {{ asset.owner.name }} - {% else %} - PUBLIC - {% endif %} - - {{ asset.sensors | length }} - - - - - - - /assets/{{ asset.id }} -
-
-
+ +
+
+ {% endif %} +
+ + +
+ +
+
+
+
+

Account

+ Account: {{ account.name }} +
+ + + + + + + + + + + {% if account.consultancy_account_name %} + + + + + {% endif %} {% if account.primary_color %} + + + + + {% endif %} {% if account.secondary_color %} + + + + + {% endif %} {% if account.logo_url %} + + + + + {% endif %} + +
ID{{ account.id }}
Roles + {{ account.account_roles | map(attribute='name') | join(", ") + }} +
Consultancy{{ account.consultancy_account_name }}
Primary Color +
+
Secondary Color +
+
Logo URL + Logo +
+
+
+
+

All users

+
+ +
+
+ +
+
+
+

Assets

+
+ +
+
+
+
+
-
-{% block paginate_tables_script %} {{ super() }} {% endblock %} -{% endblock %} + + + +{% block paginate_tables_script %} {{ super() }} {% endblock %} {% endblock %} diff --git a/flexmeasures/ui/templates/crud/account_audit_log.html b/flexmeasures/ui/templates/crud/account_audit_log.html index 191bb12f4..05ea204d6 100644 --- a/flexmeasures/ui/templates/crud/account_audit_log.html +++ b/flexmeasures/ui/templates/crud/account_audit_log.html @@ -2,7 +2,7 @@ {% set active_page = "accounts" %} -{% block title %} Account {{ account.name }} actions history {% endblock %} +{% block title %} Account {{ account.name }} actions history {% endblock %} {% block divs %} @@ -11,23 +11,27 @@

History of actions for account {{ account.name }}

- + - + + {% for audit_log in audit_logs: %} - + @@ -42,8 +46,16 @@

History of actions for account {{ accou {% endblock %} \ No newline at end of file diff --git a/flexmeasures/ui/templates/crud/accounts.html b/flexmeasures/ui/templates/crud/accounts.html index bcc365824..b8cbd952d 100644 --- a/flexmeasures/ui/templates/crud/accounts.html +++ b/flexmeasures/ui/templates/crud/accounts.html @@ -11,43 +11,9 @@

All Accounts

-

- - - - - - - - - - - - {% for account in accounts %} - - - - - - - - - {% endfor %} - -
AccountIDAssetsUsersRolesURL
- {{ account.name }} - - {{ account.id }} - - {{ account.asset_count }} - - {{ account.user_count }} - - {{ account.account_roles | map(attribute='name') | join(", ") }} - - /accounts/{{ account.id }} -
-
+ + +
@@ -55,4 +21,78 @@

All Accounts + + {% block paginate_tables_script %} {{ super() }} {% endblock %} {% endblock%} \ No newline at end of file diff --git a/flexmeasures/ui/templates/crud/asset.html b/flexmeasures/ui/templates/crud/asset.html index 5cc03fb87..973fca5f4 100644 --- a/flexmeasures/ui/templates/crud/asset.html +++ b/flexmeasures/ui/templates/crud/asset.html @@ -50,7 +50,7 @@ {% if user_can_delete_asset %}
- +
+ + +{% block paginate_tables_script %} {{ super() }} {% endblock %} {% endblock %} diff --git a/flexmeasures/ui/templates/crud/asset_audit_log.html b/flexmeasures/ui/templates/crud/asset_audit_log.html index 8a804f718..6ca5a8a1d 100644 --- a/flexmeasures/ui/templates/crud/asset_audit_log.html +++ b/flexmeasures/ui/templates/crud/asset_audit_log.html @@ -10,9 +10,10 @@

History of actions for asset {{ asset.name }}

- + + @@ -21,6 +22,9 @@

History of actions for asset {{ asset.name

{% for audit_log in audit_logs: %} + @@ -40,8 +44,16 @@

History of actions for asset {{ asset.name {% endblock %} \ No newline at end of file diff --git a/flexmeasures/ui/templates/crud/assets.html b/flexmeasures/ui/templates/crud/assets.html index be7ba6260..6e46aa79b 100644 --- a/flexmeasures/ui/templates/crud/assets.html +++ b/flexmeasures/ui/templates/crud/assets.html @@ -6,6 +6,77 @@ {% block divs %} + +
@@ -27,54 +98,7 @@

Asset overview {% endif %}

-

- - - - - - - - - - - - - {% for asset in assets: %} - - - - - - - - - - {% endfor %} - -
NameLocationAsset IDAccountSensorsStatusURL
- {{ asset.name }} - - {% if asset.latitude and asset.longitude %} - LAT: {{ "{:,.4f}".format( asset.latitude ) }} LONG: {{ "{:,.4f}".format( asset.longitude ) }} - {% endif %} - - {{ asset.id }} - - {% if asset.owner %} - {{ asset.owner.name }} - {% else %} - PUBLIC - {% endif %} - - {{ asset.sensors | length }} - - - - - - /assets/{{ asset.id }} -
+
diff --git a/flexmeasures/ui/templates/crud/user.html b/flexmeasures/ui/templates/crud/user.html index 49e754ee4..614f759eb 100644 --- a/flexmeasures/ui/templates/crud/user.html +++ b/flexmeasures/ui/templates/crud/user.html @@ -12,6 +12,7 @@
+ {% if current_user.has_role('admin') or current_user.has_role('account-admin') %}
+ {% endif %}
@@ -105,7 +107,14 @@

User overview

-
+
+ {% if can_view_user_auditlog %} +
+ +
+ {% endif %} +
diff --git a/flexmeasures/ui/templates/crud/user_audit_log.html b/flexmeasures/ui/templates/crud/user_audit_log.html index 5e2927d6a..7cb66314e 100644 --- a/flexmeasures/ui/templates/crud/user_audit_log.html +++ b/flexmeasures/ui/templates/crud/user_audit_log.html @@ -11,9 +11,10 @@

History of actions for user {{ user.username }}

- + + @@ -22,6 +23,9 @@

History of actions for user {{ user.username

{% for audit_log in audit_logs: %} + @@ -42,8 +46,16 @@

History of actions for user {{ user.username {% endblock %} diff --git a/flexmeasures/ui/templates/crud/users.html b/flexmeasures/ui/templates/crud/users.html index 529d77f34..6bc6aca09 100644 --- a/flexmeasures/ui/templates/crud/users.html +++ b/flexmeasures/ui/templates/crud/users.html @@ -1,82 +1,36 @@ -{% extends "base.html" %} - -{% set active_page = "users" %} - -{% block title %} User listing {% endblock %} - +{% extends "base.html" %} +{% set active_page = "users" %} +{% block title %} User listing {% endblock %} {% block divs %}
-
-
+
+
-

All {% if not include_inactive %}active {% endif %}users

-
-
- -
- +

All users

+ +
+ +
-

- - - - - - - - - - - - - - - {% for user in users %} - - - - - - - - - - - - {% endfor %} - -
UsernameEmailRolesAccountTimezoneLast LoginLast SeenActiveURL
- {{ user.username }} - - {{ user.email }} - - {% for role in user.flexmeasures_roles %} - {{ role.name }}{{ "," if not loop.last }} - {% endfor %} - - {{ user.account.name }} - - {{ user.timezone }} - - {{ user.last_login_at | naturalized_datetime}} - - {{ user.last_seen_at | naturalized_datetime}} - - {{ user.active }} - - /users/{{ user.id }} -
+
+
-
-{% block paginate_tables_script %} {{ super() }} {% endblock %} - -{% endblock%} +{% block paginate_tables_script %} {{ super() }} {% endblock %} {% endblock%} diff --git a/flexmeasures/ui/templates/rq_dashboard/base.html b/flexmeasures/ui/templates/rq_dashboard/base.html index 3abc4854b..d7cdede76 100644 --- a/flexmeasures/ui/templates/rq_dashboard/base.html +++ b/flexmeasures/ui/templates/rq_dashboard/base.html @@ -73,6 +73,7 @@

Do you really want to ?

{% endblock scripts %} diff --git a/flexmeasures/ui/templates/rq_dashboard/update_links.js b/flexmeasures/ui/templates/rq_dashboard/update_links.js new file mode 100644 index 000000000..67294d71d --- /dev/null +++ b/flexmeasures/ui/templates/rq_dashboard/update_links.js @@ -0,0 +1,26 @@ +{#- Override sort order to prefer looking at the most recent past first, workaround for https://github.com/Parallels/rq-dashboard/issues/123 -#} +function updateLinks() { + document.querySelectorAll('td.failed a, td.narrow a').forEach(link => { + const href = link.getAttribute('href'); + if (href.includes('/canceled/') || href.includes('/failed/') || href.includes('/finished/')) { + const newHref = href.replace('/asc/', '/dsc/'); + link.setAttribute('href', newHref); + } + }); +} + +// Function to check if the table is done loading +function checkTableAndUpdate(mutationsList, observer) { + const loadingRow = document.querySelector('tbody tr'); + if (loadingRow && loadingRow.textContent.trim() !== 'Loading...') { + updateLinks(); + observer.disconnect(); // Stop observing once the table is updated + } +} + +// Monitor the tbody for changes +const observer = new MutationObserver(checkTableAndUpdate); + +// Start observing the tbody element for child list changes +const tbody = document.querySelector('tbody'); +observer.observe(tbody, { childList: true, subtree: true }); diff --git a/flexmeasures/ui/templates/views/sensors.html b/flexmeasures/ui/templates/views/sensors.html index 599f736bc..9b58e0252 100644 --- a/flexmeasures/ui/templates/views/sensors.html +++ b/flexmeasures/ui/templates/views/sensors.html @@ -25,8 +25,6 @@ {% endfor %} - -
@@ -34,23 +32,24 @@
-
+
Select dates
-
+
@@ -58,6 +57,149 @@ Loading...
+
+
+ +
Properties
+ + + + + + + + + + + + + + + + + + + + + + +
Name{{ sensor.name }}
Unit{{ sensor.unit }}
Event resolution{{ sensor.event_resolution }}
Timezone{{ sensor.timezone }}
Knowledge horizon type{{ sensor.knowledge_horizon_fnc }}
+
+
+ +
Statistics
+ + + +
+ + + +
+ There is no data for this sensor yet. +
+ +
+ There was a problem fetching statistics for this sensor's data. +
+ +
+
@@ -93,7 +235,7 @@ endDate = encodeURIComponent(toIsoString(endDate.toJSDate())); var base_url = window.location.href.split("?")[0]; var new_url = `${base_url}?start_time=${startDate}&end_time=${endDate}`; - + // change current url without reloading the page window.history.pushState({}, null, new_url); }); diff --git a/flexmeasures/ui/templates/views/status.html b/flexmeasures/ui/templates/views/status.html index 092958868..edc0fb3f8 100644 --- a/flexmeasures/ui/templates/views/status.html +++ b/flexmeasures/ui/templates/views/status.html @@ -20,7 +20,7 @@

Data connectivity for sensors of {{ asset.name }}

- +
@@ -62,21 +62,24 @@

Data connectivity for sensors of {{ asset.name }}

Name

Latest jobs of {{ asset.name }}

- + + - {% for job_data in jobs_data: %} + @@ -122,9 +125,6 @@ - {% endfor %} @@ -137,10 +137,18 @@ - + {% endblock %} diff --git a/flexmeasures/ui/tests/conftest.py b/flexmeasures/ui/tests/conftest.py index d2b78f811..0a3f080e6 100644 --- a/flexmeasures/ui/tests/conftest.py +++ b/flexmeasures/ui/tests/conftest.py @@ -2,6 +2,7 @@ from flexmeasures.data.services.users import create_user from flexmeasures.ui.tests.utils import login, logout +from flexmeasures import Asset @pytest.fixture(scope="function") @@ -41,3 +42,21 @@ def setup_ui_test_data( account_name=setup_accounts["Prosumer"].name, user_roles=dict(name="admin", description="A site admin."), ) + + +@pytest.fixture +def assets_prosumer(db, setup_accounts, setup_generic_asset_types): + assets = [] + for name in ["TestAsset", "TestAsset2"]: + asset = Asset( + name=name, + generic_asset_type=setup_generic_asset_types["battery"], + owner=setup_accounts["Prosumer"], + latitude=70.4, + longitude=30.9, + ) + assets.append(asset) + + db.session.add_all(assets) + + return assets diff --git a/flexmeasures/ui/tests/test_asset_crud.py b/flexmeasures/ui/tests/test_asset_crud.py index 2e2c8aa3f..47048c91a 100644 --- a/flexmeasures/ui/tests/test_asset_crud.py +++ b/flexmeasures/ui/tests/test_asset_crud.py @@ -16,38 +16,18 @@ def test_assets_page_empty(db, client, requests_mock, as_prosumer_user1): - requests_mock.get(f"{api_path_assets}?account_id=1", status_code=200, json={}) - requests_mock.get(f"{api_path_assets}/public", status_code=200, json={}) + requests_mock.get(f"{api_path_assets}", status_code=200, json=[]) + requests_mock.get(f"{api_path_assets}/public", status_code=200, json=[]) asset_index = client.get(url_for("AssetCrudUI:index"), follow_redirects=True) assert asset_index.status_code == 200 def test_get_assets_by_account(db, client, requests_mock, as_prosumer_user1): mock_assets = mock_asset_response(multiple=True) - requests_mock.get( - f"{api_path_assets}?account_id=1", status_code=200, json=mock_assets - ) + requests_mock.get(f"{api_path_assets}", status_code=200, json=mock_assets) assert get_assets_by_account(1)[1].name == "TestAsset2" -@pytest.mark.parametrize("use_owned_by", [False, True]) -def test_assets_page_nonempty( - db, client, requests_mock, as_prosumer_user1, use_owned_by -): - mock_assets = mock_asset_response(multiple=True) - requests_mock.get( - f"{api_path_assets}?account_id=1", status_code=200, json=mock_assets - ) - if use_owned_by: - asset_index = client.get( - url_for("AssetCrudUI:owned_by", account_id=mock_assets[0]["account_id"]) - ) - else: - asset_index = client.get(url_for("AssetCrudUI:index")) - for asset in mock_assets: - assert asset["name"].encode() in asset_index.data - - def test_new_asset_page(client, setup_assets, as_admin): asset_page = client.get(url_for("AssetCrudUI:get", id="new"), follow_redirects=True) assert asset_page.status_code == 200 @@ -75,6 +55,7 @@ def test_asset_page(db, client, setup_assets, requests_mock, as_prosumer_user1): assert ("Edit %s" % mock_asset["name"]).encode() in asset_page.data assert str(mock_asset["latitude"]).encode() in asset_page.data assert str(mock_asset["longitude"]).encode() in asset_page.data + print("asset_page.data:\n%s" % asset_page.data) assert ( "storeStartDate = new Date('2022-10-01T00:00:00+02:00')".encode() in asset_page.data @@ -152,6 +133,10 @@ def test_add_asset(db, client, setup_assets, requests_mock, as_admin): """Add a new asset""" user = find_user_by_email("test_prosumer_user@seita.nl") mock_asset = mock_asset_response(account_id=user.account.id, as_list=False) + del mock_asset[ + "generic_asset_type" + ] # API gives back more info here than a POST sends + mock_asset["generic_asset_type_id"] = 1 requests_mock.post(api_path_assets, status_code=201, json=mock_asset) response = client.post( url_for("AssetCrudUI:post", id="create"), diff --git a/flexmeasures/ui/tests/test_assets_forms.py b/flexmeasures/ui/tests/test_assets_forms.py index 15aac009e..51fdec681 100644 --- a/flexmeasures/ui/tests/test_assets_forms.py +++ b/flexmeasures/ui/tests/test_assets_forms.py @@ -79,7 +79,7 @@ def test_with_price_sensors( return_value=allowed_price_sensor_data, ) as mock_method: form.with_price_sensors(new_asset_decorator.test_battery, 1) - assert mock_method.called_once_with(1) + mock_method.assert_called_once_with(1) # check production_price_sensor only as consumption_price is processed the same way assert form.production_price_sensor_id.choices == expect_choices assert form.production_price_sensor_id.default == expect_default @@ -155,6 +155,6 @@ def test_with_inflexible_sensors( return_value=allowed_inflexible_sensor_data, ) as mock_method: form.with_inflexible_sensors(new_asset_decorator.test_battery, 1) - assert mock_method.called_once_with(1) + mock_method.assert_called_once_with(1) assert form.inflexible_device_sensor_ids.choices == expect_choices assert form.inflexible_device_sensor_ids.default == expect_default diff --git a/flexmeasures/ui/tests/test_color_defaults.py b/flexmeasures/ui/tests/test_color_defaults.py new file mode 100644 index 000000000..0c5bde208 --- /dev/null +++ b/flexmeasures/ui/tests/test_color_defaults.py @@ -0,0 +1,12 @@ +from flexmeasures.ui.utils.color_defaults import darken_color, lighten_color, rgba_color + + +def test_color_utils() -> None: + primary_color: str = "#1a3443" + secondary_color: str = "f1a122" + + assert darken_color(primary_color, 7) == "#18303e" + assert darken_color(primary_color, 4) == "#183140" + assert rgba_color(primary_color, 0.2) == "rgba(26, 52, 67, 0.2)" + assert lighten_color(secondary_color, 4) == "#f1a42a" + assert rgba_color(secondary_color, 0.2) == "rgba(241, 161, 34, 0.2)" diff --git a/flexmeasures/ui/tests/test_user_crud.py b/flexmeasures/ui/tests/test_user_crud.py index 62e075c5c..1a27a7de0 100644 --- a/flexmeasures/ui/tests/test_user_crud.py +++ b/flexmeasures/ui/tests/test_user_crud.py @@ -1,10 +1,8 @@ import pytest from flask import url_for -from flask_login import current_user from flexmeasures.data.services.users import find_user_by_email from flexmeasures.ui.tests.utils import mock_user_response -from flexmeasures.ui.crud.users import get_users_by_account """ @@ -13,15 +11,6 @@ """ -def test_get_users_by_account(client, requests_mock, as_prosumer_user1): - requests_mock.get( - f"http://localhost//api/v3_0/users?account_id={current_user.account.id}", - status_code=200, - json=mock_user_response(multiple=True), - ) - assert get_users_by_account(current_user.account.id)[0].username == "Alex" - - def test_user_list(client, as_admin, requests_mock): requests_mock.get( "http://localhost//api/v3_0/users", @@ -31,8 +20,6 @@ def test_user_list(client, as_admin, requests_mock): user_index = client.get(url_for("UserCrudUI:index"), follow_redirects=True) assert user_index.status_code == 200 assert b"All active users" in user_index.data - assert b"alex@seita.nl" in user_index.data - assert b"bert@seita.nl" in user_index.data @pytest.mark.parametrize("view", ["get", "toggle_active"]) diff --git a/flexmeasures/ui/tests/test_views.py b/flexmeasures/ui/tests/test_views.py index c2d13536b..78d218915 100644 --- a/flexmeasures/ui/tests/test_views.py +++ b/flexmeasures/ui/tests/test_views.py @@ -21,9 +21,14 @@ def test_dashboard_responds_only_for_logged_in_users(client, as_prosumer_user1): def test_assets_responds(client, requests_mock, as_prosumer_user1): requests_mock.get( - "http://localhost//api/v3_0/assets?account_id=1", + "http://localhost//api/v3_0/assets", status_code=200, - json={}, + json=[], + ) + requests_mock.get( + "http://localhost//api/v3_0/assets/public", + status_code=200, + json=[], ) assets_page = client.get(url_for("AssetCrudUI:index"), follow_redirects=True) assert assets_page.status_code == 200 diff --git a/flexmeasures/ui/tests/utils.py b/flexmeasures/ui/tests/utils.py index c5658198d..6e22173f2 100644 --- a/flexmeasures/ui/tests/utils.py +++ b/flexmeasures/ui/tests/utils.py @@ -20,15 +20,19 @@ def logout(client): def mock_asset_response( - asset_id: int = 1, + asset_id: int = 2, account_id: int = 1, as_list: bool = True, multiple: bool = False, ) -> dict | list[dict]: + """ + Mock response from asset API. + Does not mock output of paginated assets endpoint! + """ asset = dict( id=asset_id, name="TestAsset", - generic_asset_type_id=1, + generic_asset_type={"id": 1, "name": "battery"}, account_id=int(account_id), latitude=70.4, longitude=30.9, @@ -38,6 +42,7 @@ def mock_asset_response( if multiple: asset2 = copy.deepcopy(asset) asset2["name"] = "TestAsset2" + asset2["id"] += 1 asset_list.append(asset2) return asset_list return asset diff --git a/flexmeasures/ui/utils/color_defaults.py b/flexmeasures/ui/utils/color_defaults.py new file mode 100644 index 000000000..26fb6fc0c --- /dev/null +++ b/flexmeasures/ui/utils/color_defaults.py @@ -0,0 +1,71 @@ +from __future__ import annotations +from flexmeasures.data.models.user import Account + + +def get_color_settings(account: Account | None) -> dict: + """ + This function returns the primary and secondary color settings for the UI. + + It also provides variations of the primary and secondary colors, such as border color, hover color, and transparent color. + """ + + primary_color: str = "#1a3443" + secondary_color: str = "#f1a122" + if account: + primary_color = str( + account.primary_color + or ( + account.consultancy_account + and account.consultancy_account.primary_color + ) + or primary_color + ) + secondary_color = str( + account.secondary_color + or ( + account.consultancy_account + and account.consultancy_account.secondary_color + ) + or secondary_color + ) + + # Compute variations + primary_border_color = darken_color(primary_color, 7) + primary_hover_color = darken_color(primary_color, 4) + primary_transparent = rgba_color(primary_color, 0.2) + secondary_hover_color = lighten_color(secondary_color, 4) + secondary_transparent = rgba_color(secondary_color, 0.2) + + return { + "primary_color": primary_color, + "primary_border_color": primary_border_color, + "primary_hover_color": primary_hover_color, + "primary_transparent": primary_transparent, + "secondary_color": secondary_color, + "secondary_hover_color": secondary_hover_color, + "secondary_transparent": secondary_transparent, + } + + +def darken_color(hex_color: str, percentage: int) -> str: + hex_color = hex_color.lstrip("#") + r, g, b = int(hex_color[0:2], 16), int(hex_color[2:4], 16), int(hex_color[4:6], 16) + r = int(r * (1 - percentage / 100)) + g = int(g * (1 - percentage / 100)) + b = int(b * (1 - percentage / 100)) + return f"#{r:02x}{g:02x}{b:02x}" + + +def lighten_color(hex_color: str, percentage: int) -> str: + hex_color = hex_color.lstrip("#") + r, g, b = int(hex_color[0:2], 16), int(hex_color[2:4], 16), int(hex_color[4:6], 16) + r = int(r + (255 - r) * percentage / 100) + g = int(g + (255 - g) * percentage / 100) + b = int(b + (255 - b) * percentage / 100) + return f"#{r:02x}{g:02x}{b:02x}" + + +def rgba_color(hex_color: str, alpha: float) -> str: + hex_color = hex_color.lstrip("#") + r, g, b = int(hex_color[0:2], 16), int(hex_color[2:4], 16), int(hex_color[4:6], 16) + return f"rgba({r}, {g}, {b}, {alpha})" diff --git a/flexmeasures/ui/utils/view_utils.py b/flexmeasures/ui/utils/view_utils.py index 33baf7fe5..61a025ea6 100644 --- a/flexmeasures/ui/utils/view_utils.py +++ b/flexmeasures/ui/utils/view_utils.py @@ -1,4 +1,5 @@ """Utilities for views""" + from __future__ import annotations import json @@ -16,6 +17,7 @@ from flexmeasures.ui import flexmeasures_ui from flexmeasures.data.models.user import User, Account from flexmeasures.ui.utils.chart_defaults import chart_options +from flexmeasures.ui.utils.color_defaults import get_color_settings def render_flexmeasures_template(html_filename: str, **variables): @@ -31,11 +33,12 @@ def render_flexmeasures_template(html_filename: str, **variables): # use event_starts_after and event_ends_before from session if not given variables["event_starts_after"] = variables.get( - "event_starts_after", session.get("event_starts_after") - ) - variables["event_ends_before"] = variables.get( - "event_ends_before", session.get("event_ends_before") + "event_starts_after" + ) or session.get("event_starts_after") + variables["event_ends_before"] = variables.get("event_ends_before") or session.get( + "event_ends_before" ) + variables["chart_type"] = session.get("chart_type", "bar_chart") variables["page"] = html_filename.split("/")[-1].replace(".html", "") @@ -67,9 +70,9 @@ def render_flexmeasures_template(html_filename: str, **variables): variables["user_has_admin_reader_rights"] = user_has_admin_access( current_user, "read" ) - variables[ - "user_is_anonymous" - ] = current_user.is_authenticated and current_user.has_role("anonymous") + variables["user_is_anonymous"] = ( + current_user.is_authenticated and current_user.has_role("anonymous") + ) variables["user_email"] = current_user.is_authenticated and current_user.email or "" variables["user_name"] = ( current_user.is_authenticated and current_user.username or "" @@ -85,11 +88,25 @@ def render_flexmeasures_template(html_filename: str, **variables): options["downloadFileName"] = f"asset-{asset.id}-{asset.name}" variables["chart_options"] = json.dumps(options) - variables["menu_logo"] = current_app.config.get("FLEXMEASURES_MENU_LOGO_PATH") + account: Account | None = ( + current_user.account if current_user.is_authenticated else None + ) + + # check if user/consultant has logo_url set + if account: + variables["menu_logo"] = ( + account.logo_url + or (account.consultancy_account and account.consultancy_account.logo_url) + or current_app.config.get("FLEXMEASURES_MENU_LOGO_PATH") + ) + else: + variables["menu_logo"] = current_app.config.get("FLEXMEASURES_MENU_LOGO_PATH") + variables["extra_css"] = current_app.config.get("FLEXMEASURES_EXTRA_CSS_PATH") if "asset" in variables: variables["breadcrumb_info"] = get_breadcrumb_info(asset) + variables.update(get_color_settings(account)) # add color settings to variables return render_template(html_filename, **variables) @@ -161,6 +178,23 @@ def _minimal_ext_cmd(cmd: list): return version, commits_since, sha +ICON_MAPPING = { + # site structure + "evse": "icon-charging_station", + "charge point": "icon-charging_station", + "project": "icon-calculator", + "tariff": "icon-time", + "renewables": "icon-wind", + "site": "icon-empty-marker", + "scenario": "icon-binoculars", + # weather + "irradiance": "wi wi-horizon-alt", + "temperature": "wi wi-thermometer", + "wind direction": "wi wi-wind-direction", + "wind speed": "wi wi-strong-wind", +} + + def asset_icon_name(asset_type_name: str) -> str: """Icon name for this asset type. @@ -172,27 +206,9 @@ def asset_icon_name(asset_type_name: str) -> str: becomes (for a battery): """ - icon_mapping = { - # site structure - "evse": "icon-charging_station", - "charge point": "icon-charging_station", - "project": "icon-calculator", - "tariff": "icon-time", - "renewables": "icon-wind", - "site": "icon-empty-marker", - "scenario": "icon-binoculars", - # weather - "irradiance": "wi wi-horizon-alt", - "temperature": "wi wi-thermometer", - "wind direction": "wi wi-wind-direction", - "wind speed": "wi wi-strong-wind", - } - - for asset_group_name, icon_name in icon_mapping.items(): - if asset_group_name in asset_type_name.lower(): - return icon_name - - return f"icon-{asset_type_name}" + if asset_type_name: + asset_type_name = asset_type_name.lower() + return ICON_MAPPING.get(asset_type_name, f"icon-{asset_type_name}") def username(user_id) -> str: diff --git a/flexmeasures/ui/views/logged_in_user.py b/flexmeasures/ui/views/logged_in_user.py index 09616edc7..9897b906b 100644 --- a/flexmeasures/ui/views/logged_in_user.py +++ b/flexmeasures/ui/views/logged_in_user.py @@ -1,6 +1,13 @@ +from sqlalchemy import select +from werkzeug.exceptions import Forbidden, Unauthorized from flask_security.core import current_user from flask_security import login_required +from flexmeasures.auth.policy import check_access + +from flexmeasures.data import db +from flexmeasures.data.models.audit_log import AuditLog +from flexmeasures.data.models.user import Account from flexmeasures.ui.views import flexmeasures_ui from flexmeasures.data.services.accounts import ( get_number_of_assets_in_account, @@ -18,6 +25,21 @@ def logged_in_user_view(): """ account_roles = get_account_roles(current_user.account_id) account_role_names = [account_role.name for account_role in account_roles] + account = db.session.execute( + select(Account).filter_by(id=current_user.account_id) + ).scalar() + + user_can_view_account_auditlog = True + try: + check_access(AuditLog.account_table_acl(account), "read") + except (Forbidden, Unauthorized): + user_can_view_account_auditlog = False + + user_view_user_auditlog = True + try: + check_access(AuditLog.user_table_acl(current_user), "read") + except (Forbidden, Unauthorized): + user_view_user_auditlog = False return render_flexmeasures_template( "admin/logged_in_user.html", @@ -25,4 +47,6 @@ def logged_in_user_view(): roles=",".join([role.name for role in current_user.roles]), num_assets=get_number_of_assets_in_account(current_user.account_id), account_role_names=account_role_names, + can_view_account_auditlog=user_can_view_account_auditlog, + can_view_user_auditlog=user_view_user_auditlog, ) diff --git a/flexmeasures/ui/views/sensors.py b/flexmeasures/ui/views/sensors.py index 5ac96fdc6..ffaef611c 100644 --- a/flexmeasures/ui/views/sensors.py +++ b/flexmeasures/ui/views/sensors.py @@ -76,11 +76,12 @@ def get(self, id: int, **kwargs): - start_time: minimum time of the events to be shown - end_time: maximum time of the events to be shown """ + sensor = db.session.get(Sensor, id) return render_flexmeasures_template( "views/sensors.html", - sensor_id=id, + sensor=sensor, msg="", - breadcrumb_info=get_breadcrumb_info(db.session.get(Sensor, id)), + breadcrumb_info=get_breadcrumb_info(sensor), event_starts_after=request.args.get("start_time"), event_ends_before=request.args.get("end_time"), ) diff --git a/flexmeasures/utils/calculations.py b/flexmeasures/utils/calculations.py index 993fc6cc6..c5760c635 100644 --- a/flexmeasures/utils/calculations.py +++ b/flexmeasures/utils/calculations.py @@ -1,4 +1,5 @@ """ Various calculations """ + from __future__ import annotations from datetime import timedelta diff --git a/flexmeasures/utils/coding_utils.py b/flexmeasures/utils/coding_utils.py index 8dd01d5e8..d5f26a4a3 100644 --- a/flexmeasures/utils/coding_utils.py +++ b/flexmeasures/utils/coding_utils.py @@ -1,4 +1,5 @@ """ Various coding utils (e.g. around function decoration) """ + from __future__ import annotations import functools @@ -69,19 +70,29 @@ def sort_dict(unsorted_dict: dict) -> dict: return sorted_dict +# This function is used for sensors_to_show in follow-up PR it will be moved and renamed to flatten_sensors_to_show def flatten_unique(nested_list_of_objects: list) -> list: - """Returns unique objects in a possibly nested (one level) list of objects. + """ + Get unique sensor IDs from a list of `sensors_to_show`. + + Handles: + - Lists of sensor IDs + - Dictionaries with a `sensors` key + - Nested lists (one level) - Preserves the original order in which unique objects first occurred. + Example: + Input: + [1, [2, 20, 6], 10, [6, 2], {"title":None,"sensors": [10, 15]}, 15] - For example: - >>> flatten_unique([1, [2, 20, 6], 10, [6, 2]]) - <<< [1, 2, 20, 6, 10] + Output: + [1, 2, 20, 6, 10, 15] """ all_objects = [] for s in nested_list_of_objects: if isinstance(s, list): all_objects.extend(s) + elif isinstance(s, dict): + all_objects.extend(s["sensors"]) else: all_objects.append(s) return list(dict.fromkeys(all_objects).keys()) diff --git a/flexmeasures/utils/config_defaults.py b/flexmeasures/utils/config_defaults.py index c7f4248f4..b28556b9b 100644 --- a/flexmeasures/utils/config_defaults.py +++ b/flexmeasures/utils/config_defaults.py @@ -110,12 +110,16 @@ class Config(object): FLEXMEASURES_LP_SOLVER: str = "appsi_highs" FLEXMEASURES_JOB_TTL: timedelta = timedelta(days=1) FLEXMEASURES_PLANNING_HORIZON: timedelta = timedelta(days=2) - FLEXMEASURES_MAX_PLANNING_HORIZON: timedelta | int | None = 2520 # smallest number divisible by 1-10, which yields pleasant-looking durations for common sensor resolutions + FLEXMEASURES_MAX_PLANNING_HORIZON: timedelta | int | None = ( + 2520 # smallest number divisible by 1-10, which yields pleasant-looking durations for common sensor resolutions + ) FLEXMEASURES_PLANNING_TTL: timedelta = timedelta( days=7 ) # Time to live for UDI event ids of successful scheduling jobs. Set a negative timedelta to persist forever. FLEXMEASURES_DEFAULT_DATASOURCE: str = "FlexMeasures" - FLEXMEASURES_JOB_CACHE_TTL: int = 3600 # Time to live for the job caching keys in seconds. Set a negative timedelta to persist forever. + FLEXMEASURES_JOB_CACHE_TTL: int = ( + 3600 # Time to live for the job caching keys in seconds. Set a negative timedelta to persist forever. + ) FLEXMEASURES_TASK_CHECK_AUTH_TOKEN: str | None = None FLEXMEASURES_REDIS_URL: str = "localhost" FLEXMEASURES_REDIS_PORT: int = 6379 @@ -129,13 +133,18 @@ class Config(object): # todo: expand with other js versions used in FlexMeasures ) FLEXMEASURES_JSON_COMPACT = False + JSON_SORT_KEYS = False FLEXMEASURES_FALLBACK_REDIRECT: bool = False # Custom sunset switches - FLEXMEASURES_API_SUNSET_ACTIVE: bool = False # if True, sunset endpoints return 410 (Gone) responses; if False, they return 404 (Not Found) responses or will work as before, depending on whether the current FlexMeasures version still contains the endpoint logic + FLEXMEASURES_API_SUNSET_ACTIVE: bool = ( + False # if True, sunset endpoints return 410 (Gone) responses; if False, they return 404 (Not Found) responses or will work as before, depending on whether the current FlexMeasures version still contains the endpoint logic + ) FLEXMEASURES_API_SUNSET_DATE: str | None = None # e.g. 2023-05-01 - FLEXMEASURES_API_SUNSET_LINK: str | None = None # e.g. https://flexmeasures.readthedocs.io/en/latest/api/introduction.html#deprecation-and-sunset + FLEXMEASURES_API_SUNSET_LINK: str | None = ( + None # e.g. https://flexmeasures.readthedocs.io/en/latest/api/introduction.html#deprecation-and-sunset + ) # if True, all requests are forced to be via HTTPS. FLEXMEASURES_FORCE_HTTPS: bool = False @@ -183,7 +192,6 @@ class DevelopmentConfig(Config): # PRESERVE_CONTEXT_ON_EXCEPTION: bool = False # might need this to make our transaction handling work in debug mode FLEXMEASURES_MODE: str = "development" FLEXMEASURES_PROFILE_REQUESTS: bool = True - FLEXMEASURES_JSON_COMPACT = False class TestingConfig(Config): diff --git a/flexmeasures/utils/error_utils.py b/flexmeasures/utils/error_utils.py index 131d970af..2bdcad889 100644 --- a/flexmeasures/utils/error_utils.py +++ b/flexmeasures/utils/error_utils.py @@ -18,7 +18,8 @@ def log_error(exc: Exception, error_msg: str): """Collect meta data about the exception and log it. - error_msg comes in as an extra attribute because Exception implementations differ here.""" + error_msg comes in as an extra attribute because Exception implementations differ here. + """ exc_info = sys.exc_info() last_traceback = exc_info[2] diff --git a/flexmeasures/utils/tests/test_time_utils.py b/flexmeasures/utils/tests/test_time_utils.py index 15c9056f3..b66ff540d 100644 --- a/flexmeasures/utils/tests/test_time_utils.py +++ b/flexmeasures/utils/tests/test_time_utils.py @@ -13,6 +13,7 @@ naturalized_datetime_str, get_most_recent_clocktime_window, apply_offset_chain, + to_utc_timestamp, ) @@ -206,3 +207,21 @@ def test_apply_offset_chain( output_date: pd.Timestamp | datetime, ): assert apply_offset_chain(input_date, offset_chain) == output_date + + +@pytest.mark.parametrize( + "input_value, expected_output", + [ + (datetime(2024, 4, 28, 8, 55, 58), 1714294558.0), + (pd.Timestamp("2024-04-28 08:55:58", tz="UTC").to_pydatetime(), 1714294558.0), + ("Sun, 28 Apr 2024 08:55:58 GMT", 1714294558.0), + ("Invalid date string", None), + (None, None), + ], +) +def test_to_utc_timestamp(input_value, expected_output): + result = to_utc_timestamp(input_value) + if expected_output is None: + assert result is None + else: + assert result == pytest.approx(expected_output) diff --git a/flexmeasures/utils/time_utils.py b/flexmeasures/utils/time_utils.py index ad350cf12..8467ea769 100644 --- a/flexmeasures/utils/time_utils.py +++ b/flexmeasures/utils/time_utils.py @@ -411,3 +411,50 @@ def apply_offset_chain( if isinstance(dt, datetime): return _dt.to_pydatetime() return _dt + + +def to_utc_timestamp(value): + """ + Convert a datetime object or string to a UTC timestamp (seconds since epoch). + + The dt parameter can be either naive or tz-aware. We assume UTC in the naive case. + If dt parameter is a string, it is expected to look like 'Sun, 28 Apr 2024 08:55:58 GMT'. + String format is supported for cases when we process JSON from internal API responses, + e.g., ui/crud/users auditlog router. + + + Returns: + - Float: seconds since Unix epoch (1970-01-01 00:00:00 UTC) + - None: if input is None + + + Hint: This can be set as a jinja filter to display UTC timestamps in the app, e.g.: + app.jinja_env.filters['to_utc_timestamp'] = to_utc_timestamp + + Example usage: + >>> to_utc_timestamp(datetime(2024, 4, 28, 8, 55, 58)) + 1714294558.0 + >>> to_utc_timestamp("Sun, 28 Apr 2024 08:55:58 GMT") + 1714294558.0 + >>> to_utc_timestamp(None) + None + """ + if value is None: + return None + if isinstance(value, str): + # Parse string datetime in the format 'Tue, 13 Dec 2022 14:06:23 GMT' + try: + value = datetime.strptime(value, "%a, %d %b %Y %H:%M:%S %Z") + except ValueError: + # Return None or raise an error if the string is in an unexpected format + return None + if isinstance(value, datetime): + if value.tzinfo is None: + # If naive, assume UTC + value = pd.Timestamp(value).tz_localize("utc") + else: + # Convert to UTC if already timezone-aware + value = pd.Timestamp(value).tz_convert("utc") + + # Return Unix timestamp (seconds since epoch) + return value.timestamp() diff --git a/flexmeasures/utils/unit_utils.py b/flexmeasures/utils/unit_utils.py index 806ae38f0..b70bf5e75 100644 --- a/flexmeasures/utils/unit_utils.py +++ b/flexmeasures/utils/unit_utils.py @@ -8,6 +8,7 @@ Time series with fixed resolution can be converted from units of flow to units of stock (such as 'kW' to 'kWh'), and vice versa. Percentages can be converted to units of some physical capacity if a capacity is known (such as '%' to 'kWh'). """ + from __future__ import annotations from datetime import timedelta @@ -273,9 +274,7 @@ def convert_units( from_magnitudes = ( data.to_numpy() if isinstance(data, pd.Series) - else np.asarray(data) - if isinstance(data, list) - else np.array([data]) + else np.asarray(data) if isinstance(data, list) else np.array([data]) ) try: from_quantities = ur.Quantity(from_magnitudes, from_unit) diff --git a/flexmeasures/utils/validation_utils.py b/flexmeasures/utils/validation_utils.py new file mode 100644 index 000000000..50935ed54 --- /dev/null +++ b/flexmeasures/utils/validation_utils.py @@ -0,0 +1,53 @@ +import re + + +def validate_color_hex(value): + """ + Validates that a given value is a valid hex color code. + + Parameters: + :value: The color code to validate. + """ + if value is None: + return value + + if value and not value.startswith("#"): + value = f"#{value}" + + hex_pattern = re.compile(r"^#?([A-Fa-f0-9]{6}|[A-Fa-f0-9]{3})$") + + if re.match(hex_pattern, value): + return value + else: + raise ValueError(f"{value} is not a valid hex color code.") + + +def validate_url(value): + """ + Validates that a given value is a valid URL format using regex. + + Parameters: + :value: The URL to validate. + """ + if value is None: + return value + + url_regex = re.compile( + r"^(https?|ftp)://" # Protocol: http, https, or ftp + r"((([A-Za-z0-9-]+\.)+[A-Za-z]{2,6})|" # Domain name + r"(\d{1,3}\.){3}\d{1,3})" # OR IPv4 + r"(:\d+)?" # Port + r"(/([A-Za-z0-9$_.+!*\'(),;?&=-]|%[0-9A-Fa-f]{2})*)*" # Path + r"(\?([A-Za-z0-9$_.+!*\'(),;?&=-]|%[0-9A-Fa-f]{2})*)?" # Query string + ) + + if not url_regex.match(value): + raise ValueError(f"'{value}' is not a valid URL.") + + # check if more than 255 characters + if len(value) > 255: + raise ValueError( + "provided logo-url is too long. Maximum length is 255 characters." + ) + + return value diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 000000000..485504b4f --- /dev/null +++ b/requirements.txt @@ -0,0 +1,371 @@ +# NOTE: This file is copied from 3.11/app.txt and only here so LFX's Snyk can find it +# +# This file is autogenerated by pip-compile with Python 3.11 +# by the following command: +# +# pip-compile --output-file=requirements/3.11/app.txt requirements/app.in +# +alembic==1.13.1 + # via flask-migrate +altair==5.3.0 + # via -r requirements/app.in +arrow==1.3.0 + # via rq-dashboard +attrs==23.2.0 + # via + # jsonschema + # referencing +babel==2.14.0 + # via py-moneyed +bcrypt==4.0.1 + # via -r requirements/app.in +blinker==1.7.0 + # via + # flask-mail + # flask-principal + # flask-security-too + # sentry-sdk +certifi==2024.2.2 + # via + # requests + # sentry-sdk +charset-normalizer==3.3.2 + # via requests +click==8.1.7 + # via + # -r requirements/app.in + # click-default-group + # flask + # rq +click-default-group==1.2.4 + # via -r requirements/app.in +colour==0.1.5 + # via -r requirements/app.in +contourpy==1.2.1 + # via matplotlib +convertdate==2.4.0 + # via workalendar +cycler==0.12.1 + # via matplotlib +dill==0.3.8 + # via openturns +dnspython==2.6.1 + # via email-validator +email-validator==2.1.1 + # via + # -r requirements/app.in + # flask-security-too +filelock==3.13.3 + # via tldextract +flask==2.2.5 + # via + # -r requirements/app.in + # flask-classful + # flask-cors + # flask-json + # flask-login + # flask-mail + # flask-marshmallow + # flask-migrate + # flask-principal + # flask-security-too + # flask-sqlalchemy + # flask-sslify + # flask-wtf + # rq-dashboard + # sentry-sdk +flask-classful==0.16.0 + # via -r requirements/app.in +flask-cors==4.0.0 + # via -r requirements/app.in +flask-json==0.4.0 + # via -r requirements/app.in +flask-login==0.6.3 + # via + # -r requirements/app.in + # flask-security-too +flask-mail==0.9.1 + # via -r requirements/app.in +flask-marshmallow==1.2.1 + # via -r requirements/app.in +flask-migrate==4.0.7 + # via -r requirements/app.in +flask-principal==0.4.0 + # via flask-security-too +flask-security-too==5.1.2 + # via -r requirements/app.in +flask-sqlalchemy==3.1.1 + # via + # -r requirements/app.in + # flask-migrate +flask-sslify==0.1.5 + # via -r requirements/app.in +flask-wtf==1.2.1 + # via + # -r requirements/app.in + # flask-security-too +fonttools==4.51.0 + # via matplotlib +greenlet==3.0.3 + # via sqlalchemy +humanize==4.9.0 + # via -r requirements/app.in +idna==3.6 + # via + # email-validator + # requests + # tldextract +importlib-metadata==7.1.0 + # via + # -r requirements/app.in + # timely-beliefs +inflect==6.0.2 + # via -r requirements/app.in +inflection==0.5.1 + # via -r requirements/app.in +iso8601==2.1.0 + # via -r requirements/app.in +isodate==0.6.1 + # via + # -r requirements/app.in + # timely-beliefs +itsdangerous==2.1.2 + # via + # flask + # flask-security-too + # flask-wtf +jinja2==3.1.3 + # via + # altair + # flask +joblib==1.3.2 + # via scikit-learn +jsonschema==4.21.1 + # via altair +jsonschema-specifications==2023.12.1 + # via jsonschema +kiwisolver==1.4.5 + # via matplotlib +lunardate==0.2.2 + # via workalendar +mako==1.3.2 + # via alembic +markupsafe==2.1.5 + # via + # jinja2 + # mako + # sentry-sdk + # werkzeug + # wtforms +marshmallow==3.21.1 + # via + # -r requirements/app.in + # flask-marshmallow + # marshmallow-polyfield + # marshmallow-sqlalchemy + # webargs +marshmallow-polyfield==5.11 + # via -r requirements/app.in +marshmallow-sqlalchemy==1.0.0 + # via -r requirements/app.in +matplotlib==3.8.4 + # via timetomodel +numpy==1.24.4 + # via + # -r requirements/app.in + # altair + # contourpy + # matplotlib + # pandas + # patsy + # properscoring + # scikit-learn + # scipy + # sktime + # statsmodels + # timely-beliefs + # timetomodel + # uniplot +openturns==1.22 + # via timely-beliefs +packaging==24.0 + # via + # altair + # marshmallow + # matplotlib + # sktime + # statsmodels + # webargs +pandas==2.2.1 + # via + # -r requirements/app.in + # altair + # sktime + # statsmodels + # timely-beliefs + # timetomodel +passlib==1.7.4 + # via flask-security-too +patsy==0.5.6 + # via statsmodels +pillow==10.3.0 + # via + # -r requirements/app.in + # matplotlib +pint==0.23 + # via -r requirements/app.in +ply==3.11 + # via pyomo +properscoring==0.1 + # via timely-beliefs +pscript==0.7.7 + # via -r requirements/app.in +psutil==5.9.8 + # via openturns +psycopg2-binary==2.9.9 + # via + # -r requirements/app.in + # timely-beliefs +py-moneyed==3.0 + # via -r requirements/app.in +pydantic==1.10.15 + # via + # -r requirements/app.in + # inflect +pyluach==2.2.0 + # via workalendar +pymeeus==0.5.12 + # via convertdate +pyomo==6.7.1 + # via -r requirements/app.in +pyparsing==3.1.2 + # via matplotlib +python-dateutil==2.9.0.post0 + # via + # arrow + # matplotlib + # pandas + # timetomodel + # workalendar +python-dotenv==1.0.1 + # via -r requirements/app.in +pytz==2024.1 + # via + # -r requirements/app.in + # pandas + # timely-beliefs + # timetomodel +pyyaml==6.0.1 + # via -r requirements/app.in +redis==4.6.0 + # via + # -r requirements/app.in + # redis-sentinel-url + # rq + # rq-dashboard +redis-sentinel-url==1.0.1 + # via rq-dashboard +referencing==0.34.0 + # via + # jsonschema + # jsonschema-specifications +requests==2.31.0 + # via + # requests-file + # tldextract +requests-file==2.0.0 + # via tldextract +rpds-py==0.18.0 + # via + # jsonschema + # referencing +rq==1.16.1 + # via + # -r requirements/app.in + # rq-dashboard +rq-dashboard==0.8.0.2 + # via -r requirements/app.in +scikit-base==0.7.5 + # via sktime +scikit-learn==1.4.1.post1 + # via + # sktime + # timetomodel +scipy==1.13.0 + # via + # properscoring + # scikit-learn + # sktime + # statsmodels + # timely-beliefs + # timetomodel +sentry-sdk[flask]==1.44.1 + # via -r requirements/app.in +six==1.16.0 + # via + # isodate + # patsy + # python-dateutil +sktime==0.28.0 + # via timely-beliefs +sqlalchemy==2.0.29 + # via + # -r requirements/app.in + # alembic + # flask-sqlalchemy + # marshmallow-sqlalchemy + # timely-beliefs + # timetomodel +statsmodels==0.14.1 + # via timetomodel +tabulate==0.9.0 + # via -r requirements/app.in +threadpoolctl==3.4.0 + # via scikit-learn +timely-beliefs[forecast]==3.0.2 + # via -r requirements/app.in +timetomodel==0.7.3 + # via -r requirements/app.in +tldextract==5.1.2 + # via -r requirements/app.in +toolz==0.12.1 + # via altair +types-python-dateutil==2.9.0.20240316 + # via arrow +typing-extensions==4.11.0 + # via + # alembic + # pint + # py-moneyed + # pydantic + # sqlalchemy +tzdata==2024.1 + # via pandas +uniplot==0.14.1 + # via -r requirements/app.in +urllib3==2.2.1 + # via + # requests + # sentry-sdk +vl-convert-python==1.3.0 + # via -r requirements/app.in +webargs==8.4.0 + # via -r requirements/app.in +werkzeug==2.2.3 + # via + # -r requirements/app.in + # flask + # flask-login +workalendar==17.0.0 + # via -r requirements/app.in +wtforms==3.1.2 + # via + # flask-security-too + # flask-wtf +xlrd==2.0.1 + # via -r requirements/app.in +zipp==3.18.1 + # via importlib-metadata + +# The following packages are considered to be unsafe in a requirements file: +# setuptools diff --git a/requirements/3.10/app.txt b/requirements/3.10/app.txt index b54e1d81c..379326ab9 100644 --- a/requirements/3.10/app.txt +++ b/requirements/3.10/app.txt @@ -4,32 +4,40 @@ # # pip-compile --output-file=requirements/3.10/app.txt requirements/app.in # -alembic==1.13.1 +alembic==1.13.2 # via flask-migrate -altair==5.3.0 +altair==5.4.1 # via -r requirements/app.in +appdirs==1.4.4 + # via pint +argon2-cffi==23.1.0 + # via -r requirements/app.in +argon2-cffi-bindings==21.2.0 + # via argon2-cffi arrow==1.3.0 # via rq-dashboard async-timeout==4.0.3 # via redis -attrs==23.2.0 +attrs==24.2.0 # via # jsonschema # referencing -babel==2.14.0 +babel==2.16.0 # via py-moneyed bcrypt==4.0.1 # via -r requirements/app.in -blinker==1.7.0 +blinker==1.8.2 # via + # flask # flask-mail # flask-principal - # flask-security-too # sentry-sdk -certifi==2024.2.2 +certifi==2024.8.30 # via # requests # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings charset-normalizer==3.3.2 # via requests click==8.1.7 @@ -40,9 +48,7 @@ click==8.1.7 # rq click-default-group==1.2.4 # via -r requirements/app.in -colour==0.1.5 - # via -r requirements/app.in -contourpy==1.2.1 +contourpy==1.3.0 # via matplotlib convertdate==2.4.0 # via workalendar @@ -52,13 +58,13 @@ dill==0.3.8 # via openturns dnspython==2.6.1 # via email-validator -email-validator==2.1.1 +email-validator==2.2.0 # via # -r requirements/app.in # flask-security-too -filelock==3.13.3 +filelock==3.16.1 # via tldextract -flask==2.2.5 +flask==3.0.3 # via # -r requirements/app.in # flask-classful @@ -77,7 +83,7 @@ flask==2.2.5 # sentry-sdk flask-classful==0.16.0 # via -r requirements/app.in -flask-cors==4.0.0 +flask-cors==5.0.0 # via -r requirements/app.in flask-json==0.4.0 # via -r requirements/app.in @@ -85,7 +91,7 @@ flask-login==0.6.3 # via # -r requirements/app.in # flask-security-too -flask-mail==0.9.1 +flask-mail==0.10.0 # via -r requirements/app.in flask-marshmallow==1.2.1 # via -r requirements/app.in @@ -93,7 +99,7 @@ flask-migrate==4.0.7 # via -r requirements/app.in flask-principal==0.4.0 # via flask-security-too -flask-security-too==5.1.2 +flask-security-too==5.5.2 # via -r requirements/app.in flask-sqlalchemy==3.1.1 # via @@ -105,21 +111,27 @@ flask-wtf==1.2.1 # via # -r requirements/app.in # flask-security-too -fonttools==4.51.0 +flexcache==0.3 + # via pint +flexparser==0.3.1 + # via pint +fonttools==4.53.1 # via matplotlib -greenlet==3.0.3 +greenlet==3.1.0 # via sqlalchemy -humanize==4.9.0 +humanize==4.10.0 # via -r requirements/app.in -idna==3.6 +idna==3.10 # via # email-validator # requests # tldextract -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -r requirements/app.in # timely-beliefs +importlib-resources==6.4.5 + # via flask-security-too inflect==6.0.2 # via -r requirements/app.in inflection==0.5.1 @@ -130,35 +142,37 @@ isodate==0.6.1 # via # -r requirements/app.in # timely-beliefs -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # flask - # flask-security-too # flask-wtf -jinja2==3.1.3 +jinja2==3.1.4 # via # altair # flask -joblib==1.3.2 - # via scikit-learn -jsonschema==4.21.1 +joblib==1.4.2 + # via + # scikit-learn + # sktime +jsonschema==4.23.0 # via altair jsonschema-specifications==2023.12.1 # via jsonschema -kiwisolver==1.4.5 +kiwisolver==1.4.7 # via matplotlib lunardate==0.2.2 # via workalendar -mako==1.3.2 +mako==1.3.5 # via alembic markupsafe==2.1.5 # via + # flask-security-too # jinja2 # mako # sentry-sdk # werkzeug # wtforms -marshmallow==3.21.1 +marshmallow==3.22.0 # via # -r requirements/app.in # flask-marshmallow @@ -167,14 +181,14 @@ marshmallow==3.21.1 # webargs marshmallow-polyfield==5.11 # via -r requirements/app.in -marshmallow-sqlalchemy==1.0.0 +marshmallow-sqlalchemy==1.1.0 # via -r requirements/app.in -matplotlib==3.8.4 +matplotlib==3.9.2 # via timetomodel -numpy==1.24.4 +narwhals==1.8.2 + # via altair +numpy==1.26.4 # via - # -r requirements/app.in - # altair # contourpy # matplotlib # pandas @@ -187,9 +201,9 @@ numpy==1.24.4 # timely-beliefs # timetomodel # uniplot -openturns==1.22 +openturns==1.23 # via timely-beliefs -packaging==24.0 +packaging==24.1 # via # altair # marshmallow @@ -200,7 +214,6 @@ packaging==24.0 pandas==2.2.1 # via # -r requirements/app.in - # altair # sktime # statsmodels # timely-beliefs @@ -209,19 +222,17 @@ passlib==1.7.4 # via flask-security-too patsy==0.5.6 # via statsmodels -pillow==10.3.0 +pillow==10.4.0 # via # -r requirements/app.in # matplotlib -pint==0.23 +pint==0.24.3 # via -r requirements/app.in ply==3.11 # via pyomo properscoring==0.1 # via timely-beliefs -pscript==0.7.7 - # via -r requirements/app.in -psutil==5.9.8 +psutil==6.0.0 # via openturns psycopg2-binary==2.9.9 # via @@ -229,7 +240,9 @@ psycopg2-binary==2.9.9 # timely-beliefs py-moneyed==3.0 # via -r requirements/app.in -pydantic==1.10.15 +pycparser==2.22 + # via cffi +pydantic==1.10.18 # via # -r requirements/app.in # inflect @@ -237,9 +250,9 @@ pyluach==2.2.0 # via workalendar pymeeus==0.5.12 # via convertdate -pyomo==6.7.1 +pyomo==6.8.0 # via -r requirements/app.in -pyparsing==3.1.2 +pyparsing==3.1.4 # via matplotlib python-dateutil==2.9.0.post0 # via @@ -250,15 +263,15 @@ python-dateutil==2.9.0.post0 # workalendar python-dotenv==1.0.1 # via -r requirements/app.in -pytz==2024.1 +pytz==2024.2 # via # -r requirements/app.in # pandas # timely-beliefs # timetomodel -pyyaml==6.0.1 +pyyaml==6.0.2 # via -r requirements/app.in -redis==4.6.0 +redis==5.0.8 # via # -r requirements/app.in # redis-sentinel-url @@ -266,33 +279,33 @@ redis==4.6.0 # rq-dashboard redis-sentinel-url==1.0.1 # via rq-dashboard -referencing==0.34.0 +referencing==0.35.1 # via # jsonschema # jsonschema-specifications -requests==2.31.0 +requests==2.32.3 # via # requests-file # tldextract -requests-file==2.0.0 +requests-file==2.1.0 # via tldextract -rpds-py==0.18.0 +rpds-py==0.20.0 # via # jsonschema # referencing -rq==1.16.1 +rq==1.16.2 # via # -r requirements/app.in # rq-dashboard -rq-dashboard==0.6.7.2 +rq-dashboard==0.8.0.2 # via -r requirements/app.in -scikit-base==0.7.5 +scikit-base==0.9.0 # via sktime -scikit-learn==1.4.1.post1 +scikit-learn==1.5.2 # via # sktime # timetomodel -scipy==1.13.0 +scipy==1.14.1 # via # properscoring # scikit-learn @@ -300,16 +313,16 @@ scipy==1.13.0 # statsmodels # timely-beliefs # timetomodel -sentry-sdk[flask]==1.44.1 +sentry-sdk[flask]==2.14.0 # via -r requirements/app.in six==1.16.0 # via # isodate # patsy # python-dateutil -sktime==0.28.0 +sktime==0.33.0 # via timely-beliefs -sqlalchemy==2.0.29 +sqlalchemy==2.0.35 # via # -r requirements/app.in # alembic @@ -317,43 +330,43 @@ sqlalchemy==2.0.29 # marshmallow-sqlalchemy # timely-beliefs # timetomodel -statsmodels==0.14.1 +statsmodels==0.14.3 # via timetomodel tabulate==0.9.0 # via -r requirements/app.in -threadpoolctl==3.4.0 +threadpoolctl==3.5.0 # via scikit-learn -timely-beliefs[forecast]==2.3.0 +timely-beliefs[forecast]==3.1.0 # via -r requirements/app.in timetomodel==0.7.3 # via -r requirements/app.in tldextract==5.1.2 # via -r requirements/app.in -toolz==0.12.1 - # via altair -types-python-dateutil==2.9.0.20240316 +types-python-dateutil==2.9.0.20240906 # via arrow -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # alembic # altair + # flexcache + # flexparser # pint # py-moneyed # pydantic # sqlalchemy tzdata==2024.1 # via pandas -uniplot==0.12.5 +uniplot==0.14.1 # via -r requirements/app.in -urllib3==2.2.1 +urllib3==2.2.3 # via # requests # sentry-sdk -vl-convert-python==1.3.0 +vl-convert-python==1.6.1 # via -r requirements/app.in -webargs==8.4.0 +webargs==8.6.0 # via -r requirements/app.in -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -r requirements/app.in # flask @@ -366,8 +379,5 @@ wtforms==3.1.2 # flask-wtf xlrd==2.0.1 # via -r requirements/app.in -zipp==3.18.1 +zipp==3.20.2 # via importlib-metadata - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/3.10/dev.txt b/requirements/3.10/dev.txt index de3162508..54c12824b 100644 --- a/requirements/3.10/dev.txt +++ b/requirements/3.10/dev.txt @@ -4,7 +4,7 @@ # # pip-compile --constraint=requirements/3.10/app.txt --constraint=requirements/3.10/test.txt --output-file=requirements/3.10/dev.txt requirements/dev.in # -black==22.3.0 +black==24.8.0 # via -r requirements/dev.in cfgv==3.4.0 # via pre-commit @@ -15,52 +15,53 @@ click==8.1.7 # black distlib==0.3.8 # via virtualenv -filelock==3.13.3 +filelock==3.16.1 # via # -c requirements/3.10/app.txt # virtualenv -flake8==4.0.1 +flake8==7.1.1 # via -r requirements/dev.in flake8-blind-except==0.2.1 # via -r requirements/dev.in -identify==2.5.35 +identify==2.6.1 # via pre-commit -mccabe==0.6.1 +mccabe==0.7.0 # via flake8 -mypy==1.9.0 +mypy==1.11.2 # via -r requirements/dev.in mypy-extensions==1.0.0 # via # black # mypy -nodeenv==1.8.0 +nodeenv==1.9.1 # via pre-commit -packaging==24.0 +packaging==24.1 # via # -c requirements/3.10/app.txt # -c requirements/3.10/test.txt + # black # setuptools-scm pathspec==0.12.1 # via black -platformdirs==4.2.0 +platformdirs==4.3.6 # via # black # virtualenv -pre-commit==3.7.0 +pre-commit==3.8.0 # via -r requirements/dev.in -pycodestyle==2.8.0 +pycodestyle==2.12.1 # via flake8 -pyflakes==2.4.0 +pyflakes==3.2.0 # via flake8 -pyinstrument==4.6.2 +pyinstrument==4.7.3 # via -r requirements/dev.in pytest-runner==6.0.1 # via -r requirements/dev.in -pyyaml==6.0.1 +pyyaml==6.0.2 # via # -c requirements/3.10/app.txt # pre-commit -setuptools-scm==8.0.4 +setuptools-scm==8.1.0 # via -r requirements/dev.in tomli==2.0.1 # via @@ -68,14 +69,15 @@ tomli==2.0.1 # black # mypy # setuptools-scm -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # -c requirements/3.10/app.txt + # -c requirements/3.10/test.txt + # black # mypy - # setuptools-scm -virtualenv==20.25.1 +virtualenv==20.26.5 # via pre-commit -watchdog==4.0.0 +watchdog==5.0.2 # via -r requirements/dev.in # The following packages are considered to be unsafe in a requirements file: diff --git a/requirements/3.10/docs.txt b/requirements/3.10/docs.txt index b072c380b..3ce1b3945 100644 --- a/requirements/3.10/docs.txt +++ b/requirements/3.10/docs.txt @@ -6,11 +6,11 @@ # alabaster==0.7.16 # via sphinx -babel==2.14.0 +babel==2.16.0 # via # -c requirements/3.10/app.txt # sphinx -certifi==2024.2.2 +certifi==2024.8.30 # via # -c requirements/3.10/app.txt # requests @@ -23,13 +23,13 @@ docutils==0.20.1 # sphinx # sphinx-rtd-theme # sphinx-tabs -idna==3.6 +idna==3.10 # via # -c requirements/3.10/app.txt # requests imagesize==1.4.1 # via sphinx -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.10/app.txt # sphinx @@ -37,15 +37,15 @@ markupsafe==2.1.5 # via # -c requirements/3.10/app.txt # jinja2 -packaging==24.0 +packaging==24.1 # via # -c requirements/3.10/app.txt # sphinx -pygments==2.17.2 +pygments==2.18.0 # via # sphinx # sphinx-tabs -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.10/app.txt # sphinx @@ -55,7 +55,7 @@ six==1.16.0 # sphinxcontrib-httpdomain snowballstemmer==2.2.0 # via sphinx -sphinx==7.2.6 +sphinx==7.3.7 # via # -r requirements/docs.in # sphinx-copybutton @@ -72,11 +72,11 @@ sphinx-rtd-theme==2.0.0 # via -r requirements/docs.in sphinx-tabs==3.4.5 # via -r requirements/docs.in -sphinxcontrib-applehelp==1.0.8 +sphinxcontrib-applehelp==2.0.0 # via sphinx -sphinxcontrib-devhelp==1.0.6 +sphinxcontrib-devhelp==2.0.0 # via sphinx -sphinxcontrib-htmlhelp==2.0.5 +sphinxcontrib-htmlhelp==2.1.0 # via sphinx sphinxcontrib-httpdomain==1.8.1 # via -r requirements/docs.in @@ -84,11 +84,11 @@ sphinxcontrib-jquery==4.1 # via sphinx-rtd-theme sphinxcontrib-jsmath==1.0.1 # via sphinx -sphinxcontrib-qthelp==1.0.7 +sphinxcontrib-qthelp==2.0.0 # via sphinx -sphinxcontrib-serializinghtml==1.1.10 +sphinxcontrib-serializinghtml==2.0.0 # via sphinx -urllib3==2.2.1 +urllib3==2.2.3 # via # -c requirements/3.10/app.txt # requests diff --git a/requirements/3.10/test.txt b/requirements/3.10/test.txt index 80f38cd06..563ab2923 100644 --- a/requirements/3.10/test.txt +++ b/requirements/3.10/test.txt @@ -8,7 +8,11 @@ async-timeout==4.0.3 # via # -c requirements/3.10/app.txt # redis -certifi==2024.2.2 +blinker==1.8.2 + # via + # -c requirements/3.10/app.txt + # flask +certifi==2024.8.30 # via # -c requirements/3.10/app.txt # requests @@ -20,47 +24,51 @@ click==8.1.7 # via # -c requirements/3.10/app.txt # flask -coverage[toml]==7.4.4 +coverage[toml]==7.6.1 # via pytest-cov -exceptiongroup==1.2.0 +exceptiongroup==1.2.2 # via pytest -fakeredis==2.21.3 +fakeredis==2.24.1 # via -r requirements/test.in -flask==2.2.5 +flask==3.0.3 # via # -c requirements/3.10/app.txt # pytest-flask -highspy==1.5.3 +highspy==1.7.2 # via -r requirements/test.in -idna==3.6 +idna==3.10 # via # -c requirements/3.10/app.txt # requests iniconfig==2.0.0 # via pytest -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # -c requirements/3.10/app.txt # flask -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.10/app.txt # flask -lupa==2.1 +lupa==2.2 # via -r requirements/test.in markupsafe==2.1.5 # via # -c requirements/3.10/app.txt # jinja2 # werkzeug -packaging==24.0 +numpy==1.26.4 + # via + # -c requirements/3.10/app.txt + # highspy +packaging==24.1 # via # -c requirements/3.10/app.txt # pytest # pytest-sugar -pluggy==1.4.0 +pluggy==1.5.0 # via pytest -pytest==8.1.1 +pytest==8.3.3 # via # -r requirements/test.in # pytest-cov @@ -75,11 +83,11 @@ pytest-mock==3.14.0 # via -r requirements/test.in pytest-sugar==1.0.0 # via -r requirements/test.in -redis==4.6.0 +redis==5.0.8 # via # -c requirements/3.10/app.txt # fakeredis -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.10/app.txt # -r requirements/test.in @@ -94,11 +102,15 @@ tomli==2.0.1 # via # coverage # pytest -urllib3==2.2.1 +typing-extensions==4.12.2 + # via + # -c requirements/3.10/app.txt + # fakeredis +urllib3==2.2.3 # via # -c requirements/3.10/app.txt # requests -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -c requirements/3.10/app.txt # flask diff --git a/requirements/3.11/app.txt b/requirements/3.11/app.txt index 866b3c729..6cbdf868a 100644 --- a/requirements/3.11/app.txt +++ b/requirements/3.11/app.txt @@ -4,30 +4,38 @@ # # pip-compile --output-file=requirements/3.11/app.txt requirements/app.in # -alembic==1.13.1 +alembic==1.13.2 # via flask-migrate -altair==5.3.0 +altair==5.4.1 # via -r requirements/app.in +appdirs==1.4.4 + # via pint +argon2-cffi==23.1.0 + # via -r requirements/app.in +argon2-cffi-bindings==21.2.0 + # via argon2-cffi arrow==1.3.0 # via rq-dashboard -attrs==23.2.0 +attrs==24.2.0 # via # jsonschema # referencing -babel==2.14.0 +babel==2.16.0 # via py-moneyed bcrypt==4.0.1 # via -r requirements/app.in -blinker==1.7.0 +blinker==1.8.2 # via + # flask # flask-mail # flask-principal - # flask-security-too # sentry-sdk -certifi==2024.2.2 +certifi==2024.8.30 # via # requests # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings charset-normalizer==3.3.2 # via requests click==8.1.7 @@ -38,9 +46,7 @@ click==8.1.7 # rq click-default-group==1.2.4 # via -r requirements/app.in -colour==0.1.5 - # via -r requirements/app.in -contourpy==1.2.1 +contourpy==1.3.0 # via matplotlib convertdate==2.4.0 # via workalendar @@ -50,13 +56,13 @@ dill==0.3.8 # via openturns dnspython==2.6.1 # via email-validator -email-validator==2.1.1 +email-validator==2.2.0 # via # -r requirements/app.in # flask-security-too -filelock==3.13.3 +filelock==3.16.1 # via tldextract -flask==2.2.5 +flask==3.0.3 # via # -r requirements/app.in # flask-classful @@ -75,7 +81,7 @@ flask==2.2.5 # sentry-sdk flask-classful==0.16.0 # via -r requirements/app.in -flask-cors==4.0.0 +flask-cors==5.0.0 # via -r requirements/app.in flask-json==0.4.0 # via -r requirements/app.in @@ -83,7 +89,7 @@ flask-login==0.6.3 # via # -r requirements/app.in # flask-security-too -flask-mail==0.9.1 +flask-mail==0.10.0 # via -r requirements/app.in flask-marshmallow==1.2.1 # via -r requirements/app.in @@ -91,7 +97,7 @@ flask-migrate==4.0.7 # via -r requirements/app.in flask-principal==0.4.0 # via flask-security-too -flask-security-too==5.1.2 +flask-security-too==5.5.2 # via -r requirements/app.in flask-sqlalchemy==3.1.1 # via @@ -103,21 +109,27 @@ flask-wtf==1.2.1 # via # -r requirements/app.in # flask-security-too -fonttools==4.51.0 +flexcache==0.3 + # via pint +flexparser==0.3.1 + # via pint +fonttools==4.53.1 # via matplotlib -greenlet==3.0.3 +greenlet==3.1.0 # via sqlalchemy -humanize==4.9.0 +humanize==4.10.0 # via -r requirements/app.in -idna==3.6 +idna==3.10 # via # email-validator # requests # tldextract -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -r requirements/app.in # timely-beliefs +importlib-resources==6.4.5 + # via flask-security-too inflect==6.0.2 # via -r requirements/app.in inflection==0.5.1 @@ -128,35 +140,37 @@ isodate==0.6.1 # via # -r requirements/app.in # timely-beliefs -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # flask - # flask-security-too # flask-wtf -jinja2==3.1.3 +jinja2==3.1.4 # via # altair # flask -joblib==1.3.2 - # via scikit-learn -jsonschema==4.21.1 +joblib==1.4.2 + # via + # scikit-learn + # sktime +jsonschema==4.23.0 # via altair jsonschema-specifications==2023.12.1 # via jsonschema -kiwisolver==1.4.5 +kiwisolver==1.4.7 # via matplotlib lunardate==0.2.2 # via workalendar -mako==1.3.2 +mako==1.3.5 # via alembic markupsafe==2.1.5 # via + # flask-security-too # jinja2 # mako # sentry-sdk # werkzeug # wtforms -marshmallow==3.21.1 +marshmallow==3.22.0 # via # -r requirements/app.in # flask-marshmallow @@ -165,14 +179,14 @@ marshmallow==3.21.1 # webargs marshmallow-polyfield==5.11 # via -r requirements/app.in -marshmallow-sqlalchemy==1.0.0 +marshmallow-sqlalchemy==1.1.0 # via -r requirements/app.in -matplotlib==3.8.4 +matplotlib==3.9.2 # via timetomodel -numpy==1.24.4 +narwhals==1.8.2 + # via altair +numpy==1.26.4 # via - # -r requirements/app.in - # altair # contourpy # matplotlib # pandas @@ -185,9 +199,9 @@ numpy==1.24.4 # timely-beliefs # timetomodel # uniplot -openturns==1.22 +openturns==1.23 # via timely-beliefs -packaging==24.0 +packaging==24.1 # via # altair # marshmallow @@ -198,7 +212,6 @@ packaging==24.0 pandas==2.2.1 # via # -r requirements/app.in - # altair # sktime # statsmodels # timely-beliefs @@ -207,19 +220,17 @@ passlib==1.7.4 # via flask-security-too patsy==0.5.6 # via statsmodels -pillow==10.3.0 +pillow==10.4.0 # via # -r requirements/app.in # matplotlib -pint==0.23 +pint==0.24.3 # via -r requirements/app.in ply==3.11 # via pyomo properscoring==0.1 # via timely-beliefs -pscript==0.7.7 - # via -r requirements/app.in -psutil==5.9.8 +psutil==6.0.0 # via openturns psycopg2-binary==2.9.9 # via @@ -227,7 +238,9 @@ psycopg2-binary==2.9.9 # timely-beliefs py-moneyed==3.0 # via -r requirements/app.in -pydantic==1.10.15 +pycparser==2.22 + # via cffi +pydantic==1.10.18 # via # -r requirements/app.in # inflect @@ -235,9 +248,9 @@ pyluach==2.2.0 # via workalendar pymeeus==0.5.12 # via convertdate -pyomo==6.7.1 +pyomo==6.8.0 # via -r requirements/app.in -pyparsing==3.1.2 +pyparsing==3.1.4 # via matplotlib python-dateutil==2.9.0.post0 # via @@ -248,15 +261,15 @@ python-dateutil==2.9.0.post0 # workalendar python-dotenv==1.0.1 # via -r requirements/app.in -pytz==2024.1 +pytz==2024.2 # via # -r requirements/app.in # pandas # timely-beliefs # timetomodel -pyyaml==6.0.1 +pyyaml==6.0.2 # via -r requirements/app.in -redis==4.6.0 +redis==5.0.8 # via # -r requirements/app.in # redis-sentinel-url @@ -264,33 +277,33 @@ redis==4.6.0 # rq-dashboard redis-sentinel-url==1.0.1 # via rq-dashboard -referencing==0.34.0 +referencing==0.35.1 # via # jsonschema # jsonschema-specifications -requests==2.31.0 +requests==2.32.3 # via # requests-file # tldextract -requests-file==2.0.0 +requests-file==2.1.0 # via tldextract -rpds-py==0.18.0 +rpds-py==0.20.0 # via # jsonschema # referencing -rq==1.16.1 +rq==1.16.2 # via # -r requirements/app.in # rq-dashboard -rq-dashboard==0.6.7.2 +rq-dashboard==0.8.0.2 # via -r requirements/app.in -scikit-base==0.7.5 +scikit-base==0.9.0 # via sktime -scikit-learn==1.4.1.post1 +scikit-learn==1.5.2 # via # sktime # timetomodel -scipy==1.13.0 +scipy==1.14.1 # via # properscoring # scikit-learn @@ -298,16 +311,16 @@ scipy==1.13.0 # statsmodels # timely-beliefs # timetomodel -sentry-sdk[flask]==1.44.1 +sentry-sdk[flask]==2.14.0 # via -r requirements/app.in six==1.16.0 # via # isodate # patsy # python-dateutil -sktime==0.28.0 +sktime==0.33.0 # via timely-beliefs -sqlalchemy==2.0.29 +sqlalchemy==2.0.35 # via # -r requirements/app.in # alembic @@ -315,42 +328,43 @@ sqlalchemy==2.0.29 # marshmallow-sqlalchemy # timely-beliefs # timetomodel -statsmodels==0.14.1 +statsmodels==0.14.3 # via timetomodel tabulate==0.9.0 # via -r requirements/app.in -threadpoolctl==3.4.0 +threadpoolctl==3.5.0 # via scikit-learn -timely-beliefs[forecast]==2.3.0 +timely-beliefs[forecast]==3.1.0 # via -r requirements/app.in timetomodel==0.7.3 # via -r requirements/app.in tldextract==5.1.2 # via -r requirements/app.in -toolz==0.12.1 - # via altair -types-python-dateutil==2.9.0.20240316 +types-python-dateutil==2.9.0.20240906 # via arrow -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # alembic + # altair + # flexcache + # flexparser # pint # py-moneyed # pydantic # sqlalchemy tzdata==2024.1 # via pandas -uniplot==0.12.5 +uniplot==0.14.1 # via -r requirements/app.in -urllib3==2.2.1 +urllib3==2.2.3 # via # requests # sentry-sdk -vl-convert-python==1.3.0 +vl-convert-python==1.6.1 # via -r requirements/app.in -webargs==8.4.0 +webargs==8.6.0 # via -r requirements/app.in -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -r requirements/app.in # flask @@ -363,8 +377,5 @@ wtforms==3.1.2 # flask-wtf xlrd==2.0.1 # via -r requirements/app.in -zipp==3.18.1 +zipp==3.20.2 # via importlib-metadata - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/3.11/dev.txt b/requirements/3.11/dev.txt index 6dd137150..3405483c7 100644 --- a/requirements/3.11/dev.txt +++ b/requirements/3.11/dev.txt @@ -4,7 +4,7 @@ # # pip-compile --constraint=requirements/3.11/app.txt --constraint=requirements/3.11/test.txt --output-file=requirements/3.11/dev.txt requirements/dev.in # -black==22.3.0 +black==24.8.0 # via -r requirements/dev.in cfgv==3.4.0 # via pre-commit @@ -15,61 +15,61 @@ click==8.1.7 # black distlib==0.3.8 # via virtualenv -filelock==3.13.3 +filelock==3.16.1 # via # -c requirements/3.11/app.txt # virtualenv -flake8==4.0.1 +flake8==7.1.1 # via -r requirements/dev.in flake8-blind-except==0.2.1 # via -r requirements/dev.in -identify==2.5.35 +identify==2.6.1 # via pre-commit -mccabe==0.6.1 +mccabe==0.7.0 # via flake8 -mypy==1.9.0 +mypy==1.11.2 # via -r requirements/dev.in mypy-extensions==1.0.0 # via # black # mypy -nodeenv==1.8.0 +nodeenv==1.9.1 # via pre-commit -packaging==24.0 +packaging==24.1 # via # -c requirements/3.11/app.txt # -c requirements/3.11/test.txt + # black # setuptools-scm pathspec==0.12.1 # via black -platformdirs==4.2.0 +platformdirs==4.3.6 # via # black # virtualenv -pre-commit==3.7.0 +pre-commit==3.8.0 # via -r requirements/dev.in -pycodestyle==2.8.0 +pycodestyle==2.12.1 # via flake8 -pyflakes==2.4.0 +pyflakes==3.2.0 # via flake8 -pyinstrument==4.6.2 +pyinstrument==4.7.3 # via -r requirements/dev.in pytest-runner==6.0.1 # via -r requirements/dev.in -pyyaml==6.0.1 +pyyaml==6.0.2 # via # -c requirements/3.11/app.txt # pre-commit -setuptools-scm==8.0.4 +setuptools-scm==8.1.0 # via -r requirements/dev.in -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # -c requirements/3.11/app.txt # mypy - # setuptools-scm -virtualenv==20.25.1 +virtualenv==20.26.5 # via pre-commit -watchdog==4.0.0 +watchdog==5.0.2 # via -r requirements/dev.in # The following packages are considered to be unsafe in a requirements file: diff --git a/requirements/3.11/docs.txt b/requirements/3.11/docs.txt index 51fdfa5ac..5000e7f9d 100644 --- a/requirements/3.11/docs.txt +++ b/requirements/3.11/docs.txt @@ -6,11 +6,11 @@ # alabaster==0.7.16 # via sphinx -babel==2.14.0 +babel==2.16.0 # via # -c requirements/3.11/app.txt # sphinx -certifi==2024.2.2 +certifi==2024.8.30 # via # -c requirements/3.11/app.txt # requests @@ -23,13 +23,13 @@ docutils==0.20.1 # sphinx # sphinx-rtd-theme # sphinx-tabs -idna==3.6 +idna==3.10 # via # -c requirements/3.11/app.txt # requests imagesize==1.4.1 # via sphinx -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.11/app.txt # sphinx @@ -37,15 +37,15 @@ markupsafe==2.1.5 # via # -c requirements/3.11/app.txt # jinja2 -packaging==24.0 +packaging==24.1 # via # -c requirements/3.11/app.txt # sphinx -pygments==2.17.2 +pygments==2.18.0 # via # sphinx # sphinx-tabs -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.11/app.txt # sphinx @@ -55,7 +55,7 @@ six==1.16.0 # sphinxcontrib-httpdomain snowballstemmer==2.2.0 # via sphinx -sphinx==7.2.6 +sphinx==7.3.7 # via # -r requirements/docs.in # sphinx-copybutton @@ -72,11 +72,11 @@ sphinx-rtd-theme==2.0.0 # via -r requirements/docs.in sphinx-tabs==3.4.5 # via -r requirements/docs.in -sphinxcontrib-applehelp==1.0.8 +sphinxcontrib-applehelp==2.0.0 # via sphinx -sphinxcontrib-devhelp==1.0.6 +sphinxcontrib-devhelp==2.0.0 # via sphinx -sphinxcontrib-htmlhelp==2.0.5 +sphinxcontrib-htmlhelp==2.1.0 # via sphinx sphinxcontrib-httpdomain==1.8.1 # via -r requirements/docs.in @@ -84,11 +84,11 @@ sphinxcontrib-jquery==4.1 # via sphinx-rtd-theme sphinxcontrib-jsmath==1.0.1 # via sphinx -sphinxcontrib-qthelp==1.0.7 +sphinxcontrib-qthelp==2.0.0 # via sphinx -sphinxcontrib-serializinghtml==1.1.10 +sphinxcontrib-serializinghtml==2.0.0 # via sphinx -urllib3==2.2.1 +urllib3==2.2.3 # via # -c requirements/3.11/app.txt # requests diff --git a/requirements/3.11/test.txt b/requirements/3.11/test.txt index db90f8874..1e54b4542 100644 --- a/requirements/3.11/test.txt +++ b/requirements/3.11/test.txt @@ -4,7 +4,11 @@ # # pip-compile --constraint=requirements/3.11/app.txt --output-file=requirements/3.11/test.txt requirements/test.in # -certifi==2024.2.2 +blinker==1.8.2 + # via + # -c requirements/3.11/app.txt + # flask +certifi==2024.8.30 # via # -c requirements/3.11/app.txt # requests @@ -16,45 +20,49 @@ click==8.1.7 # via # -c requirements/3.11/app.txt # flask -coverage[toml]==7.4.4 +coverage[toml]==7.6.1 # via pytest-cov -fakeredis==2.21.3 +fakeredis==2.24.1 # via -r requirements/test.in -flask==2.2.5 +flask==3.0.3 # via # -c requirements/3.11/app.txt # pytest-flask -highspy==1.5.3 +highspy==1.7.2 # via -r requirements/test.in -idna==3.6 +idna==3.10 # via # -c requirements/3.11/app.txt # requests iniconfig==2.0.0 # via pytest -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # -c requirements/3.11/app.txt # flask -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.11/app.txt # flask -lupa==2.1 +lupa==2.2 # via -r requirements/test.in markupsafe==2.1.5 # via # -c requirements/3.11/app.txt # jinja2 # werkzeug -packaging==24.0 +numpy==1.26.4 + # via + # -c requirements/3.11/app.txt + # highspy +packaging==24.1 # via # -c requirements/3.11/app.txt # pytest # pytest-sugar -pluggy==1.4.0 +pluggy==1.5.0 # via pytest -pytest==8.1.1 +pytest==8.3.3 # via # -r requirements/test.in # pytest-cov @@ -69,11 +77,11 @@ pytest-mock==3.14.0 # via -r requirements/test.in pytest-sugar==1.0.0 # via -r requirements/test.in -redis==4.6.0 +redis==5.0.8 # via # -c requirements/3.11/app.txt # fakeredis -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.11/app.txt # -r requirements/test.in @@ -84,11 +92,11 @@ sortedcontainers==2.4.0 # via fakeredis termcolor==2.4.0 # via pytest-sugar -urllib3==2.2.1 +urllib3==2.2.3 # via # -c requirements/3.11/app.txt # requests -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -c requirements/3.11/app.txt # flask diff --git a/requirements/3.12/app.txt b/requirements/3.12/app.txt new file mode 100644 index 000000000..f495d929c --- /dev/null +++ b/requirements/3.12/app.txt @@ -0,0 +1,380 @@ +# +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: +# +# pip-compile --output-file=requirements/3.12/app.txt requirements/app.in +# +alembic==1.14.0 + # via flask-migrate +altair==5.4.1 + # via -r requirements/app.in +appdirs==1.4.4 + # via pint +argon2-cffi==23.1.0 + # via -r requirements/app.in +argon2-cffi-bindings==21.2.0 + # via argon2-cffi +arrow==1.3.0 + # via rq-dashboard +attrs==24.2.0 + # via + # jsonschema + # referencing +babel==2.16.0 + # via py-moneyed +bcrypt==4.0.1 + # via -r requirements/app.in +blinker==1.8.2 + # via + # flask + # flask-mail + # flask-principal + # sentry-sdk +certifi==2024.8.30 + # via + # requests + # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings +charset-normalizer==3.4.0 + # via requests +click==8.1.7 + # via + # -r requirements/app.in + # click-default-group + # flask + # rq +click-default-group==1.2.4 + # via -r requirements/app.in +contourpy==1.3.0 + # via matplotlib +convertdate==2.4.0 + # via workalendar +cycler==0.12.1 + # via matplotlib +dill==0.3.9 + # via openturns +dnspython==2.7.0 + # via email-validator +email-validator==2.2.0 + # via + # -r requirements/app.in + # flask-security-too +filelock==3.16.1 + # via tldextract +flask==3.0.3 + # via + # -r requirements/app.in + # flask-classful + # flask-cors + # flask-json + # flask-login + # flask-mail + # flask-marshmallow + # flask-migrate + # flask-principal + # flask-security-too + # flask-sqlalchemy + # flask-sslify + # flask-wtf + # rq-dashboard + # sentry-sdk +flask-classful==0.16.0 + # via -r requirements/app.in +flask-cors==5.0.0 + # via -r requirements/app.in +flask-json==0.4.0 + # via -r requirements/app.in +flask-login==0.6.3 + # via + # -r requirements/app.in + # flask-security-too +flask-mail==0.10.0 + # via -r requirements/app.in +flask-marshmallow==1.2.1 + # via -r requirements/app.in +flask-migrate==4.0.7 + # via -r requirements/app.in +flask-principal==0.4.0 + # via flask-security-too +flask-security-too==5.5.2 + # via -r requirements/app.in +flask-sqlalchemy==3.1.1 + # via + # -r requirements/app.in + # flask-migrate +flask-sslify==0.1.5 + # via -r requirements/app.in +flask-wtf==1.2.2 + # via + # -r requirements/app.in + # flask-security-too +flexcache==0.3 + # via pint +flexparser==0.3.1 + # via pint +fonttools==4.54.1 + # via matplotlib +greenlet==3.1.1 + # via sqlalchemy +humanize==4.11.0 + # via -r requirements/app.in +idna==3.10 + # via + # email-validator + # requests + # tldextract +importlib-metadata==8.5.0 + # via + # -r requirements/app.in + # timely-beliefs +importlib-resources==6.4.5 + # via flask-security-too +inflect==6.0.2 + # via -r requirements/app.in +inflection==0.5.1 + # via -r requirements/app.in +iso8601==2.1.0 + # via -r requirements/app.in +isodate==0.7.2 + # via + # -r requirements/app.in + # timely-beliefs +itsdangerous==2.2.0 + # via + # flask + # flask-wtf +jinja2==3.1.4 + # via + # altair + # flask +joblib==1.4.2 + # via + # scikit-learn + # sktime +jsonschema==4.23.0 + # via altair +jsonschema-specifications==2024.10.1 + # via jsonschema +kiwisolver==1.4.7 + # via matplotlib +lunardate==0.2.2 + # via workalendar +mako==1.3.6 + # via alembic +markupsafe==3.0.2 + # via + # flask-security-too + # jinja2 + # mako + # sentry-sdk + # werkzeug + # wtforms +marshmallow==3.23.1 + # via + # -r requirements/app.in + # flask-marshmallow + # marshmallow-polyfield + # marshmallow-sqlalchemy + # webargs +marshmallow-polyfield==5.11 + # via -r requirements/app.in +marshmallow-sqlalchemy==1.1.0 + # via -r requirements/app.in +matplotlib==3.9.2 + # via timetomodel +narwhals==1.13.1 + # via altair +numpy==1.26.4 + # via + # contourpy + # matplotlib + # pandas + # patsy + # properscoring + # scikit-learn + # scipy + # sktime + # statsmodels + # timely-beliefs + # timetomodel + # uniplot +openturns==1.23 + # via timely-beliefs +packaging==24.1 + # via + # altair + # marshmallow + # matplotlib + # sktime + # statsmodels + # webargs +pandas==2.2.1 + # via + # -r requirements/app.in + # sktime + # statsmodels + # timely-beliefs + # timetomodel +passlib==1.7.4 + # via flask-security-too +patsy==0.5.6 + # via statsmodels +pillow==11.0.0 + # via + # -r requirements/app.in + # matplotlib +pint==0.24.3 + # via -r requirements/app.in +ply==3.11 + # via pyomo +properscoring==0.1 + # via timely-beliefs +psutil==6.1.0 + # via openturns +psycopg2-binary==2.9.10 + # via + # -r requirements/app.in + # timely-beliefs +py-moneyed==3.0 + # via -r requirements/app.in +pycparser==2.22 + # via cffi +pydantic==1.10.18 + # via + # -r requirements/app.in + # inflect +pyluach==2.2.0 + # via workalendar +pymeeus==0.5.12 + # via convertdate +pyomo==6.8.0 + # via -r requirements/app.in +pyparsing==3.2.0 + # via matplotlib +python-dateutil==2.9.0.post0 + # via + # arrow + # matplotlib + # pandas + # timetomodel + # workalendar +python-dotenv==1.0.1 + # via -r requirements/app.in +pytz==2024.2 + # via + # -r requirements/app.in + # pandas + # timely-beliefs + # timetomodel +pyyaml==6.0.2 + # via -r requirements/app.in +redis==5.2.0 + # via + # -r requirements/app.in + # redis-sentinel-url + # rq + # rq-dashboard +redis-sentinel-url==1.0.1 + # via rq-dashboard +referencing==0.35.1 + # via + # jsonschema + # jsonschema-specifications +requests==2.32.3 + # via + # requests-file + # tldextract +requests-file==2.1.0 + # via tldextract +rpds-py==0.20.1 + # via + # jsonschema + # referencing +rq==2.0.0 + # via + # -r requirements/app.in + # rq-dashboard +rq-dashboard==0.8.2.2 + # via -r requirements/app.in +scikit-base==0.11.0 + # via sktime +scikit-learn==1.5.2 + # via + # sktime + # timetomodel +scipy==1.14.1 + # via + # properscoring + # scikit-learn + # sktime + # statsmodels + # timely-beliefs + # timetomodel +sentry-sdk[flask]==2.18.0 + # via -r requirements/app.in +six==1.16.0 + # via + # patsy + # python-dateutil +sktime==0.34.0 + # via timely-beliefs +sqlalchemy==2.0.36 + # via + # -r requirements/app.in + # alembic + # flask-sqlalchemy + # marshmallow-sqlalchemy + # timely-beliefs + # timetomodel +statsmodels==0.14.4 + # via timetomodel +tabulate==0.9.0 + # via -r requirements/app.in +threadpoolctl==3.5.0 + # via scikit-learn +timely-beliefs[forecast]==3.1.0 + # via -r requirements/app.in +timetomodel==0.7.3 + # via -r requirements/app.in +tldextract==5.1.3 + # via -r requirements/app.in +types-python-dateutil==2.9.0.20241003 + # via arrow +typing-extensions==4.12.2 + # via + # alembic + # altair + # flexcache + # flexparser + # pint + # py-moneyed + # pydantic + # sqlalchemy +tzdata==2024.2 + # via pandas +uniplot==0.15.1 + # via -r requirements/app.in +urllib3==2.2.3 + # via + # requests + # sentry-sdk +vl-convert-python==1.7.0 + # via -r requirements/app.in +webargs==8.6.0 + # via -r requirements/app.in +werkzeug==3.1.2 + # via + # -r requirements/app.in + # flask + # flask-login +workalendar==17.0.0 + # via -r requirements/app.in +wtforms==3.2.1 + # via + # flask-security-too + # flask-wtf +xlrd==2.0.1 + # via -r requirements/app.in +zipp==3.20.2 + # via importlib-metadata diff --git a/requirements/3.12/dev.txt b/requirements/3.12/dev.txt new file mode 100644 index 000000000..070a04f9b --- /dev/null +++ b/requirements/3.12/dev.txt @@ -0,0 +1,76 @@ +# +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: +# +# pip-compile --constraint=requirements/3.12/app.txt --constraint=requirements/3.12/test.txt --output-file=requirements/3.12/dev.txt requirements/dev.in +# +black==24.8.0 + # via -r requirements/dev.in +cfgv==3.4.0 + # via pre-commit +click==8.1.7 + # via + # -c requirements/3.12/app.txt + # -c requirements/3.12/test.txt + # black +distlib==0.3.9 + # via virtualenv +filelock==3.16.1 + # via + # -c requirements/3.12/app.txt + # virtualenv +flake8==7.1.1 + # via -r requirements/dev.in +flake8-blind-except==0.2.1 + # via -r requirements/dev.in +identify==2.6.1 + # via pre-commit +mccabe==0.7.0 + # via flake8 +mypy==1.13.0 + # via -r requirements/dev.in +mypy-extensions==1.0.0 + # via + # black + # mypy +nodeenv==1.9.1 + # via pre-commit +packaging==24.1 + # via + # -c requirements/3.12/app.txt + # -c requirements/3.12/test.txt + # black + # setuptools-scm +pathspec==0.12.1 + # via black +platformdirs==4.3.6 + # via + # black + # virtualenv +pre-commit==4.0.1 + # via -r requirements/dev.in +pycodestyle==2.12.1 + # via flake8 +pyflakes==3.2.0 + # via flake8 +pyinstrument==5.0.0 + # via -r requirements/dev.in +pytest-runner==6.0.1 + # via -r requirements/dev.in +pyyaml==6.0.2 + # via + # -c requirements/3.12/app.txt + # pre-commit +setuptools-scm==8.1.0 + # via -r requirements/dev.in +typing-extensions==4.12.2 + # via + # -c requirements/3.12/app.txt + # mypy +virtualenv==20.27.1 + # via pre-commit +watchdog==6.0.0 + # via -r requirements/dev.in + +# The following packages are considered to be unsafe in a requirements file: +# setuptools diff --git a/requirements/3.12/docs.txt b/requirements/3.12/docs.txt new file mode 100644 index 000000000..12130fe0d --- /dev/null +++ b/requirements/3.12/docs.txt @@ -0,0 +1,94 @@ +# +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: +# +# pip-compile --constraint=requirements/3.12/app.txt --output-file=requirements/3.12/docs.txt requirements/docs.in +# +alabaster==1.0.0 + # via sphinx +babel==2.16.0 + # via + # -c requirements/3.12/app.txt + # sphinx +certifi==2024.8.30 + # via + # -c requirements/3.12/app.txt + # requests +charset-normalizer==3.4.0 + # via + # -c requirements/3.12/app.txt + # requests +docutils==0.21.2 + # via + # sphinx + # sphinx-rtd-theme + # sphinx-tabs +idna==3.10 + # via + # -c requirements/3.12/app.txt + # requests +imagesize==1.4.1 + # via sphinx +jinja2==3.1.4 + # via + # -c requirements/3.12/app.txt + # sphinx +markupsafe==3.0.2 + # via + # -c requirements/3.12/app.txt + # jinja2 +packaging==24.1 + # via + # -c requirements/3.12/app.txt + # sphinx +pygments==2.18.0 + # via + # sphinx + # sphinx-tabs +requests==2.32.3 + # via + # -c requirements/3.12/app.txt + # sphinx +six==1.16.0 + # via + # -c requirements/3.12/app.txt + # sphinxcontrib-httpdomain +snowballstemmer==2.2.0 + # via sphinx +sphinx==8.1.3 + # via + # -r requirements/docs.in + # sphinx-copybutton + # sphinx-fontawesome + # sphinx-rtd-theme + # sphinx-tabs + # sphinxcontrib-httpdomain + # sphinxcontrib-jquery +sphinx-copybutton==0.5.2 + # via -r requirements/docs.in +sphinx-fontawesome==0.0.6 + # via -r requirements/docs.in +sphinx-rtd-theme==3.0.1 + # via -r requirements/docs.in +sphinx-tabs==3.4.7 + # via -r requirements/docs.in +sphinxcontrib-applehelp==2.0.0 + # via sphinx +sphinxcontrib-devhelp==2.0.0 + # via sphinx +sphinxcontrib-htmlhelp==2.1.0 + # via sphinx +sphinxcontrib-httpdomain==1.8.1 + # via -r requirements/docs.in +sphinxcontrib-jquery==4.1 + # via sphinx-rtd-theme +sphinxcontrib-jsmath==1.0.1 + # via sphinx +sphinxcontrib-qthelp==2.0.0 + # via sphinx +sphinxcontrib-serializinghtml==2.0.0 + # via sphinx +urllib3==2.2.3 + # via + # -c requirements/3.12/app.txt + # requests diff --git a/requirements/3.12/test.txt b/requirements/3.12/test.txt new file mode 100644 index 000000000..de3c6ac65 --- /dev/null +++ b/requirements/3.12/test.txt @@ -0,0 +1,103 @@ +# +# This file is autogenerated by pip-compile with Python 3.12 +# by the following command: +# +# pip-compile --constraint=requirements/3.12/app.txt --output-file=requirements/3.12/test.txt requirements/test.in +# +blinker==1.8.2 + # via + # -c requirements/3.12/app.txt + # flask +certifi==2024.8.30 + # via + # -c requirements/3.12/app.txt + # requests +charset-normalizer==3.4.0 + # via + # -c requirements/3.12/app.txt + # requests +click==8.1.7 + # via + # -c requirements/3.12/app.txt + # flask +coverage[toml]==7.6.4 + # via pytest-cov +fakeredis==2.26.1 + # via -r requirements/test.in +flask==3.0.3 + # via + # -c requirements/3.12/app.txt + # pytest-flask +highspy==1.8.0 + # via -r requirements/test.in +idna==3.10 + # via + # -c requirements/3.12/app.txt + # requests +iniconfig==2.0.0 + # via pytest +itsdangerous==2.2.0 + # via + # -c requirements/3.12/app.txt + # flask +jinja2==3.1.4 + # via + # -c requirements/3.12/app.txt + # flask +lupa==2.2 + # via -r requirements/test.in +markupsafe==3.0.2 + # via + # -c requirements/3.12/app.txt + # jinja2 + # werkzeug +numpy==1.26.4 + # via + # -c requirements/3.12/app.txt + # highspy +packaging==24.1 + # via + # -c requirements/3.12/app.txt + # pytest + # pytest-sugar +pluggy==1.5.0 + # via pytest +pytest==8.3.3 + # via + # -r requirements/test.in + # pytest-cov + # pytest-flask + # pytest-mock + # pytest-sugar +pytest-cov==6.0.0 + # via -r requirements/test.in +pytest-flask==1.3.0 + # via -r requirements/test.in +pytest-mock==3.14.0 + # via -r requirements/test.in +pytest-sugar==1.0.0 + # via -r requirements/test.in +redis==5.2.0 + # via + # -c requirements/3.12/app.txt + # fakeredis +requests==2.32.3 + # via + # -c requirements/3.12/app.txt + # -r requirements/test.in + # requests-mock +requests-mock==1.12.1 + # via -r requirements/test.in +sortedcontainers==2.4.0 + # via fakeredis +termcolor==2.5.0 + # via pytest-sugar +urllib3==2.2.3 + # via + # -c requirements/3.12/app.txt + # requests +werkzeug==3.1.2 + # via + # -c requirements/3.12/app.txt + # flask + # pytest-flask diff --git a/requirements/3.8/app.txt b/requirements/3.8/app.txt index b822d92b8..fbf61d932 100644 --- a/requirements/3.8/app.txt +++ b/requirements/3.8/app.txt @@ -4,34 +4,40 @@ # # pip-compile --output-file=requirements/3.8/app.txt requirements/app.in # -alembic==1.13.1 +alembic==1.13.2 # via flask-migrate -altair==5.3.0 +altair==5.4.1 # via -r requirements/app.in +argon2-cffi==23.1.0 + # via -r requirements/app.in +argon2-cffi-bindings==21.2.0 + # via argon2-cffi arrow==1.3.0 # via rq-dashboard async-timeout==4.0.3 # via redis -attrs==23.2.0 +attrs==24.2.0 # via # jsonschema # referencing -babel==2.14.0 +babel==2.16.0 # via py-moneyed backports-zoneinfo==0.2.1 # via workalendar bcrypt==4.0.1 # via -r requirements/app.in -blinker==1.7.0 +blinker==1.8.2 # via + # flask # flask-mail # flask-principal - # flask-security-too # sentry-sdk -certifi==2024.2.2 +certifi==2024.8.30 # via # requests # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings charset-normalizer==3.3.2 # via requests click==8.1.7 @@ -42,8 +48,6 @@ click==8.1.7 # rq click-default-group==1.2.4 # via -r requirements/app.in -colour==0.1.5 - # via -r requirements/app.in contourpy==1.1.1 # via matplotlib convertdate==2.4.0 @@ -54,13 +58,13 @@ dill==0.3.8 # via openturns dnspython==2.6.1 # via email-validator -email-validator==2.1.1 +email-validator==2.2.0 # via # -r requirements/app.in # flask-security-too -filelock==3.13.3 +filelock==3.16.1 # via tldextract -flask==2.2.5 +flask==3.0.3 # via # -r requirements/app.in # flask-classful @@ -79,7 +83,7 @@ flask==2.2.5 # sentry-sdk flask-classful==0.16.0 # via -r requirements/app.in -flask-cors==4.0.0 +flask-cors==5.0.0 # via -r requirements/app.in flask-json==0.4.0 # via -r requirements/app.in @@ -87,7 +91,7 @@ flask-login==0.6.3 # via # -r requirements/app.in # flask-security-too -flask-mail==0.9.1 +flask-mail==0.10.0 # via -r requirements/app.in flask-marshmallow==1.2.1 # via -r requirements/app.in @@ -95,7 +99,7 @@ flask-migrate==4.0.7 # via -r requirements/app.in flask-principal==0.4.0 # via flask-security-too -flask-security-too==5.1.2 +flask-security-too==5.4.3 # via -r requirements/app.in flask-sqlalchemy==3.1.1 # via @@ -107,26 +111,27 @@ flask-wtf==1.2.1 # via # -r requirements/app.in # flask-security-too -fonttools==4.51.0 +fonttools==4.53.1 # via matplotlib -greenlet==3.0.3 +greenlet==3.1.0 # via sqlalchemy -humanize==4.9.0 +humanize==4.10.0 # via -r requirements/app.in -idna==3.6 +idna==3.10 # via # email-validator # requests # tldextract -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -r requirements/app.in # alembic # flask # timely-beliefs -importlib-resources==6.4.0 +importlib-resources==6.4.5 # via # alembic + # flask-security-too # jsonschema # jsonschema-specifications # matplotlib @@ -140,35 +145,35 @@ isodate==0.6.1 # via # -r requirements/app.in # timely-beliefs -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # flask - # flask-security-too # flask-wtf -jinja2==3.1.3 +jinja2==3.1.4 # via # altair # flask -joblib==1.3.2 +joblib==1.4.2 # via scikit-learn -jsonschema==4.21.1 +jsonschema==4.23.0 # via altair jsonschema-specifications==2023.12.1 # via jsonschema -kiwisolver==1.4.5 +kiwisolver==1.4.7 # via matplotlib lunardate==0.2.2 # via workalendar -mako==1.3.2 +mako==1.3.5 # via alembic markupsafe==2.1.5 # via + # flask-security-too # jinja2 # mako # sentry-sdk # werkzeug # wtforms -marshmallow==3.21.1 +marshmallow==3.22.0 # via # -r requirements/app.in # flask-marshmallow @@ -177,14 +182,14 @@ marshmallow==3.21.1 # webargs marshmallow-polyfield==5.11 # via -r requirements/app.in -marshmallow-sqlalchemy==1.0.0 +marshmallow-sqlalchemy==1.1.0 # via -r requirements/app.in matplotlib==3.7.5 # via timetomodel +narwhals==1.8.2 + # via altair numpy==1.24.4 # via - # -r requirements/app.in - # altair # contourpy # matplotlib # pandas @@ -197,9 +202,9 @@ numpy==1.24.4 # timely-beliefs # timetomodel # uniplot -openturns==1.22 +openturns==1.23 # via timely-beliefs -packaging==24.0 +packaging==24.1 # via # altair # marshmallow @@ -210,7 +215,6 @@ packaging==24.0 pandas==2.0.3 # via # -r requirements/app.in - # altair # sktime # statsmodels # timely-beliefs @@ -219,7 +223,7 @@ passlib==1.7.4 # via flask-security-too patsy==0.5.6 # via statsmodels -pillow==10.3.0 +pillow==10.4.0 # via # -r requirements/app.in # matplotlib @@ -231,9 +235,7 @@ ply==3.11 # via pyomo properscoring==0.1 # via timely-beliefs -pscript==0.7.7 - # via -r requirements/app.in -psutil==5.9.8 +psutil==6.0.0 # via openturns psycopg2-binary==2.9.9 # via @@ -241,7 +243,9 @@ psycopg2-binary==2.9.9 # timely-beliefs py-moneyed==3.0 # via -r requirements/app.in -pydantic==1.10.15 +pycparser==2.22 + # via cffi +pydantic==1.10.18 # via # -r requirements/app.in # inflect @@ -249,9 +253,9 @@ pyluach==2.2.0 # via workalendar pymeeus==0.5.12 # via convertdate -pyomo==6.7.1 +pyomo==6.8.0 # via -r requirements/app.in -pyparsing==3.1.2 +pyparsing==3.1.4 # via matplotlib python-dateutil==2.9.0.post0 # via @@ -262,16 +266,16 @@ python-dateutil==2.9.0.post0 # workalendar python-dotenv==1.0.1 # via -r requirements/app.in -pytz==2024.1 +pytz==2024.2 # via # -r requirements/app.in # babel # pandas # timely-beliefs # timetomodel -pyyaml==6.0.1 +pyyaml==6.0.2 # via -r requirements/app.in -redis==4.6.0 +redis==5.0.8 # via # -r requirements/app.in # redis-sentinel-url @@ -279,27 +283,27 @@ redis==4.6.0 # rq-dashboard redis-sentinel-url==1.0.1 # via rq-dashboard -referencing==0.34.0 +referencing==0.35.1 # via # jsonschema # jsonschema-specifications -requests==2.31.0 +requests==2.32.3 # via # requests-file # tldextract -requests-file==2.0.0 +requests-file==2.1.0 # via tldextract -rpds-py==0.18.0 +rpds-py==0.20.0 # via # jsonschema # referencing -rq==1.16.1 +rq==1.16.2 # via # -r requirements/app.in # rq-dashboard -rq-dashboard==0.6.7.2 +rq-dashboard==0.8.0.2 # via -r requirements/app.in -scikit-base==0.7.5 +scikit-base==0.7.8 # via sktime scikit-learn==1.3.2 # via @@ -313,16 +317,16 @@ scipy==1.10.1 # statsmodels # timely-beliefs # timetomodel -sentry-sdk[flask]==1.44.1 +sentry-sdk[flask]==2.14.0 # via -r requirements/app.in six==1.16.0 # via # isodate # patsy # python-dateutil -sktime==0.28.0 +sktime==0.29.1 # via timely-beliefs -sqlalchemy==2.0.29 +sqlalchemy==2.0.35 # via # -r requirements/app.in # alembic @@ -334,19 +338,17 @@ statsmodels==0.14.1 # via timetomodel tabulate==0.9.0 # via -r requirements/app.in -threadpoolctl==3.4.0 +threadpoolctl==3.5.0 # via scikit-learn -timely-beliefs[forecast]==2.3.0 +timely-beliefs[forecast]==3.1.0 # via -r requirements/app.in timetomodel==0.7.3 # via -r requirements/app.in tldextract==5.1.2 # via -r requirements/app.in -toolz==0.12.1 - # via altair -types-python-dateutil==2.9.0.20240316 +types-python-dateutil==2.9.0.20240906 # via arrow -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # alembic # altair @@ -355,17 +357,17 @@ typing-extensions==4.11.0 # sqlalchemy tzdata==2024.1 # via pandas -uniplot==0.12.5 +uniplot==0.14.1 # via -r requirements/app.in -urllib3==2.2.1 +urllib3==2.2.3 # via # requests # sentry-sdk -vl-convert-python==1.3.0 +vl-convert-python==1.6.1 # via -r requirements/app.in -webargs==8.4.0 +webargs==8.6.0 # via -r requirements/app.in -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -r requirements/app.in # flask @@ -378,10 +380,7 @@ wtforms==3.1.2 # flask-wtf xlrd==2.0.1 # via -r requirements/app.in -zipp==3.18.1 +zipp==3.20.2 # via # importlib-metadata # importlib-resources - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/3.8/dev.txt b/requirements/3.8/dev.txt index b7ba06b41..a1320af90 100644 --- a/requirements/3.8/dev.txt +++ b/requirements/3.8/dev.txt @@ -4,7 +4,7 @@ # # pip-compile --constraint=requirements/3.8/app.txt --constraint=requirements/3.8/test.txt --output-file=requirements/3.8/dev.txt requirements/dev.in # -black==22.3.0 +black==24.8.0 # via -r requirements/dev.in cfgv==3.4.0 # via pre-commit @@ -15,52 +15,53 @@ click==8.1.7 # black distlib==0.3.8 # via virtualenv -filelock==3.13.3 +filelock==3.16.1 # via # -c requirements/3.8/app.txt # virtualenv -flake8==4.0.1 +flake8==7.1.1 # via -r requirements/dev.in flake8-blind-except==0.2.1 # via -r requirements/dev.in -identify==2.5.35 +identify==2.6.1 # via pre-commit -mccabe==0.6.1 +mccabe==0.7.0 # via flake8 -mypy==1.9.0 +mypy==1.11.2 # via -r requirements/dev.in mypy-extensions==1.0.0 # via # black # mypy -nodeenv==1.8.0 +nodeenv==1.9.1 # via pre-commit -packaging==24.0 +packaging==24.1 # via # -c requirements/3.8/app.txt # -c requirements/3.8/test.txt + # black # setuptools-scm pathspec==0.12.1 # via black -platformdirs==4.2.0 +platformdirs==4.3.6 # via # black # virtualenv pre-commit==3.5.0 # via -r requirements/dev.in -pycodestyle==2.8.0 +pycodestyle==2.12.1 # via flake8 -pyflakes==2.4.0 +pyflakes==3.2.0 # via flake8 -pyinstrument==4.6.2 +pyinstrument==4.7.3 # via -r requirements/dev.in pytest-runner==6.0.1 # via -r requirements/dev.in -pyyaml==6.0.1 +pyyaml==6.0.2 # via # -c requirements/3.8/app.txt # pre-commit -setuptools-scm==8.0.4 +setuptools-scm==8.1.0 # via -r requirements/dev.in tomli==2.0.1 # via @@ -68,15 +69,16 @@ tomli==2.0.1 # black # mypy # setuptools-scm -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # -c requirements/3.8/app.txt + # -c requirements/3.8/test.txt # black # mypy # setuptools-scm -virtualenv==20.25.1 +virtualenv==20.26.5 # via pre-commit -watchdog==4.0.0 +watchdog==4.0.2 # via -r requirements/dev.in # The following packages are considered to be unsafe in a requirements file: diff --git a/requirements/3.8/docs.txt b/requirements/3.8/docs.txt index 8ec8b78d2..f10561f04 100644 --- a/requirements/3.8/docs.txt +++ b/requirements/3.8/docs.txt @@ -6,11 +6,11 @@ # alabaster==0.7.13 # via sphinx -babel==2.14.0 +babel==2.16.0 # via # -c requirements/3.8/app.txt # sphinx -certifi==2024.2.2 +certifi==2024.8.30 # via # -c requirements/3.8/app.txt # requests @@ -23,17 +23,17 @@ docutils==0.20.1 # sphinx # sphinx-rtd-theme # sphinx-tabs -idna==3.6 +idna==3.10 # via # -c requirements/3.8/app.txt # requests imagesize==1.4.1 # via sphinx -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -c requirements/3.8/app.txt # sphinx -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.8/app.txt # sphinx @@ -41,19 +41,19 @@ markupsafe==2.1.5 # via # -c requirements/3.8/app.txt # jinja2 -packaging==24.0 +packaging==24.1 # via # -c requirements/3.8/app.txt # sphinx -pygments==2.17.2 +pygments==2.18.0 # via # sphinx # sphinx-tabs -pytz==2024.1 +pytz==2024.2 # via # -c requirements/3.8/app.txt # babel -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.8/app.txt # sphinx @@ -96,11 +96,11 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx -urllib3==2.2.1 +urllib3==2.2.3 # via # -c requirements/3.8/app.txt # requests -zipp==3.18.1 +zipp==3.20.2 # via # -c requirements/3.8/app.txt # importlib-metadata diff --git a/requirements/3.8/test.txt b/requirements/3.8/test.txt index be8227fad..eb0a50b42 100644 --- a/requirements/3.8/test.txt +++ b/requirements/3.8/test.txt @@ -8,7 +8,11 @@ async-timeout==4.0.3 # via # -c requirements/3.8/app.txt # redis -certifi==2024.2.2 +blinker==1.8.2 + # via + # -c requirements/3.8/app.txt + # flask +certifi==2024.8.30 # via # -c requirements/3.8/app.txt # requests @@ -20,51 +24,55 @@ click==8.1.7 # via # -c requirements/3.8/app.txt # flask -coverage[toml]==7.4.4 +coverage[toml]==7.6.1 # via pytest-cov -exceptiongroup==1.2.0 +exceptiongroup==1.2.2 # via pytest -fakeredis==2.21.3 +fakeredis==2.24.1 # via -r requirements/test.in -flask==2.2.5 +flask==3.0.3 # via # -c requirements/3.8/app.txt # pytest-flask -highspy==1.5.3 +highspy==1.7.2 # via -r requirements/test.in -idna==3.6 +idna==3.10 # via # -c requirements/3.8/app.txt # requests -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -c requirements/3.8/app.txt # flask iniconfig==2.0.0 # via pytest -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # -c requirements/3.8/app.txt # flask -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.8/app.txt # flask -lupa==2.1 +lupa==2.2 # via -r requirements/test.in markupsafe==2.1.5 # via # -c requirements/3.8/app.txt # jinja2 # werkzeug -packaging==24.0 +numpy==1.24.4 + # via + # -c requirements/3.8/app.txt + # highspy +packaging==24.1 # via # -c requirements/3.8/app.txt # pytest # pytest-sugar -pluggy==1.4.0 +pluggy==1.5.0 # via pytest -pytest==8.1.1 +pytest==8.3.3 # via # -r requirements/test.in # pytest-cov @@ -79,11 +87,11 @@ pytest-mock==3.14.0 # via -r requirements/test.in pytest-sugar==1.0.0 # via -r requirements/test.in -redis==4.6.0 +redis==5.0.8 # via # -c requirements/3.8/app.txt # fakeredis -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.8/app.txt # -r requirements/test.in @@ -98,16 +106,20 @@ tomli==2.0.1 # via # coverage # pytest -urllib3==2.2.1 +typing-extensions==4.12.2 + # via + # -c requirements/3.8/app.txt + # fakeredis +urllib3==2.2.3 # via # -c requirements/3.8/app.txt # requests -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -c requirements/3.8/app.txt # flask # pytest-flask -zipp==3.18.1 +zipp==3.20.2 # via # -c requirements/3.8/app.txt # importlib-metadata diff --git a/requirements/3.9/app.txt b/requirements/3.9/app.txt index f8a466457..34857dc35 100644 --- a/requirements/3.9/app.txt +++ b/requirements/3.9/app.txt @@ -4,32 +4,40 @@ # # pip-compile --output-file=requirements/3.9/app.txt requirements/app.in # -alembic==1.13.1 +alembic==1.13.2 # via flask-migrate -altair==5.3.0 +altair==5.4.1 # via -r requirements/app.in +appdirs==1.4.4 + # via pint +argon2-cffi==23.1.0 + # via -r requirements/app.in +argon2-cffi-bindings==21.2.0 + # via argon2-cffi arrow==1.3.0 # via rq-dashboard async-timeout==4.0.3 # via redis -attrs==23.2.0 +attrs==24.2.0 # via # jsonschema # referencing -babel==2.14.0 +babel==2.16.0 # via py-moneyed bcrypt==4.0.1 # via -r requirements/app.in -blinker==1.7.0 +blinker==1.8.2 # via + # flask # flask-mail # flask-principal - # flask-security-too # sentry-sdk -certifi==2024.2.2 +certifi==2024.8.30 # via # requests # sentry-sdk +cffi==1.17.1 + # via argon2-cffi-bindings charset-normalizer==3.3.2 # via requests click==8.1.7 @@ -40,9 +48,7 @@ click==8.1.7 # rq click-default-group==1.2.4 # via -r requirements/app.in -colour==0.1.5 - # via -r requirements/app.in -contourpy==1.2.1 +contourpy==1.3.0 # via matplotlib convertdate==2.4.0 # via workalendar @@ -52,13 +58,13 @@ dill==0.3.8 # via openturns dnspython==2.6.1 # via email-validator -email-validator==2.1.1 +email-validator==2.2.0 # via # -r requirements/app.in # flask-security-too -filelock==3.13.3 +filelock==3.16.1 # via tldextract -flask==2.2.5 +flask==3.0.3 # via # -r requirements/app.in # flask-classful @@ -77,7 +83,7 @@ flask==2.2.5 # sentry-sdk flask-classful==0.16.0 # via -r requirements/app.in -flask-cors==4.0.0 +flask-cors==5.0.0 # via -r requirements/app.in flask-json==0.4.0 # via -r requirements/app.in @@ -85,7 +91,7 @@ flask-login==0.6.3 # via # -r requirements/app.in # flask-security-too -flask-mail==0.9.1 +flask-mail==0.10.0 # via -r requirements/app.in flask-marshmallow==1.2.1 # via -r requirements/app.in @@ -93,7 +99,7 @@ flask-migrate==4.0.7 # via -r requirements/app.in flask-principal==0.4.0 # via flask-security-too -flask-security-too==5.1.2 +flask-security-too==5.5.2 # via -r requirements/app.in flask-sqlalchemy==3.1.1 # via @@ -105,24 +111,30 @@ flask-wtf==1.2.1 # via # -r requirements/app.in # flask-security-too -fonttools==4.51.0 +flexcache==0.3 + # via pint +flexparser==0.3.1 + # via pint +fonttools==4.53.1 # via matplotlib -greenlet==3.0.3 +greenlet==3.1.0 # via sqlalchemy -humanize==4.9.0 +humanize==4.10.0 # via -r requirements/app.in -idna==3.6 +idna==3.10 # via # email-validator # requests # tldextract -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -r requirements/app.in # flask # timely-beliefs -importlib-resources==6.4.0 - # via matplotlib +importlib-resources==6.4.5 + # via + # flask-security-too + # matplotlib inflect==6.0.2 # via -r requirements/app.in inflection==0.5.1 @@ -133,35 +145,37 @@ isodate==0.6.1 # via # -r requirements/app.in # timely-beliefs -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # flask - # flask-security-too # flask-wtf -jinja2==3.1.3 +jinja2==3.1.4 # via # altair # flask -joblib==1.3.2 - # via scikit-learn -jsonschema==4.21.1 +joblib==1.4.2 + # via + # scikit-learn + # sktime +jsonschema==4.23.0 # via altair jsonschema-specifications==2023.12.1 # via jsonschema -kiwisolver==1.4.5 +kiwisolver==1.4.7 # via matplotlib lunardate==0.2.2 # via workalendar -mako==1.3.2 +mako==1.3.5 # via alembic markupsafe==2.1.5 # via + # flask-security-too # jinja2 # mako # sentry-sdk # werkzeug # wtforms -marshmallow==3.21.1 +marshmallow==3.22.0 # via # -r requirements/app.in # flask-marshmallow @@ -170,14 +184,14 @@ marshmallow==3.21.1 # webargs marshmallow-polyfield==5.11 # via -r requirements/app.in -marshmallow-sqlalchemy==1.0.0 +marshmallow-sqlalchemy==1.1.0 # via -r requirements/app.in -matplotlib==3.8.4 +matplotlib==3.9.2 # via timetomodel -numpy==1.24.4 +narwhals==1.8.2 + # via altair +numpy==1.26.4 # via - # -r requirements/app.in - # altair # contourpy # matplotlib # pandas @@ -190,9 +204,9 @@ numpy==1.24.4 # timely-beliefs # timetomodel # uniplot -openturns==1.22 +openturns==1.23 # via timely-beliefs -packaging==24.0 +packaging==24.1 # via # altair # marshmallow @@ -203,7 +217,6 @@ packaging==24.0 pandas==2.2.1 # via # -r requirements/app.in - # altair # sktime # statsmodels # timely-beliefs @@ -212,19 +225,17 @@ passlib==1.7.4 # via flask-security-too patsy==0.5.6 # via statsmodels -pillow==10.3.0 +pillow==10.4.0 # via # -r requirements/app.in # matplotlib -pint==0.23 +pint==0.24.3 # via -r requirements/app.in ply==3.11 # via pyomo properscoring==0.1 # via timely-beliefs -pscript==0.7.7 - # via -r requirements/app.in -psutil==5.9.8 +psutil==6.0.0 # via openturns psycopg2-binary==2.9.9 # via @@ -232,7 +243,9 @@ psycopg2-binary==2.9.9 # timely-beliefs py-moneyed==3.0 # via -r requirements/app.in -pydantic==1.10.15 +pycparser==2.22 + # via cffi +pydantic==1.10.18 # via # -r requirements/app.in # inflect @@ -240,9 +253,9 @@ pyluach==2.2.0 # via workalendar pymeeus==0.5.12 # via convertdate -pyomo==6.7.1 +pyomo==6.8.0 # via -r requirements/app.in -pyparsing==3.1.2 +pyparsing==3.1.4 # via matplotlib python-dateutil==2.9.0.post0 # via @@ -253,15 +266,15 @@ python-dateutil==2.9.0.post0 # workalendar python-dotenv==1.0.1 # via -r requirements/app.in -pytz==2024.1 +pytz==2024.2 # via # -r requirements/app.in # pandas # timely-beliefs # timetomodel -pyyaml==6.0.1 +pyyaml==6.0.2 # via -r requirements/app.in -redis==4.6.0 +redis==5.0.8 # via # -r requirements/app.in # redis-sentinel-url @@ -269,33 +282,33 @@ redis==4.6.0 # rq-dashboard redis-sentinel-url==1.0.1 # via rq-dashboard -referencing==0.34.0 +referencing==0.35.1 # via # jsonschema # jsonschema-specifications -requests==2.31.0 +requests==2.32.3 # via # requests-file # tldextract -requests-file==2.0.0 +requests-file==2.1.0 # via tldextract -rpds-py==0.18.0 +rpds-py==0.20.0 # via # jsonschema # referencing -rq==1.16.1 +rq==1.16.2 # via # -r requirements/app.in # rq-dashboard -rq-dashboard==0.6.7.2 +rq-dashboard==0.8.0.2 # via -r requirements/app.in -scikit-base==0.7.5 +scikit-base==0.9.0 # via sktime -scikit-learn==1.4.1.post1 +scikit-learn==1.5.2 # via # sktime # timetomodel -scipy==1.13.0 +scipy==1.13.1 # via # properscoring # scikit-learn @@ -303,16 +316,16 @@ scipy==1.13.0 # statsmodels # timely-beliefs # timetomodel -sentry-sdk[flask]==1.44.1 +sentry-sdk[flask]==2.14.0 # via -r requirements/app.in six==1.16.0 # via # isodate # patsy # python-dateutil -sktime==0.28.0 +sktime==0.33.0 # via timely-beliefs -sqlalchemy==2.0.29 +sqlalchemy==2.0.35 # via # -r requirements/app.in # alembic @@ -320,43 +333,43 @@ sqlalchemy==2.0.29 # marshmallow-sqlalchemy # timely-beliefs # timetomodel -statsmodels==0.14.1 +statsmodels==0.14.3 # via timetomodel tabulate==0.9.0 # via -r requirements/app.in -threadpoolctl==3.4.0 +threadpoolctl==3.5.0 # via scikit-learn -timely-beliefs[forecast]==2.3.0 +timely-beliefs[forecast]==3.1.0 # via -r requirements/app.in timetomodel==0.7.3 # via -r requirements/app.in tldextract==5.1.2 # via -r requirements/app.in -toolz==0.12.1 - # via altair -types-python-dateutil==2.9.0.20240316 +types-python-dateutil==2.9.0.20240906 # via arrow -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # alembic # altair + # flexcache + # flexparser # pint # py-moneyed # pydantic # sqlalchemy tzdata==2024.1 # via pandas -uniplot==0.12.5 +uniplot==0.14.1 # via -r requirements/app.in -urllib3==2.2.1 +urllib3==2.2.3 # via # requests # sentry-sdk -vl-convert-python==1.3.0 +vl-convert-python==1.6.1 # via -r requirements/app.in -webargs==8.4.0 +webargs==8.6.0 # via -r requirements/app.in -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -r requirements/app.in # flask @@ -369,10 +382,7 @@ wtforms==3.1.2 # flask-wtf xlrd==2.0.1 # via -r requirements/app.in -zipp==3.18.1 +zipp==3.20.2 # via # importlib-metadata # importlib-resources - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/requirements/3.9/dev.txt b/requirements/3.9/dev.txt index 081c37254..f94b1e95d 100644 --- a/requirements/3.9/dev.txt +++ b/requirements/3.9/dev.txt @@ -4,7 +4,7 @@ # # pip-compile --constraint=requirements/3.9/app.txt --constraint=requirements/3.9/test.txt --output-file=requirements/3.9/dev.txt requirements/dev.in # -black==22.3.0 +black==24.8.0 # via -r requirements/dev.in cfgv==3.4.0 # via pre-commit @@ -15,52 +15,53 @@ click==8.1.7 # black distlib==0.3.8 # via virtualenv -filelock==3.13.3 +filelock==3.16.1 # via # -c requirements/3.9/app.txt # virtualenv -flake8==4.0.1 +flake8==7.1.1 # via -r requirements/dev.in flake8-blind-except==0.2.1 # via -r requirements/dev.in -identify==2.5.35 +identify==2.6.1 # via pre-commit -mccabe==0.6.1 +mccabe==0.7.0 # via flake8 -mypy==1.9.0 +mypy==1.11.2 # via -r requirements/dev.in mypy-extensions==1.0.0 # via # black # mypy -nodeenv==1.8.0 +nodeenv==1.9.1 # via pre-commit -packaging==24.0 +packaging==24.1 # via # -c requirements/3.9/app.txt # -c requirements/3.9/test.txt + # black # setuptools-scm pathspec==0.12.1 # via black -platformdirs==4.2.0 +platformdirs==4.3.6 # via # black # virtualenv -pre-commit==3.7.0 +pre-commit==3.8.0 # via -r requirements/dev.in -pycodestyle==2.8.0 +pycodestyle==2.12.1 # via flake8 -pyflakes==2.4.0 +pyflakes==3.2.0 # via flake8 -pyinstrument==4.6.2 +pyinstrument==4.7.3 # via -r requirements/dev.in pytest-runner==6.0.1 # via -r requirements/dev.in -pyyaml==6.0.1 +pyyaml==6.0.2 # via # -c requirements/3.9/app.txt # pre-commit -setuptools-scm==8.0.4 +setuptools-scm==8.1.0 # via -r requirements/dev.in tomli==2.0.1 # via @@ -68,15 +69,16 @@ tomli==2.0.1 # black # mypy # setuptools-scm -typing-extensions==4.11.0 +typing-extensions==4.12.2 # via # -c requirements/3.9/app.txt + # -c requirements/3.9/test.txt # black # mypy # setuptools-scm -virtualenv==20.25.1 +virtualenv==20.26.5 # via pre-commit -watchdog==4.0.0 +watchdog==5.0.2 # via -r requirements/dev.in # The following packages are considered to be unsafe in a requirements file: diff --git a/requirements/3.9/docs.txt b/requirements/3.9/docs.txt index aad2522c2..6f1f3e99e 100644 --- a/requirements/3.9/docs.txt +++ b/requirements/3.9/docs.txt @@ -6,11 +6,11 @@ # alabaster==0.7.16 # via sphinx -babel==2.14.0 +babel==2.16.0 # via # -c requirements/3.9/app.txt # sphinx -certifi==2024.2.2 +certifi==2024.8.30 # via # -c requirements/3.9/app.txt # requests @@ -23,17 +23,17 @@ docutils==0.20.1 # sphinx # sphinx-rtd-theme # sphinx-tabs -idna==3.6 +idna==3.10 # via # -c requirements/3.9/app.txt # requests imagesize==1.4.1 # via sphinx -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -c requirements/3.9/app.txt # sphinx -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.9/app.txt # sphinx @@ -41,15 +41,15 @@ markupsafe==2.1.5 # via # -c requirements/3.9/app.txt # jinja2 -packaging==24.0 +packaging==24.1 # via # -c requirements/3.9/app.txt # sphinx -pygments==2.17.2 +pygments==2.18.0 # via # sphinx # sphinx-tabs -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.9/app.txt # sphinx @@ -59,7 +59,7 @@ six==1.16.0 # sphinxcontrib-httpdomain snowballstemmer==2.2.0 # via sphinx -sphinx==7.2.6 +sphinx==7.3.7 # via # -r requirements/docs.in # sphinx-copybutton @@ -76,11 +76,11 @@ sphinx-rtd-theme==2.0.0 # via -r requirements/docs.in sphinx-tabs==3.4.5 # via -r requirements/docs.in -sphinxcontrib-applehelp==1.0.8 +sphinxcontrib-applehelp==2.0.0 # via sphinx -sphinxcontrib-devhelp==1.0.6 +sphinxcontrib-devhelp==2.0.0 # via sphinx -sphinxcontrib-htmlhelp==2.0.5 +sphinxcontrib-htmlhelp==2.1.0 # via sphinx sphinxcontrib-httpdomain==1.8.1 # via -r requirements/docs.in @@ -88,15 +88,15 @@ sphinxcontrib-jquery==4.1 # via sphinx-rtd-theme sphinxcontrib-jsmath==1.0.1 # via sphinx -sphinxcontrib-qthelp==1.0.7 +sphinxcontrib-qthelp==2.0.0 # via sphinx -sphinxcontrib-serializinghtml==1.1.10 +sphinxcontrib-serializinghtml==2.0.0 # via sphinx -urllib3==2.2.1 +urllib3==2.2.3 # via # -c requirements/3.9/app.txt # requests -zipp==3.18.1 +zipp==3.20.2 # via # -c requirements/3.9/app.txt # importlib-metadata diff --git a/requirements/3.9/test.txt b/requirements/3.9/test.txt index eb9c07a0a..94b351224 100644 --- a/requirements/3.9/test.txt +++ b/requirements/3.9/test.txt @@ -8,7 +8,11 @@ async-timeout==4.0.3 # via # -c requirements/3.9/app.txt # redis -certifi==2024.2.2 +blinker==1.8.2 + # via + # -c requirements/3.9/app.txt + # flask +certifi==2024.8.30 # via # -c requirements/3.9/app.txt # requests @@ -20,51 +24,55 @@ click==8.1.7 # via # -c requirements/3.9/app.txt # flask -coverage[toml]==7.4.4 +coverage[toml]==7.6.1 # via pytest-cov -exceptiongroup==1.2.0 +exceptiongroup==1.2.2 # via pytest -fakeredis==2.21.3 +fakeredis==2.24.1 # via -r requirements/test.in -flask==2.2.5 +flask==3.0.3 # via # -c requirements/3.9/app.txt # pytest-flask -highspy==1.5.3 +highspy==1.7.2 # via -r requirements/test.in -idna==3.6 +idna==3.10 # via # -c requirements/3.9/app.txt # requests -importlib-metadata==7.1.0 +importlib-metadata==8.5.0 # via # -c requirements/3.9/app.txt # flask iniconfig==2.0.0 # via pytest -itsdangerous==2.1.2 +itsdangerous==2.2.0 # via # -c requirements/3.9/app.txt # flask -jinja2==3.1.3 +jinja2==3.1.4 # via # -c requirements/3.9/app.txt # flask -lupa==2.1 +lupa==2.2 # via -r requirements/test.in markupsafe==2.1.5 # via # -c requirements/3.9/app.txt # jinja2 # werkzeug -packaging==24.0 +numpy==1.26.4 + # via + # -c requirements/3.9/app.txt + # highspy +packaging==24.1 # via # -c requirements/3.9/app.txt # pytest # pytest-sugar -pluggy==1.4.0 +pluggy==1.5.0 # via pytest -pytest==8.1.1 +pytest==8.3.3 # via # -r requirements/test.in # pytest-cov @@ -79,11 +87,11 @@ pytest-mock==3.14.0 # via -r requirements/test.in pytest-sugar==1.0.0 # via -r requirements/test.in -redis==4.6.0 +redis==5.0.8 # via # -c requirements/3.9/app.txt # fakeredis -requests==2.31.0 +requests==2.32.3 # via # -c requirements/3.9/app.txt # -r requirements/test.in @@ -98,16 +106,20 @@ tomli==2.0.1 # via # coverage # pytest -urllib3==2.2.1 +typing-extensions==4.12.2 + # via + # -c requirements/3.9/app.txt + # fakeredis +urllib3==2.2.3 # via # -c requirements/3.9/app.txt # requests -werkzeug==2.2.3 +werkzeug==3.0.4 # via # -c requirements/3.9/app.txt # flask # pytest-flask -zipp==3.18.1 +zipp==3.20.2 # via # -c requirements/3.9/app.txt # importlib-metadata diff --git a/requirements/app.in b/requirements/app.in index 75798d39e..0a11ef773 100644 --- a/requirements/app.in +++ b/requirements/app.in @@ -1,8 +1,5 @@ -# see ui/utils/plotting_utils: separate_legend() and create_hover_tool() pyyaml altair -colour -pscript pandas pint>=0.19.1 py-moneyed @@ -15,27 +12,28 @@ inflect<=6.0.2 pydantic<2 humanize psycopg2-binary -# due to https://github.com/Flask-Middleware/flask-security/issues/907 +# due to https://github.com/pallets-eco/flask-security/issues/907 (should be re-opened as of Sep 24) -> https://foss.heptapod.net/python-libs/passlib/-/issues/187 bcrypt<=4.0.1 +# hashing backend for passlib, not sure why we need it now +argon2_cffi pytz -# limit the numpy version to make it compatible with dependencies in timely-beliefs >=1.18 (libraries sktime, numba). -numpy<1.25 isodate click click-default-group email_validator rq -rq-dashboard +# sort order +rq-dashboard>=0.8.0.2 # the following uses environment markers (see PEP 496) rq-win; os_name == 'nt' or os_name == 'win' -# This limit resolves a conflict with test.in. The culprit is fakeredis (check their pyproject.toml) -redis >4.5, <5 +# Note: we sometimes limit redis due to our use of fakeredis (check their pyproject.toml) +redis>4.5 tldextract pyomo>=5.6 tabulate timetomodel>=0.7.3 -# significantly faster db queries -timely-beliefs[forecast]>=2.2 +# 3.0.2: significantly faster db queries, incl. bulk saving, 3.1: numpy/pandas upgrade +timely-beliefs[forecast]>=3.1 python-dotenv # a backport, not needed in Python3.8 importlib_metadata @@ -58,13 +56,12 @@ marshmallow>=3 marshmallow-polyfield marshmallow-sqlalchemy>=0.23.1 webargs -# Minimum version that correctly aligns time series that include NaN values -uniplot>=0.7.0 +# Minimum version that supports datetimes on the (x-)axis. +uniplot>=0.12.1 # >=2.5.2: https://github.com/marshmallow-code/flask-marshmallow/issues/262 Flask-SQLAlchemy>=2.5.2 # flask should be after all the flask plugins, because setup might find they ARE flask -# <3.0 https://github.com/FlexMeasures/flexmeasures/issues/936 -flask>=1.0, <3.0 -werkzeug<=2.2.3 +flask>=1.0 +werkzeug vl-convert-python -Pillow>=10.0.1 # https://github.com/FlexMeasures/flexmeasures/security/dependabot/91 \ No newline at end of file +Pillow>=10.0.1 # https://github.com/FlexMeasures/flexmeasures/security/dependabot/91 diff --git a/requirements/dev.in b/requirements/dev.in index 5fddad151..bf58d1171 100644 --- a/requirements/dev.in +++ b/requirements/dev.in @@ -1,7 +1,7 @@ pre-commit # we're pinning the following two to what .pre-commit-config.yaml says -black==22.3.0 -flake8==4.0.1 +black==24.8.0 +flake8==7.1.1 flake8-blind-except mypy pytest-runner diff --git a/requirements/docs.in b/requirements/docs.in index 230fcc053..fb9363171 100644 --- a/requirements/docs.in +++ b/requirements/docs.in @@ -1,4 +1,4 @@ -sphinx +sphinx!=7.4.* sphinx-rtd-theme >= 1.2 sphinxcontrib.httpdomain sphinx_fontawesome diff --git a/setup.py b/setup.py index 7aed8cbd6..1a7a81956 100644 --- a/setup.py +++ b/setup.py @@ -54,6 +54,7 @@ def load_requirements(use_case): "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", "Framework :: Flask", "Development Status :: 5 - Production/Stable", "License :: OSI Approved :: Apache Software License",