diff --git a/CHANGES.rst b/CHANGES.rst index 62976b9a9..c01cb48ac 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -7,7 +7,10 @@ Changes `Unreleased `_ (latest) ------------------------------------------------------------------------------------ -* Nothing yet. +Features / Changes +~~~~~~~~~~~~~~~~~~~~~ +* Add ``BROWSE`` permission for ``ServiceTHREDDS`` to parse request against *metadata* or *data* contents according to + specified configuration of the specific service (resolves `#361 `_). `3.0.0 `_ (2020-10-19) ------------------------------------------------------------------------------------ @@ -513,7 +516,7 @@ Bug Fixes Features / Changes ~~~~~~~~~~~~~~~~~~~~~ -* Prioritize settings (ie: `magpie.ini` values) before environment variables and ``magpie.constants`` globals. +* Prioritize settings (ie: ``magpie.ini`` values) before environment variables and ``magpie.constants`` globals. * Allow specifying ``magpie.scheme`` setting to generate the ``magpie.url`` with it if the later was omitted. * Look in settings for required parameters for function ``get_admin_cookies``. * Use API definitions instead of literal strings for routes employed in ``MagpieAdapter``. diff --git a/docs/configuration.rst b/docs/configuration.rst index 20b75ec3f..74b693a83 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -5,14 +5,22 @@ Configuration ============= At startup, `Magpie` application will load multiple configuration files to define various behaviours or setup -operations. These are defined though the configuration settings presented below. +operations. These are defined though the configuration settings presented in below sections. -All generic `Magpie` configuration settings can be defined through either the `magpie.ini`_ file -or environment variables. Values defined in `magpie.ini`_ are expected to follow the -``magpie.[variable_name]`` format, and corresponding ``MAGPIE_[VARIABLE_NAME]`` format is used for environment -variables. Both of these alternatives match the constants defined in `constants.py`_ and can be used -interchangeably. Order of resolution will prioritize setting values over environment variables in case of duplicate -configurations resulting into different values. +All generic `Magpie` configuration settings can be defined through either the `magpie.ini`_ file or environment +variables. Values defined in `magpie.ini`_ are expected to follow the ``magpie.[variable_name]`` format, and +corresponding ``MAGPIE_[VARIABLE_NAME]`` format is used for environment variables. Both of these alternatives match +the constants defined in `constants.py`_ and can be used interchangeably. + +.. versionchanged:: 1.1 + Order of resolution will prioritize *setting configurations* over *environment variables* in case of duplicates + resulting into different values. Environment variables will not override already specified setting values. + + Previous versions of `Magpie` would instead prioritize environment variables, but this behaviour was deemed as + counter intuitive. This is attributed to the global scope nature of environment variables that often made it hard + to understand why some custom INI file would not behave as intended since those variable would inconsistently take + precedence whether or not they were defined. Using a common configuration file makes it easier to maintain and + understand the applied settings, and is therefore preferable. .. _constants.py: https://github.com/Ouranosinc/Magpie/tree/master/magpie/constants.py @@ -35,15 +43,14 @@ By default, `Magpie` will try to load a ``magpie.env`` file which can define fur used to setup the application (see ``MAGPIE_ENV_FILE`` setting further below). An example of expected format and common variables for this file is presented in `magpie.env.example`_. -**Important Notes:** - -If ``magpie.env`` cannot be found (using setting ``MAGPIE_ENV_FILE``) but ``magpie.env.example`` is available -(after resolving any previously set ``MAGPIE_ENV_DIR`` variable), this example file will be used to make a copy -saved as ``magpie.env`` and will be used as the base ``.env`` file to load its contained environment variables. -This behaviour is intended to reduce initial configuration and preparation of `Magpie` for a new user. +.. warning:: + If ``magpie.env`` cannot be found (e.g.: using setting ``MAGPIE_ENV_FILE``) but `magpie.env.example`_ is available + after resolving any previously set ``MAGPIE_ENV_DIR`` variable, this example file will be used to make a copy saved + as ``magpie.env`` and will be used as the base ``.env`` file to load its contained environment variables. + This behaviour is intended to reduce initial configuration and preparation of `Magpie` for a new user. -When loading variables from the ``.env`` file, any conflicting environment variable will **NOT** be overridden. -Therefore, only *missing but required* values will be added to the environment to ensure proper setup of `Magpie`. + When loading variables from the ``.env`` file, any conflicting environment variable will **NOT** be overridden. + Therefore, only *missing but required* values will be added to the environment to ensure proper setup of `Magpie`. .. _magpie.env.example: https://github.com/Ouranosinc/Magpie/tree/master/env/magpie.env.example @@ -67,6 +74,10 @@ to the desired URL. See ``MAGPIE_PROVIDERS_CONFIG_PATH`` setting below to setup configuration. Please refer to the comment header of sample file `providers.cfg`_ for specific format and parameter details. +.. versionchanged:: 3.1 + Some services, such as :ref:`ServiceTHREDDS` for instance, can take additional parameters to customize some of + their behaviour. Please refer to :ref:`Services` chapter for specific configuration supported. + File: permissions.cfg ~~~~~~~~~~~~~~~~~~~~~~ @@ -84,9 +95,11 @@ behaviour of each parameter according to encountered use cases. Combined Configuration File ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. versionadded:: 2.0 + Since contents of all different configurations files (`providers.cfg`_, `permissions.cfg`_) reside under distinct -top-level objects (of same name), it is actually possible to use an unique file to define everything. For example, -one could define a combined configuration as follows. +top-level objects, it is actually possible to use an unique file to define everything. For example, one could define +a combined configuration as follows. .. code-block:: YAML @@ -141,7 +154,7 @@ Settings and Constants Environment variables can be used to define all following configurations (unless mentioned otherwise with ``[constant]`` keyword next to the parameter name). Most values are parsed as plain strings, unless they refer to an -activable setting (e.g.: ``True`` or ``False``), or when specified with more specific ``[]`` notation. +activatable setting (e.g.: ``True`` or ``False``), or when specified with more specific ``[]`` notation. Configuration variables will be used by `Magpie` on startup unless prior definition is found within `magpie.ini`_. All variables (i.e.: non-``[constant]`` parameters) can also be specified by their ``magpie.[variable_name]`` setting diff --git a/docs/references.rst b/docs/references.rst index 485e27150..c67373809 100644 --- a/docs/references.rst +++ b/docs/references.rst @@ -13,6 +13,7 @@ .. _Pyramid: https://docs.pylonsproject.org/projects/pyramid/ .. _ReadTheDocs: https://pavics-magpie.readthedocs.io/ .. _SQLAlchemy: https://www.sqlalchemy.org/ +.. _THREDDS: https://www.unidata.ucar.edu/software/tds/ .. _Twitcher: https://github.com/bird-house/twitcher .. _Waitress: https://github.com/Pylons/waitress .. _Ziggurat-Foundations: https://github.com/ergo/ziggurat_foundations diff --git a/docs/services.rst b/docs/services.rst index 41b342bf0..4123098d6 100644 --- a/docs/services.rst +++ b/docs/services.rst @@ -49,7 +49,99 @@ ServiceAPI ServiceTHREDDS ~~~~~~~~~~~~~~~~~~~~~ -.. todo: +The implementation of this service is handled by class :class:`magpie.services.ServiceTHREDDS`. It refers to a remote +data server named `Thematic Real-time Environmental Distributed Data Services` (`THREDDS`_). The service employs two (2) +types of :term:`Resources`, namely :class:`magpie.models.Directory` and :class:`magpie.models.File`. All the directory +resources can be nested any number of times, and files can only reside as leaves of the hierarchy, similarly to a +traditional file system. The :term:`Allowed Permissions` on both the :term:`Service` itself or any of its children +:term:`Resource` are :attr:`Permission.BROWSE`, :attr:`Permission.READ`, and :attr:`Permission.WRITE` (see note below +regarding this last permission). + +.. versionadded:: 3.1 + The :attr:`Permission.BROWSE` permission is used to provide listing access of contents when targeting a + :term:`Resource` of type :class:`magpie.models.Directory`. When targeting a :class:`magpie.models.File`, it instead + provides *metadata* access to that file. + +Permission :attr:`Permission.READ` can be applied to all of the resources, but will only effectively make sense when +attempting access of a specific :term:`Resource of type :class:`magpie.models.File`. + +.. versionchanged:: 3.1 + Permission :attr:`Permission.READ` does not offer *metadata* content listing of :class:`magpie.models.Directory` + anymore. For this, :attr:`Permission.BROWSE` should be used instead. Setting :attr:`Permission.READ` on a + directory will only be logical when combined with :attr:`Scope.RECURSIVE`, in which case `Magpie` will interpret + the :term:`Effective Permissions` to allow read access to all :class:`magpie.models.File` under that directory, at + any depth level, unless denied by a lower-level specification. + +Finally, :attr:`Permission.WRITE` can also be applied on all of the resources, but are not explicitly employed during +parsing of incoming requests. + +.. note:: + The :attr:`Permission.WRITE` is not handled by `ServiceTHREDDS`_ itself during :term:`ACL` resolution as it is not + considered by :meth:`magpie.services.ServiceTHREDDS.permission_requested` that never returns this value. The method + only returns either :attr:`Permission.BROWSE` or :attr:`Permission.READ`. An :term:`User` or :term:`Group` can still + have this :term:`Applied Permission` to allow a third party service to interrogate `Magpie API` about the presence + of :attr:`Permission.WRITE` permission and perform the appropriate action with the result. The + :term:`Effective Permissions` API routes will provide the resolved ``access``. It is only `Twitcher`_ proxy that + will not be able to make use of it during incoming requests as it depends on + :class:`magpie.adapter.magpieowssecurity.MagpieOWSSecurity`, which in turn employs the result from the :term:`ACL`. + + The :attr:`Permission.WRITE` is mostly preserved for backward compatibility of services that employed + `ServiceTHREDDS`_ to obtain information about which directory or files (which registered `Magpie` :term:`Resource`) + are writable or not, although using another upload methodology that is not specifically executed via the actual + remote `THREDDS`_ service. + + +As presented above, the main two permissions are :attr:`Permission.BROWSE` and :attr:`Permission.READ` which +correspondingly serve to retrieve *metadata* and actual *data* of a given :term:`Resource`. To distinguish requests +between these two types of contents, `ServiceTHREDDS`_ employs two parts from the request path, the sub-path *prefix* +and the file *extension*. A default methodology is employed categorize these two types of content, but can be modified +using custom configurations as described in :ref:`Custom THREDDS Settings` section. + + + + +Custom THREDDS Settings +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + See section :ref:`ServiceTHREDDS` about the base implementation components of this service type for further details. + + +With either the `providers.cfg`_ or the `Combined Configuration File`_ presented in previous sections, a service of +type `THREDDS`_ can be slightly customized to meet the intended needs. Specifically, two additional fields +``metadata_type`` and ``data_type + + +As presented above, the main two permissions are :attr:`Permission.BROWSE` and :attr:`Permission.READ` which +correspondingly serve to retrieve *metadata* and actual *data* of a given :term:`Resource`. To distinguish requests +between these two types of contents, `ServiceTHREDDS`_ employs two parts from the request path, the sub-path *prefix* +and the file *extension*. The *default* methodology employed to categorize these two types of content is presented +below. + +.. code-block:: YAML + + providers: + thredds_service: + data_type: + - prefix: fileServer + - prefix: dodsC + - prefix: wcs + - prefix: wms + metadata_type: + # no extra sub-path, case when navigating contents using an UI browser + # (`~` is the representation method of JSON's `null`, make sure to adjust accordingly with file extension) + - prefix: ~ + - prefix: catalog + - prefix: ncml + - prefix: uddc + - prefix: iso + + + +.. note:: + The *default* categorization between *metadata*/*data* contents can be modified using custom configurations + specified within a `providers.cfg`_ or :ref:`Combined Configuration File`. + ServiceGeoserverWMS ~~~~~~~~~~~~~~~~~~~~~ diff --git a/magpie/permissions.py b/magpie/permissions.py index e9d783859..651a41240 100644 --- a/magpie/permissions.py +++ b/magpie/permissions.py @@ -23,6 +23,7 @@ class Permission(ExtendedEnum): READ = "read" WRITE = "write" ACCESS = "access" + BROWSE = "browse" # WPS permissions GET_CAPABILITIES = "getcapabilities" GET_MAP = "getmap" diff --git a/magpie/services.py b/magpie/services.py index 2e5bf4c65..2747ad846 100644 --- a/magpie/services.py +++ b/magpie/services.py @@ -656,10 +656,10 @@ class ServiceTHREDDS(ServiceInterface): service_type = "thredds" permissions = [ + Permission.BROWSE, + # read are for Permission.READ, - # FIXME: - # does WRITE permission even make sense here? - # leave it for bw-compat, but not even considered since 'permission_requested' always returns READ + # NOTE: see special usage of WRITE in docs Permission.WRITE, ]