Basic Functionality #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* link DMPs with Records via PIDs instead of directly, as this avoids a problem with the record versioning table that required an ugly hack as workaround before (and it is done in a similar fashion by Invenio-Communities)
* introduce classes for clear mappings from RDA-CS to Invenio: * an RDA-CS DMP corresponds to a DataManagementPlan in Invenio * an RDA-CS Dataset corresponds to a Dataset in Invenio * an RDA-CS Distribution corresponds to a Record in Invenio * one DMP can have multiple Datasets, and one Dataset can be used by several projects (and thus, DMPs) * one Dataset may have one Distribution (which represents a concrete deposit of the Dataset) on a host; thus, we map Distributions to Records in Invenio * to avoid issues with versioning tables and increase flexibility (e.g. enable linking to Drafts instead of Records), we link to PIDs instead of Records themselves
* invenio-files-rest has the req. 'SQLAlchemy-Utils>=0.33.1,<0.36' * we copy this requirement to avoid possible ContextualVersionConflicts
* if Python 3.6 or older is used, add a dependency for backports-fromisoformat
* separate the logic for parsing general information from maDMPs and the logic for creating concrete records/drafts from the referenced datasets * this split enables easier customization of the logic for parsing datasets into records * add RecordConverter classes to handle the dataset-to-record conversion * add RDMRecordConverter as default implementation, which translates DMP datasets into RDM-Records (resp. BibliographicDrafts)
* make the Dataset.record_pid nullable to allow for storing information about Datsets that do not (yet) have a distribution in Invenio * add management methods for DMPs and Datasets * add emission of signals when DMPs and Datasets are updated via their management methods
* add management methods for RecordConverters * add soft and hard sync versions for Datasets and their Records * disable autoflush in convert_dmp, as autoflush previously caused issues with the non-nullable Dataset.record_pids, and it is not needed
* update mappings: incorporate changes in the metadata model * update the test records * use the new API for accessing a record's PID * update module dependencies
mb-wali
reviewed
Oct 20, 2020
mb-wali
reviewed
Oct 20, 2020
mb-wali
reviewed
Oct 20, 2020
There was a problem hiding this comment.
That's a very good start for a basic integration.
I left some comments, but overall nice work. I liked the code style black badge. we will use it on other modules.
I haven't gone to the func in detail. since at this stage the module is using most of hard-coded values.
also, I noticed some default cookie-cutter setup that this module would not need.
- remove files/entrypoints for translations
- remove todos for example here
maybe we can open a separate issue for it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Conversion Between maDMPs and Records
Implemented logic for parsing maDMPs according to the RDA Common Standard, and using the parsed information to create record drafts in Invenio.
Custom subclasses of the BaseRecordConverter can be used to implement conversion into/from custom metadata models.
The default RDMRecordConverter can be used to convert between DMPs and records using the Invenio-RDM-Records metadata model.
Model Classes
Added database-persistent objects for storing information regarding which records are linked to which DMPs.
Among other things, this helps in minimizing the amount of unnecessarily sent network requests by checking if the record in question is linked to any DMPs at all.
REST API
Implemented simple REST API endpoints for allowing maDMP tools to communicate with Invenio.
Optional token-based auth. (via the "Authorization: Bearer" HTTP header, containing a shared secret) can be used to verify that the request was not forged.
CLI Commands
Created simple CLI commands to import DMPs from maDMP JSON files.
Tests
Added a few unit tests and some test data (example maDMPs from the RDA GitHub repository, and randomly generated example records [using the current RDM-Records metadata model]).