Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/metax integration #356

Merged
merged 79 commits into from
Apr 7, 2022
Merged

Feature/metax integration #356

merged 79 commits into from
Apr 7, 2022

Conversation

genie9
Copy link
Contributor

@genie9 genie9 commented Feb 23, 2022 •
edited
Loading

Description

Implements integration with Metax service:

  • metadata objects Study and Dataset are added to Metax catalog on object POST as Metax draft dataset
  • metadata objects Study and Dataset are updated to Metax catalog on object PATCH and PUT
  • metadata objects Study and Dataset are deleted from Metax catalog on object DELETE
  • metadata objects Study and Dataset are published to Metax catalog on publishing object containing folder

Adds container for Metax Mock API for testing purposes.

Related issues

closes #329

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update (this needs a follow up PR)

Changes Made

  • adds new local container for testing against mocked Metax API
  • introduces new env vars: METAX_USER, METAX_PASS, METAX_URL
  • adds new key metaxIdentifier to Study and Dataset collections containing metax id returned from Metax API
  • adds new handler MetaxServiceHandler to take care of mapping Submitter metadata to Metax metadata and to connect to Metax API
  • updates tests and adds integration tests

Testing

  • Unit Tests
  • Integration Tests

Mentions

This branch is rebased on #354
This branch is ready when it is adapted changes from #332

@genie9 genie9 force-pushed the feature/metax-integration branch from eeaa231 to 327d4f0 Compare February 23, 2022 15:26
@genie9 genie9 marked this pull request as draft February 23, 2022 16:11
@genie9 genie9 force-pushed the feature/metax-integration branch 4 times, most recently from 84dc388 to 002ee9b Compare February 28, 2022 12:28
@genie9 genie9 marked this pull request as ready for review February 28, 2022 12:37
@blankdots blankdots linked an issue Feb 28, 2022 that may be closed by this pull request
Create functionality for adding and updating Submitter metadata to Metax catalog #329
Closed
blankdots
blankdots reviewed Feb 28, 2022
View reviewed changes
tests/integration/run_tests.py Show resolved Hide resolved
tests/integration/run_tests.py Show resolved Hide resolved
metadata_backend/api/handlers/object.py Outdated Show resolved Hide resolved
blankdots
blankdots reviewed Feb 28, 2022
View reviewed changes
Copy link
Contributor

@blankdots blankdots left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the need for "metaxIdentifier": {"identifier": 12345} on the object, didn't we agree in #326 this is going to be on folder level where we will have studyIdentifier and datasetIdentifiers ?

blankdots
blankdots reviewed Feb 28, 2022
View reviewed changes
tests/integration/mock_metax_api.py Outdated Show resolved Hide resolved
tests/integration/mock_metax_api.py Outdated Show resolved Hide resolved
@blankdots
Copy link
Contributor

blankdots commented Feb 28, 2022 •
edited
Loading

adds new key metaxIdentifier to Study and Dataset collections with dict {identifier: <metax_id>, status: "draft|published"}

  • the status of published should be only on the folder level to minimize the information, as we cannot have published studies and draft datasets, either all study and datasets are published or none

metax_api_handler.py should be in helpers module as that is where we have DOI, we can think if we need another python module maybe to express integrations such as DOI, Metax, REMS?

how is the metax URL constructed, is it based on the metax_id and the fairdata url e.g. https://etsin.fairdata.fi/dataset/{metax_id} ? - this is required for publishing DOIs

@genie9 genie9 force-pushed the feature/metax-integration branch from 28b70f0 to 2906234 Compare March 1, 2022 10:59
@blankdots
Copy link
Contributor

We agreed to keep this as is and in #332 do the following:

  • add doi and metax required information on study and dataset objects
  • once the folder gets published we update this information on the folder to keep it more consistent and provide means for front-end to query and display this information to users.

@genie9
Copy link
Contributor Author

genie9 commented Mar 1, 2022

how is the metax URL constructed, is it based on the metax_id and the fairdata url e.g. https://etsin.fairdata.fi/dataset/{metax_id} ? - this is required for publishing DOIs

yes it is https://etsin.fairdata.fi/datasets/{metax_id} or with metax service https://metax.fairdata.fi/rest/v2/datasets/{metax_id}

@genie9 genie9 marked this pull request as draft March 4, 2022 14:22
@genie9 genie9 force-pushed the feature/metax-integration branch from c6e790d to 1eaceaa Compare March 4, 2022 14:33
genie9
genie9 commented Mar 8, 2022
View reviewed changes
tests/test_handlers.py Outdated Show resolved Hide resolved
genie9
genie9 commented Mar 8, 2022
View reviewed changes
tests/test_handlers.py Outdated Show resolved Hide resolved
@genie9 genie9 force-pushed the feature/metax-integration branch 4 times, most recently from 6c6b696 to f0a0c94 Compare March 9, 2022 06:04
blankdots
blankdots approved these changes Apr 4, 2022
View reviewed changes
Copy link
Contributor

@blankdots blankdots left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other than the small adjustments to folder.json and requiring familyName and givenName from contributors as well, LGTM

@genie9 genie9 force-pushed the feature/metax-integration branch from cc92c09 to 9498188 Compare April 4, 2022 12:54
@genie9
Update schemas with required fields
edb7992 
Add givenName and familyName as required fields to datacite and folder schemas
for creators and contributor fields items.
@genie9 genie9 force-pushed the feature/metax-integration branch from 9498188 to edb7992 Compare April 4, 2022 13:07
@genie9 genie9 requested a review from blankdots April 4, 2022 13:13
@genie9
Add DISCOVERY_URL to env vars
341482c 
Updated global variable DISCOVERY_URL to be environment variable.
For testing purposes .env.example file is set to point to demo env. Default
variable is set to production.
@genie9 genie9 force-pushed the feature/metax-integration branch from 0431ac8 to 341482c Compare April 5, 2022 10:01
@blankdots blankdots mentioned this pull request Apr 6, 2022
Merged
2 tasks
blankdots
blankdots suggested changes Apr 6, 2022
View reviewed changes
Copy link
Contributor

@blankdots blankdots left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#392 got lost in rebase here, this is required

@blankdots
Copy link
Contributor

tested with CSCfi/metadata-submitter-frontend#715 seems to work fine and the data is registered correctly

genie9 added 2 commits April 6, 2022 09:19
@genie9
Fix logging print format
92bf5f4
@genie9
Merge pull request #387 from CSCfi/feature/map-datacite-metadata-to-m…
4e5b528 
…etax-metadata

Feature/map datacite metadata to metax metadata
@genie9 genie9 merged commit 1692c33 into develop Apr 7, 2022
@genie9 genie9 deleted the feature/metax-integration branch April 7, 2022 06:36
@blankdots blankdots mentioned this pull request Apr 7, 2022
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@teemukataja teemukataja teemukataja approved these changes

@blankdots blankdots blankdots approved these changes

Assignees

@genie9 genie9

Labels
Metax integration
Projects
None yet
Milestone
0.13.0
Development

Successfully merging this pull request may close these issues.

Create functionality for adding and updating Submitter metadata to Metax catalog
3 participants
@genie9 @blankdots @teemukataja