Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: isolate ontology submission process steps #132

Merged

Conversation

syphax-bouazzouni
Copy link

@syphax-bouazzouni syphax-bouazzouni commented Feb 22, 2024

Changes

  • Add an abstraction for submission process steps(163f06a)
  • Extract submission generate_rdf step to a file(d773760)
  • Extract submission generate missing labels steps into a file(a0f5695)
  • Extract the submission archiving step into a file(8c6200b)
  • Add abstraction to diff tool & extract the submission step to a file(a2eb2c0)
  • Extract the submission metrics generation step to a file(70f314e)
  • Extract the submission properties indexation step into a file(ddeee2c)
  • Extract the submission terms indexation step into a file(320ae91)
  • Move the extract metadata concern to the submission process step file(fb88b6e)
  • Extract the submission generate obsolete classes step from generate rdf(f30ba29)
  • Add the global submission process that calls the sub-steps (91d5642)

@jonquet
Copy link

jonquet commented Feb 22, 2024

T'avais pas deja une issue ouverte sur ce sujet ?

@syphax-bouazzouni
Copy link
Author

T'avais pas deja une issue ouverte sur ce sujet ?

Oui y a 2 ans ncbo#143, mais jamais merger, je la remet a jour

Copy link

codecov bot commented Feb 22, 2024

Codecov Report

Attention: Patch coverage is 82.19955% with 157 lines in your changes are missing coverage. Please review.

Project coverage is 81.62%. Comparing base (026c9c4) to head (91d5642).

Files Patch % Lines
...ubmission_process/operations/submission_indexer.rb 60.97% 48 Missing ⚠️
...on_process/operations/submission_missing_labels.rb 84.61% 24 Missing ⚠️
..._process/operations/submission_extract_metadata.rb 81.45% 23 Missing ⚠️
...ocess/operations/submission_mertrics_calculator.rb 79.45% 15 Missing ⚠️
...ion_process/operations/submission_rdf_generator.rb 75.43% 14 Missing ⚠️
...rocess/operations/submission_properties_indexer.rb 82.60% 8 Missing ⚠️
..._process/operations/submission_obsolete_classes.rb 84.44% 7 Missing ⚠️
...on_process/operations/submission_diff_generator.rb 90.90% 4 Missing ⚠️
lib/ontologies_linked_data/diff/bubastis_diff.rb 81.25% 3 Missing ⚠️
lib/ontologies_linked_data/diff/diff.rb 57.14% 3 Missing ⚠️
... and 4 more
Additional details and impacted files
@@               Coverage Diff               @@
##           development     #132      +/-   ##
===============================================
+ Coverage        81.17%   81.62%   +0.45%     
===============================================
  Files               89      100      +11     
  Lines             6512     6639     +127     
===============================================
+ Hits              5286     5419     +133     
+ Misses            1226     1220       -6     
Flag Coverage Δ
unittests 81.62% <82.19%> (+0.45%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@syphax-bouazzouni syphax-bouazzouni force-pushed the feature/isolate-ontology-submission-process-steps branch 4 times, most recently from 23ca9e0 to 5cc90c4 Compare April 18, 2024 22:35
@syphax-bouazzouni syphax-bouazzouni force-pushed the feature/isolate-ontology-submission-process-steps branch 3 times, most recently from fdc43b5 to b9aac2a Compare April 20, 2024 18:50
@syphax-bouazzouni syphax-bouazzouni force-pushed the feature/isolate-ontology-submission-process-steps branch from b9aac2a to 91d5642 Compare April 20, 2024 19:19
@syphax-bouazzouni syphax-bouazzouni self-assigned this Apr 20, 2024
@syphax-bouazzouni syphax-bouazzouni merged commit d5ff793 into development Apr 20, 2024
26 checks passed
syphax-bouazzouni added a commit that referenced this pull request May 22, 2024
…ndexing ontologies data and metadata, URI data fetching (#135)

* Feature: Optimize tests run time by 50% (#107)

* update bubastis.jar v1.4.0 and fix missing import exception

* optimize mappings tests

* optimize provisional relation tests

* optimize notes tests

* fix mappings tests

* optimize instances tests

* add generate_missing_labels and extract_metadata process options

* don't index and extract by default in submission process in tests

* optimize ontology submission tests run time

* Feature: Add Virtuso, Allegrograph and Graphdb integration  to OLD  (#106)

* setup multi-store unit-tests environment

* fix unit tests

* add vo parsing optimization

* update RDF version replaced RDF::SKOS with RDF::Vocab::SKOS (#131)

* Fix: an  issue after update RDF gem to 3.0 that frozen request params (#133)

* fix an issue after update RDF gem to 3.0 that frozen request params

* handlee the case when the sparql endpoint default value is empty

* Feature: Migrate SOLR configuration files to use SOLR Schema API (#126)

* use standard SOLR in docker compose with no ontoportal old confgis

* migrate ontology properties SOLR configuration to use Schema API

* migrate ontology classes SOLR configuration to use Schema API

* migrate provisional classes indexation to use Schema API and model hooks

* update tests to handle the new indexation API

* simplify the ontology properties index schema

* update class and properties schema to use the existent dynamic names

* Feature: Index Ontologies metadata and content & Agents  (#130)

* use standard SOLR in docker compose with no ontoportal old confgis

* migrate ontology properties SOLR configuration to use Schema API

* migrate ontology classes SOLR configuration to use Schema API

* migrate provisional classes indexation to use Schema API and model hooks

* update tests to handle the new indexation API

* simplify the ontology properties index schema

* update class and properties schema to use the existent dynamic names

* index submission and ontologies metadata on save

* index agents metadata

* add ontology and agent metadata  indexation tests

* make agent, name , acronym, email and identifiers searchable

* unindex ontology submission when archived

* make ontology acronym and name searchable

* update embedded ontology to all the fields and update submission in save

* fix embed docs search tests

* rename ontology unindex to unindex_all_data to prevent conflicts

* implement index all ontology content

* fix unescaping indexed properties naming

* fix an issue after update RDF gem to 3.0 that frozen request params

* add parallel processing the index_all_data step

* clear indexed data after ontology delete

* optimize index all data in Virtuoso and GraphDb by pre-fetching all ids

- Before optimization
    - fs ⇒ 15.224490000051446s
    - ag ⇒ 19.238805999979377s
    - vo ⇒ 42.95274499990046s
    - gb ⇒ 33.52821200003382s
- After optimization
    - fs ⇒ 15.369778999942355s
    - ag ⇒ 17.367580000078306s
    - vo ⇒ 16.564614000031725s
    - gb ⇒ 15.431716999970376s

* Feature: Add URI fetching related triples and serialization in different formats  (#125)

* Add raptor library to parse ntriples data

* Add resource model to fetch id related triples and serialize it

* Add and inhance xml, ntriples, turtle and json serializers

* Updating rdf version in goo project

* updating resource model

* Adding tests for resource model and serializers

* update the resource test to have a more complete data to test (array, bnodes, typed values)

* re-implement xml serializer using RDF/XML parser instead of Raptor

* implement array handelling of resource to_object

* Enhance and refactor serializers ntriples, turtle and xml

* Enhance and refactor serializers ntriples, turtle and xml

* Handle blank nodes and reverse triples
- handle blank nodes
- fetch reverse triples
- generate random name for models in to_object, because when two model created the same time one overrides the other
- call the new serializer JSONLD and RDF_XML

* Impliment new serializers jsonld and rdf_xml

- impliment jsonld serializer that uses json-ld library
- revert changes in xml.rb file to the original implimentation, and put the new implimentation in rdf_xml.rb file
- Add the media types :jsonld and :rdf_xml

* Add json-ld gem

* Enhance the test resource

- Add some cases to the data tests
- refactor the test of the serializers formats

* Fix test for fetch-related triples and json

* clean and refactor the resource serializer code

* Removed unused methods
* Extracted duplicated code in methods
* Removed skip from the tests

---------

Co-authored-by: Syphax bouazzouni <[email protected]>

* Feature: Add submission metrics to the indexed data

* Feature: isolate ontology submission process steps (#132)

* add an abstraction for submission process steps

* extract submission generate_rdf step to a file

* extract submission generate missing labels steps into a file

* extract the submission archiving step into a file

* add  abstraction to diff tool & extract the submission step to a file

* extract the submission metrics generation step to a file

* extract the submission properties indexation step into a file

* extract the submission terms indexation step into a file

* move the extract metadata concern to submission process step file

* extract the submission generate obsolete classes step from generate rdf

* add the global submission process that call the sub-steps

* Feature: Add index all data step (#136)

* move the submission_all_data concern to a submission process service

* add index_all step to submission parsing steps

* add index all data submission status

* send note creation notification to also the admins (#137)

* change sparql client branch to use development

* fix indexing  all data been removed after the index terms step

---------

Co-authored-by: Imad Bourouche <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants