ProjectFile v2 planning

[lachlangrose]

Lets rethink the project file and make a list of all of the objects that should be stored in the project file, both input, output and temporary objects.

Input

• structure shapefile
• dtm
• fault trace
• geology polygon

Output/Intermediate

• basal contacts
• unit thickness distributions/spatial point estimates
• fault trace
• fault orientation
• fault surface
• stratigraphic/foliation surfaces
• scalar fields
• implicit function
• fault topology
• fault-stratigraphy topology
• stratigraphic column

Requirements

• compatible license
• pythonic interface
• fast read/write
• Backwards compatibility

Possible options are:

• geoh5 python library is LGPL licensed, not necessarily compatible with Loop
• geopackage extending geopackage will potentially make it incompatible with existing readers so may make it difficult to add new data types
• json/xml + binary - use a human readable file to store meta data/instructions to read binary or other file formats for each of the objects we require. This is more similar to how gocad/qgis manage the files
• Existing loop3d netcdf - keep using the existing format but make it fit for purpose
• Custom sqlite3 db - write out own schema for an sqlite database

@AngRodrigues @rabii-chaarani @noellehmcheng @markjessell @ShebMichel @LaurentAilleres

[markjessell]

hi all I guess the first thing to do is decide if the project file is just an input to loopstructural or should be able to handle the outputs from LS as well? Geopkg is good for 2D but less good for 3D, geoh5 can do both. Cheers Mark Mark Jessell Professor/ Western Australian Fellow Centre for Exploration Targeting/ SES • M006, Perth WA 6009 Australia T +61 8 6488 5803 • E ***@***.***

…

________________________________ From: Lachlan Grose ***@***.***> Sent: Thursday, October 17, 2024 5:13:55 AM To: Loop3D/LoopProjectFile ***@***.***> Cc: Mark Jessell ***@***.***>; Mention ***@***.***> Subject: [Loop3D/LoopProjectFile] ProjectFile v2 planning (Discussion #39) Lets rethink the project file and make a list of all of the objects that should be stored in the project file, both input, output and temporary objects. Should we use geoh5, geopackage Input * structure shapefile * dtm * fault trace * geology polygon Output * basal contacts @AngRodrigues<https://github.com/AngRodrigues> @rabii-chaarani<https://github.com/rabii-chaarani> @noellehmcheng<https://github.com/noellehmcheng> @markjessell<https://github.com/markjessell> @ShebMichel<https://github.com/ShebMichel> @LaurentAilleres<https://github.com/LaurentAilleres> — Reply to this email directly, view it on GitHub<#39>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUBSYDSTGLNXOSMV6ZJFG3Z35BRHAVCNFSM6AAAAABQC4NIOSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGMZDSNZYGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ShebMichel]

I agree with Mark, that geoh5 should be the way to go forward. Maybe both can be still generated and in case we have the consensus with other users, then we can switch off <.loop3d>.

Cheers
Michel

[markjessell]

Hi I didn’t say that I said it depends on what we want the format for 2D or 2D plus 3D M Mark Jessell Professor/ Western Australian Fellow Centre for Exploration Targeting/ SES • M006, Perth WA 6009 Australia T +61 8 6488 5803 • E ***@***.***

…

________________________________ From: ShebMichel ***@***.***> Sent: Thursday, October 17, 2024 11:30:19 AM To: Loop3D/LoopProjectFile ***@***.***> Cc: Mark Jessell ***@***.***>; Mention ***@***.***> Subject: Re: [Loop3D/LoopProjectFile] ProjectFile v2 planning (Discussion #39) I agree with Mark, that geoh5 should be the way to go forward. Maybe both can be still generated and in case we have the consensus with other users, then we can switch off <.loop3d>. Cheers Michel — Reply to this email directly, view it on GitHub<#39 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUBSYBPFHYZRIONHK67O3LZ36NUXAVCNFSM6AAAAABQC4NIOSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJXGA2DGMQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lachlangrose]

I'll keep updating the first post with a list of options and summary of the requirements

@ShebMichel, we are not talking about having the ability to export into different formats we are discussing the format that is used by Loop during the different processes, so it needs to be able to store the input data used by map2loop and store the outputs of map2loop + potentially the LoopStructural model results. I would also like to be able to store the state of a LoopStructural model to be loaded in to save reloading the model. We can have the ability to export from the Loop format to other formats.

geoh5 has the disadvantage of relying on mira to maintain and update the format currently it seems locked to python 3.10 and is an LGPL license so that could be problematic. It is easily extended so would be able to add a lot of the objects we want.

@markjessell ideally, I would like to define a structure for saving and loading the inputs and outputs of LoopStructural. But it doesn't have to be a single file, I would be open to using something like qgis/gocad where the "project" links to the data objects. The only issue with this is it opens up problems with missing files etc.

[LaurentAilleres]

Hi everyone, The gocad project file... was actually a directory with an external file serving the role of "project links". Gocad objects etc... were stored in that folder, mainly as binary objects which prevented users from being tempted to edit them... I never had any project corrupted and it comes down to the users not being entirely stupide... if they are, nothing will save them. So i do vote for a 2D type structure for the m2l process (input and output) and the geopackage idea is very attractive and something different for loopstructural although one of its inputs should be the geopkg at the end of m2l obviously. Saving the state of the modelling is another kettle of fish... I understand the vision is to generate 100000+ models - the only way to do this is probably to save them all in different folders if we dont want to handle VERY large single files... if we were to use geoh5 for modelling, would this help store the status of the modelling workflow? if so, it is interesting, minus the licensing, and we could imagine an index file and then a series of indexed geoh5 projects. It would be nice to NOT duplicate/multiplicate common objects (e.g., input data) in every geoh5 proj... anyway, let;s keep the discussion going ;) Cheers, L.

…

On Fri, 18 Oct 2024 at 08:01, Lachlan Grose ***@***.***> wrote: I'll keep updating the first post with a list of options and summary of the requirements @ShebMichel <https://github.com/ShebMichel>, we are not talking about having the ability to export into different formats we are discussing the format that is used by Loop during the different processes, so it needs to be able to store the input data used by map2loop and store the outputs of map2loop + potentially the LoopStructural model results. I would also like to be able to store the state of a LoopStructural model to be loaded in to save reloading the model. We can have the ability to export from the Loop format to other formats. geoh5 has the disadvantage of relying on mira to maintain and update the format currently it seems locked to python 3.10 <https://github.com/MiraGeoscience/geoh5py/blob/develop/pyproject.toml> and is an LGPL license so that could be problematic. It is easily extended so would be able to add a lot of the objects we want. @markjessell <https://github.com/markjessell> ideally, I would like to define a structure for saving and loading the inputs and outputs of LoopStructural. But it doesn't have to be a single file, I would be open to using something like qgis/gocad where the "project" links to the data objects. The only issue with this is it opens up problems with missing files etc. — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSB3P6QQHQQG35R7U3ABT3Z4AQRNAVCNFSM6AAAAABQC4NIOSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJXGYZDINY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[LaurentAilleres]

Lets rethink the project file and make a list of all of the objects that should be stored in the project file, both input, output and temporary objects.

Input

• structure shapefile
• dtm
• fault trace
• geology polygon

Output/Intermediate

• basal contacts
• unit thickness distributions/spatial point estimates
• fault trace
• fault orientation
• fault surface
• stratigraphic/foliation surfaces
• scalar fields
• implicit function
• fault topology
• fault-stratigraphy topology
• stratigraphic column

Requirements

• compatible license
• pythonic interface
• fast read/write
• Backwards compatibility

Possible options are:

• geoh5 python library is LGPL licensed, not necessarily compatible with Loop
• geopackage extending geopackage will potentially make it incompatible with existing readers so may make it difficult to add new data types
• json/xml + binary - use a human readable file to store meta data/instructions to read binary or other file formats for each of the objects we require. This is more similar to how gocad/qgis manage the files
• Existing loop3d netcdf - keep using the existing format but make it fit for purpose
• Custom sqlite3 db - write out own schema for an sqlite database

@AngRodrigues @rabii-chaarani @noellehmcheng @markjessell @ShebMichel @LaurentAilleres

Hello @AngRodrigues @rabii-chaarani @noellehmcheng @markjessell @ShebMichel @lachlangrose

So - this discussion is not actually about formats (yet) and the original post may have been a little bit misdirecting. What we would like to discuss and achieve here is understand from a user point of view what we need to store and at which step of the process. Example below at a very high level and we need to go down to the granularity of the data point! Formats - all of those cited should be writable/readable, at least for 2D input. 3D models have to be stored as well... where? in the same structure or as separate structures...
The below does not consider geophysical data but we need to think about these as well... although I think the LoopProjectFile (the next one, the perfect one... the most appropriate one) should serve the purpose of describing the geological history and the data associated with it - think time-aware objects and the data that define their geometry.

Anyway, to get the ball rolling this is what I think we need to store and when - also be careful to distinguish between lithological and stratigraphic models):
Input data - at all times

• source of the data
• contacts as an interface between two formations (not as basal or top)
• structural measurements
• fault traces
• faults parameters (strike, dip of fault, shape (listric, planar, other)
• unconformity (does not have to be parallel to eroded or infill litho/strati) so should be a separate object
• drill hole derived information: intersection of given lithological/stratigraphic contact, structural orientations as points (these are not used in m2l but can be used in LS and are essential for 3D)
• same as above from cross-sections (although these are highly interpretive but if constrained by geophysics and consistent with the geological map then... people use them to test hyp)
• structural data related to deformation events... Sx, Lx, Lxs and overprinting (or relative timing of each)
• uncertainties of all above data; maybe an objective score as well... (I am looking sideways at the cross-sections...)

Augmented data - at all times (this is supposed to eventually become the input to LS)

• sampled contacts
• stratigraphic / lithological order
• type of stratigraphy / lithology (sedimentary, intrusive, unconformable etc)
• thickness(es) of each strati/litho and the method(s) used to estimate these -
• the location where these thicknesses were calculated (Lachlan's suggestion which I agree with)
• relative time of each "object" -> we need to be able to build a graph of the geological history
• upscaled structural data (strati strike/dip to folding and faults structural info)
• topological information - although this should be capture in the relative time even for faults... but faults may not interfere with other faults so we need to capture the F-F relationships and F-strati/litho relationships

Do we need to version the augmented data? YES. We also need to flag when the augmented data are modified by the user (ref Objectivity score) rather than a reproducible automated event.

3D model:

• modelling parameters (they are a lot, esp. when you start doing folds...)
• code version
• do we want to store interpolated structural fields and the grids they were calculated on? YES
• when we start generating ensembles: need to store them all etc...

Please add to the above list

[lachlangrose]

• Track changes for the input data (time stamped, undo button)
• No editing of the augmented data
• Can we link the augmented data to the input data/m2l algorithms