Added ingest methods to move experimental data to aimmdb #23
Conversation
| @@ -22,7 +22,8 @@ | |||
| import xraydb | |||
| from tiled.adapters.dataframe import DataFrameAdapter | |||
| from tiled.adapters.mapping import MapAdapter | |||
| from tiled.server.object_cache import with_object_cache | |||
|
|
|||
| # from tiled.server.object_cache import with_object_cache | |||
There was a problem hiding this comment.
Is this uncomment intended? I don't see with_object_cache used...plus, aren't we only using tiled client code to ingest?
There was a problem hiding this comment.
These are some comments that I left there temporarily. We used this approach while we were taking the first steps of using cache to build the Adapters in tiled. But with_object_cache doesn't exist in tiled anymore. I am wondering if there is different approach now that I might have missed, or I can use the build_reader method directly.
There was a problem hiding this comment.
Did you read this diff backward, @dylanmcreynolds? The cache is being removed here.
I had a quick chat with @jmaruland on this. I think the Heald Adapter code could be repurposed on the client side as a parser, but fully agree that caching in the adapter has no value in that context.
There was a problem hiding this comment.
Did you read this diff backward, @dylanmcreynolds? The cache is being removed here.
Sorry about that.
I think the Heald Adapter code could be repurposed on the client side as a parser, but fully agree that caching in the adapter has no value in that context
If I catch you correctly, we proposing using this code once per file to get into XDI-ish format on upload rather than every time the client gets the file?
There was a problem hiding this comment.
The story behind it is that, back then, when I was assigned as the "Gatekeeper" of AIMMdb, I received raw files in multiple formats from scientists of all three facilities which led me to build this collection of scripts that would allow me to save time in the future if I ever had to ingest more files to AIMMdb.
After the meeting I had with Monika last week, I am sure that you guys are mostly interested in the aimm_ingest.py::ingest_aimm_ncm_gihyeok_sigscan method to ingest the files that you have. I saw the files that you have at the moment, and they are very similar to a collection of files that I received from Gihyeok.
Keep in mind that these ingest methods were created when we still had the tiled tree model of key/value pairs that we built with Joe. This model is not being used in the latest version of AIMMdb and you might have to remove lines like this one:
https://github.com/jmaruland/aimm-adapters/blob/add-ingesting-methods/aimm_adapters/aimm_ingest.py#L606
This PR adds the methods that were used to ingest aimmdb with all of the experimental data that was provided by the effort of beamline scientists that are part of the AIMM project