Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow IOManager to load external data at specific locations in Sidre hierarchy. #1479

Open
nselliott opened this issue Dec 12, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request Reviewed Sidre Issues related to Axom's 'sidre' component Spio Issues related to paralle I/O for Axom's Sidre component User Request Issues related to user requests

Comments

@nselliott
Copy link
Contributor

nselliott commented Dec 12, 2024

Currently, loading external data with SPIO uses a three-step process that works on the entire Sidre hierarchy stored in the output files.

IOManager reader;
// Step 1, reads from files, loading all data except for external Views, whose pointers are left as nullptr.
reader.read(group, root_file);

// Step 2, the user sets the addresses for the raw data buffers used by all external views. The calling
// code is responsible to ensure it uses valid addresses that point to buffers of the correct size.

// Step 3, call loadExternalData, which returns to the files to load data for all external Views into the
// buffers specified in Step 2. The same group and root_file from Step 1 must be passed to ensure
// that the hierarchy is traversed in the same way as Step 1.
reader.loadExternalData(group, root_file);

A user requests that we add new implementations of loadExternalData() that can load data into specified objects that may be anywhere in the hierarchy tree. Possible new method signatures would be:

void loadExternalData(sidre::Group* parent_group, sidre::Group* load_group, std::string root_file);
void loadExternalData(sidre::Group* parent_group, sidre::View* load_view, std::string root_file);

parent_group would need to be the same Group passed into read() as in Step 1 above. load_group would be a Group somewhere in the hierarchy tree of parent_group. A call to this version would load external data from the file only into Views that exist under load_group in the hierarchy. Similarly load_view would be a single View somewhere in the hierarchy of parent_group and this call would load external data only into this View.

A user code making use of these would be responsible to be sure it makes sufficient calls that will load data for all external Views in the hierarchy.

These should be reasonable to implement using mostly existing machinery.

@nselliott nselliott added enhancement New feature or request Sidre Issues related to Axom's 'sidre' component Spio Issues related to paralle I/O for Axom's Sidre component User Request Issues related to user requests labels Dec 12, 2024
@nselliott nselliott self-assigned this Dec 12, 2024
@rhornung67
Copy link
Member

Sounds like a good addition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Reviewed Sidre Issues related to Axom's 'sidre' component Spio Issues related to paralle I/O for Axom's Sidre component User Request Issues related to user requests
Projects
None yet
Development

No branches or pull requests

2 participants