Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check file existence when reading yaml #6

Open
jfcrenshaw opened this issue Aug 5, 2022 · 2 comments
Open

Check file existence when reading yaml #6

jfcrenshaw opened this issue Aug 5, 2022 · 2 comments
Assignees
Labels
good first issue Good for newcomers

Comments

@jfcrenshaw
Copy link
Contributor

jfcrenshaw commented Aug 5, 2022

When RAIL is reading the yaml to set up the pipeline, it should check (at the very beginning!) if the stated output files already exist and refuse to overwrite them, unless flag OVERWRITE=True

@aimalz aimalz added the good first issue Good for newcomers label Oct 25, 2022
@OliviaLynn OliviaLynn self-assigned this Oct 25, 2022
OliviaLynn referenced this issue in LSSTDESC/rail_attic Nov 28, 2022
@OliviaLynn
Copy link
Member

OliviaLynn commented Feb 27, 2023

I want to clarify how this should behave with respect to "resume" mode, which I understand to be a flag one can set so the pipeline will "ignore stages if all of their output already exists" (described in "Initialize the pipeline" step of Pipe_Example.ipynb, and included as a value set in pipe_example.yml). This mode seems pretty similar to me.

If we add a flag a user can toggle when reading and running a pipeline, should this just completely override the value of resume mode specified in the pipeline's .yml when the OVERWRITE flag has been set to either true or false, but default to the pipeline's resume mode if we chose not to include overwrite as a keyword?

@aimalz I remember we spoke about this but only briefly - I was wondering if you have any thoughts.

Edit to add: maybe this just needs unit tests to confirm that this flag is actually working?

@aimalz
Copy link
Collaborator

aimalz commented May 23, 2023

This might be a good place to also check if input files exist where they're expected to, and it might be related to resolving occurrences of stages needing their input files to be in src/rail.

@eacharles eacharles transferred this issue from LSSTDESC/rail_attic Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants