Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preselection: Implement Dask/condor submission for looper #1

Open
sam-may opened this issue Feb 5, 2021 · 1 comment
Open

Preselection: Implement Dask/condor submission for looper #1

sam-may opened this issue Feb 5, 2021 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@sam-may
Copy link
Collaborator

sam-may commented Feb 5, 2021

Implement functionality to submit jobs to Dask and/or condor for the preselection looper.

Implementation would go in this function in prep_helper.py:

def submit_jobs(self):

Likely the cleanest way to do it would be to make a Batch directory and build helper classes for Dask/condor submission there.

Merging of output dataframes would also need to be updated accordingly:

def merge_outputs(self):
master_file = self.output_dir + self.selections + "_" + self.output_tag + ".pkl"
master_df = pandas.DataFrame()
for file in self.outputs:
if self.debug > 0:
print("[LoopHelper] Loading file %s" % file)
if not os.path.exists(file):
continue
df = pandas.read_pickle(file)
master_df = pandas.concat([master_df, df], ignore_index=True)
master_df.to_pickle(master_file)

Another thing to keep in mind: it would be nice to have the batch submission tools not be entirely specific to the looper (or at least easily generalizable), as they will also be useful for MVA training (e.g. hyperparameter scans) and Signal Region Optimization (scanning MVA cut values).

@sam-may sam-may added the enhancement New feature or request label Feb 5, 2021
@bsathian
Copy link
Contributor

I can take this up if no one else wants to

@bsathian bsathian self-assigned this Feb 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants