Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes for OSDF support #4459

Merged
merged 11 commits into from
Aug 31, 2023
Merged

Changes for OSDF support #4459

merged 11 commits into from
Aug 31, 2023

Conversation

spxiwh
Copy link
Contributor

@spxiwh spxiwh commented Aug 14, 2023

This PR adds support for OSDF (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) transfers in PyCBC.workflow.

Pegasus added OSDF support in 5.0.6 (hence the version bump), but we need a few pointers to make it work as intended.

This is now ready to merge because we have fixed:

  • Don't yet know how this would interact with datafind. We would expect frame files to be a prime target for OSDF transfers. - datafind just takes a urltype=osdf flag. I've also added in the necessary handling to ensure pegasus does the right thing with such files.
  • I don't know how this will work for singularity images (but think I will likely do that separately in any case). - Not going to consider singularity here.
  • Need to consider how to do an OSDF transfer at runtime (if desired). - This has need added.

@spxiwh spxiwh requested a review from ahnitz August 14, 2023 15:51
@spxiwh
Copy link
Contributor Author

spxiwh commented Aug 30, 2023

This should be ready to merge, and I would like to get it into a forthcoming release as it will allow more efficient access to frame files. (This may also be implemented in GWOSC, which would also help).

Copy link
Member

@ahnitz ahnitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks reasonable to me.

@spxiwh spxiwh merged commit 617ca30 into gwastro:master Aug 31, 2023
31 checks passed
@spxiwh spxiwh deleted the pr_osdf_support branch August 31, 2023 14:29
GarethCabournDavies pushed a commit to GarethCabournDavies/pycbc that referenced this pull request Oct 11, 2023
* Changes for OSDF support

* Bump version here as well

* Add OSDF handling in workflow.datafind

* Add OSDF handling

* Add in OSDF transfer code

* Make OSDF local transfers work

* Code Climating

* Should include IGWN requirements

* Trying to fix failures

* Revert moving ciecplib import

* Missed undoing this change
PRAVEEN-mnl pushed a commit to PRAVEEN-mnl/pycbc that referenced this pull request Nov 3, 2023
* Changes for OSDF support

* Bump version here as well

* Add OSDF handling in workflow.datafind

* Add OSDF handling

* Add in OSDF transfer code

* Make OSDF local transfers work

* Code Climating

* Should include IGWN requirements

* Trying to fix failures

* Revert moving ciecplib import

* Missed undoing this change
bhooshan-gadre pushed a commit to bhooshan-gadre/pycbc that referenced this pull request Mar 4, 2024
* Changes for OSDF support

* Bump version here as well

* Add OSDF handling in workflow.datafind

* Add OSDF handling

* Add in OSDF transfer code

* Make OSDF local transfers work

* Code Climating

* Should include IGWN requirements

* Trying to fix failures

* Revert moving ciecplib import

* Missed undoing this change
acorreia61201 pushed a commit to acorreia61201/pycbc that referenced this pull request Apr 4, 2024
* Changes for OSDF support

* Bump version here as well

* Add OSDF handling in workflow.datafind

* Add OSDF handling

* Add in OSDF transfer code

* Make OSDF local transfers work

* Code Climating

* Should include IGWN requirements

* Trying to fix failures

* Revert moving ciecplib import

* Missed undoing this change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants