Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implent CC summarizer "yet_another_wizz" #17

Open
jlvdb opened this issue Jul 11, 2022 · 3 comments
Open

Implent CC summarizer "yet_another_wizz" #17

jlvdb opened this issue Jul 11, 2022 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@jlvdb
Copy link

jlvdb commented Jul 11, 2022

Implement the original code at https://github.com/jlvdb/yet_another_wizz to fit into RAIL and improve the performance to match the scale of the future datasets.

Main challenges:

  • Implementing the estimator efficiently (potentially extending the scipy.spatial.cKDTree to handle inverse distance weights)
  • Parallelization (probably spatial splitting of the data set)
@aimalz
Copy link
Collaborator

aimalz commented Jul 12, 2022

Given that RAIL now speaks ceci, it might be worth looking into what existing infrastructure from TXPipe could potentially be called in a RAIL stage specifically for the cross-correlation step. @joezuntz has thought about this a bunch and probably has additional ideas for efficiency.

@sschmidt23
Copy link
Collaborator

If I remember correctly that YAW is similar to the-wizz in switching from actual correlation function calculations to KDTree matching, then we probably don't need any of the actual TreeCorr stuff. But, I think we do need to figure out how to do proper mask generation/use and generating randoms in a way that is consistent with how other DESC people are doing those operations.

@jlvdb
Copy link
Author

jlvdb commented Jul 12, 2022

Good points, Alex and Sam, I will check TXPipe and talk to Joe if necessary. I'll try to use as much existing infrastructure as possible unless that requires changing too much of the existing code base.

@eacharles eacharles transferred this issue from LSSTDESC/rail_attic Jun 14, 2023
@aimalz aimalz added the enhancement New feature or request label Jul 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants