Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

U/jrbogart/parquet star2 #82

Merged
merged 3 commits into from
Dec 1, 2023
Merged

U/jrbogart/parquet star2 #82

merged 3 commits into from
Dec 1, 2023

Conversation

JoanneBogart
Copy link
Collaborator

@JoanneBogart JoanneBogart commented Nov 30, 2023

Add ability to make star catalogs from the new parquet files (as well as old-style sqlite file).This is essentially the same code as was used to make star catalogs for the elias region but with starting point = current main.
Note that it will not do the right thing in general because it only looks at input parquet files sufficient for the elias region, a quick and dirty approach for the joint Roman-Rubin simulation. In the future I plan to rewrite the parquet files, partitioning them in a more healpix-friendly fashion. It will then be easy to create star skyCatalogs for any region.

Copy link
Collaborator

@jchiang87 jchiang87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for what it is, but it seems like it would be much better to pass either the path to the sqlite db file or the directory containing the parquet files as an argument to the create_sc.py script instead of the format being used. The code could infer the format from that input pretty readily, and one wouldn't need to write the NERSC paths in the code itself.

@JoanneBogart
Copy link
Collaborator Author

it seems like it would be much better to pass either the path to the sqlite db file or the directory containing the parquet files as an argument to the create_sc.py script instead of the format being used

Perhaps. Passing long file paths as arguments is awkward on the command line. If I did this for stars, it would be nice to do the same for galaxies (cosmodc2 or diffsky), but there is no difference in physical format for galaxies. One would have to parse the GCRCatalogs name.
There are a number of additional items one might want to pass to create_sc, but the interface is already rather cumbersome, mitigated somewhat by the ability to use options files.
These issues should be addressed in a separate PR.

@JoanneBogart JoanneBogart merged commit d0d0d58 into main Dec 1, 2023
1 check passed
@JoanneBogart JoanneBogart deleted the u/jrbogart/parquet_star2 branch December 1, 2023 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants