Standardising atlas file downloading #351

viktorpm · 2024-08-09T14:21:14Z

Description

This PR introduces a standardized method for downloading atlas files using pooch.

Bug fix
Addition of a new feature
Other

How has this PR been tested?

Scripts were tested locally

Is this a breaking change?

No

Does this PR require an update to the documentation?

Potentially

Checklist:

The code has been tested locally
Tests have been added to cover all new functionality (unit & integration)
The documentation has been updated to reflect any changes
The code has been formatted with pre-commit

…uring packaging

for more information, see https://pre-commit.ci

alessandrofelder

Thanks a lot @viktorpm - this is good progress!
And sorry for letting this sit so long 🙈

I've suggested some minor changes and asked for minor clarifications.

Some high-level comments

there's quite a bit of commented out code that I think should be removed
I'm confused by the ### comments, but if we want to keep them we should add them everywhere.

We should open issues for

removing the current way of parallelising mesh creation everywhere
decide what to do with __version__ and __atlas__
make mesh creation parallel in a more robust way

What do you think?

alessandrofelder · 2024-10-28T14:54:57Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

@@ -1,52 +1,58 @@
-__version__ = "1"
-
+### Import


Are these comments useful and why do they start with ###? I may be missing something. If they are useful, should they be added to the admba_3d_dev_mouse script too?

They are just there to structure the code into sections.

Import: loading packages

Settings: options to turn on and off

Metadata: atlas related information

If it's confusing or not good practice, we can get rid of them. Ideally, all packaging scripts should follow this structure including admba_3d_dev_mouse. I probably just forgot to add the headers. That script is a bit of an exception as the generated atlases are treated as separate atlases not as variations of the same atlas.

If it's confusing or not good practice

My take on this is that at least the Metadata and Import ones are redundant.
I'm doubtful that we want to keep settings hard-coded on a per-atlas level going forward.

there to structure the code into sections.

Hopefully the structure introduced by #400 will sufficiently structure the scripts.
So I would lean towards removing them.

That script is a bit of an exception as the generated atlases are treated as separate atlases not as variations of the same atlas.

This, on the other hand, is possibly useful contextual explanation that might be a good comment to have in that script (if it's not there yet)!

brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/azba_zfish.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/humanatlas.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/mpin_zfish.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/osten_mouse.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/perens_lsfm_mouse.py

brainglobe_atlasapi/atlas_generation/atlas_scripts/whs_sd_rat.py

removing commented-out lines Co-authored-by: Alessandro Felder <[email protected]>

removing commented-out line Co-authored-by: Alessandro Felder <[email protected]>

removing unused code Co-authored-by: Alessandro Felder <[email protected]>

…_mouse.py removing unused code Co-authored-by: Alessandro Felder <[email protected]>

removing unnecesseray line Co-authored-by: Alessandro Felder <[email protected]>

removing unnecessary line Co-authored-by: Alessandro Felder <[email protected]>

alessandrofelder

Thanks @viktorpm - I've replied to your questions, I think.

Could you resolve the conflicts with main and ask for my review when you have please? Then we will be almost there!

alessandrofelder · 2024-11-01T10:31:02Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

@@ -1,52 +1,58 @@
-__version__ = "1"
-
+### Import


If it's confusing or not good practice

My take on this is that at least the Metadata and Import ones are redundant.
I'm doubtful that we want to keep settings hard-coded on a per-atlas level going forward.

there to structure the code into sections.

Hopefully the structure introduced by #400 will sufficiently structure the scripts.
So I would lean towards removing them.

That script is a bit of an exception as the generated atlases are treated as separate atlases not as variations of the same atlas.

This, on the other hand, is possibly useful contextual explanation that might be a good comment to have in that script (if it's not there yet)!

brainglobe_atlasapi/atlas_generation/atlas_scripts/mpin_zfish.py

for more information, see https://pre-commit.ci

alessandrofelder · 2024-11-11T13:13:13Z

We should open issues for

removing the current way of parallelising mesh creation everywhere [BUG] remove unreliable parallel running of mesh creation #428

decide what to do with version and atlas [Feature] Move away from __version__ to signify atlas minor version in packaging scripts. #423

make mesh creation parallel in a more robust way [Feature] Speed up packaging for atlases, e.g. by parallelising mesh generation (robustly!) #429

Crosslinking issues here for documentation purposes.

alessandrofelder

Thanks a lot for this @viktorpm

I've made some final beautifications and will merge when (if!) the tests pass :)

removed some unused code that was still there
fixed the formatter to make the linter happy and run the tests (The problem was the order of docstring, version and imports)

viktorpm and others added 8 commits August 9, 2024 14:59

adding known_hash argument to AtlasConfig

6f740cd

defining DEFAULT_WORKDIR in config.py to standardise atlas location d…

f348196

…uring packaging

adding DEFAULT_WORKDIR and __atlas__ to packaging scripts

dfb0db0

implementing pooch

a935fde

implementing pooch

25c844b

complete refactoring of humanatlas.py, implementing pooch

14007f8

Merge branch 'main' into standardising_atlas_file_downloading

a0d6378

[pre-commit.ci] auto fixes from pre-commit.com hooks

9623cdb

for more information, see https://pre-commit.ci

viktorpm marked this pull request as ready for review September 4, 2024 10:50

viktorpm added 2 commits September 6, 2024 10:55

fixing file downloading

14b87e5

Restructuring code: metadata on top

31c5527

viktorpm requested a review from alessandrofelder September 16, 2024 14:30

alessandrofelder requested changes Oct 28, 2024

View reviewed changes

viktorpm and others added 8 commits October 31, 2024 11:47

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

efff267

removing commented-out lines Co-authored-by: Alessandro Felder <[email protected]>

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/humanatlas.py

ede991a

removing commented-out line Co-authored-by: Alessandro Felder <[email protected]>

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/osten_mouse.py

79b75d6

removing unused code Co-authored-by: Alessandro Felder <[email protected]>

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/perens_lsfm…

728f451

…_mouse.py removing unused code Co-authored-by: Alessandro Felder <[email protected]>

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/azba_zfish.py

c6eb451

removing unnecesseray line Co-authored-by: Alessandro Felder <[email protected]>

Update brainglobe_atlasapi/atlas_generation/atlas_scripts/whs_sd_rat.py

8ae75b2

removing unnecessary line Co-authored-by: Alessandro Felder <[email protected]>

setting PARALLEL = False, removing unnecessary comment

1abffed

updating load_nii import

4beff30

viktorpm requested a review from alessandrofelder October 31, 2024 16:27

alessandrofelder requested changes Nov 1, 2024

View reviewed changes

viktorpm and others added 4 commits November 6, 2024 13:34

removing structuring headers

44a5cd1

moving __version__ back to the top

4380ec4

Merge branch 'main' into standardising_atlas_file_downloading

5cb560f

[pre-commit.ci] auto fixes from pre-commit.com hooks

2ac60d1

for more information, see https://pre-commit.ci

viktorpm requested a review from alessandrofelder November 6, 2024 13:52

Apply suggestions from code review

f56fa9a

fix module dunder, module docstring and import order

9921d72

alessandrofelder approved these changes Nov 11, 2024

View reviewed changes

alessandrofelder merged commit e9fdb99 into main Nov 11, 2024
12 of 13 checks passed

viktorpm deleted the standardising_atlas_file_downloading branch November 11, 2024 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardising atlas file downloading #351

Standardising atlas file downloading #351

viktorpm commented Aug 9, 2024 •

edited

Loading

alessandrofelder left a comment •

edited

Loading

alessandrofelder Oct 28, 2024

viktorpm Oct 31, 2024

alessandrofelder Nov 1, 2024

alessandrofelder left a comment

alessandrofelder Nov 1, 2024

alessandrofelder commented Nov 11, 2024

alessandrofelder left a comment •

edited

Loading

Standardising atlas file downloading #351

Standardising atlas file downloading #351

Conversation

viktorpm commented Aug 9, 2024 • edited Loading

Description

How has this PR been tested?

Is this a breaking change?

Does this PR require an update to the documentation?

Checklist:

alessandrofelder left a comment • edited Loading

Choose a reason for hiding this comment

alessandrofelder Oct 28, 2024

Choose a reason for hiding this comment

viktorpm Oct 31, 2024

Choose a reason for hiding this comment

alessandrofelder Nov 1, 2024

Choose a reason for hiding this comment

alessandrofelder left a comment

Choose a reason for hiding this comment

alessandrofelder Nov 1, 2024

Choose a reason for hiding this comment

alessandrofelder commented Nov 11, 2024

alessandrofelder left a comment • edited Loading

Choose a reason for hiding this comment

viktorpm commented Aug 9, 2024 •

edited

Loading

alessandrofelder left a comment •

edited

Loading

alessandrofelder left a comment •

edited

Loading