ORM: Add `get_size_on_disk` method to `RemoteData` #6584

GeigerJ2 · 2024-10-15T16:00:05Z

Required for PR #6578.

By default, the get_size_on_disk method calls the method _get_size_on_disk_du that uses du to obtain the total directory size in bytes. If the call to du fails for whatever reason, recursive stat is being used, though, that is discouraged as stat returns the apparent size of files, not the actual disk usage.

I further extended the existing tests for RemoteData to use both, LocalTransport, as well as SshTransport, and test for the functionality added in this PR.

Pinging also @npaulish, as she's currently working on retrieving RemoteData objects.

codecov · 2024-10-15T16:32:14Z

Codecov Report

Attention: Patch coverage is 82.41758% with 16 lines in your changes missing coverage. Please review.

Project coverage is 77.94%. Comparing base (0f0b88a) to head (5aceeb3).

Files with missing lines	Patch %	Lines
src/aiida/orm/nodes/data/remote/base.py	88.06%	8 Missing ⚠️
src/aiida/cmdline/commands/cmd_data/cmd_remote.py	50.00%	7 Missing ⚠️
src/aiida/common/utils.py	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6584      +/-   ##
==========================================
+ Coverage   77.92%   77.94%   +0.02%     
==========================================
  Files         563      563              
  Lines       41671    41760      +89     
==========================================
+ Hits        32467    32544      +77     
- Misses       9204     9216      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

giovannipizzi · 2024-10-17T12:14:19Z

This is maybe machine-dependent, but rather than going via our API (that is more robust, but definitely going to be slower, I think) have a first "fast" option just running du -s and parsing the output (but careful about units! E.g. it uses "blocks", on some systems it's 512 or 2048bytes!! And if it fails, fall back to your solution?

GeigerJ2 · 2024-10-17T12:31:05Z

Note to self to run du via exec_command_wait method from transport.

khsrali

Thanks @GeigerJ2 please go through more pains that I imposed 😈

src/aiida/common/utils.py

src/aiida/orm/nodes/data/remote/base.py

khsrali · 2024-11-21T15:50:14Z

src/aiida/orm/nodes/data/remote/base.py

+
+        return format_directory_size(size_in_bytes=total_size)
+
+    def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:


this whole function, doesn't really need to be separated I feel.. It's basically only executes a command, and you don't re-use it, right?

I found the design to have just one top-level method get_size_on_disk nice, which tries to call the two private methods, in terms of separation of concerns.

Though, it is true that this pollutes the API of RemoteData somewhat... So I'm also fine of either moving the private methods to some utils module, or merging them. Maybe @unkcpz can comment on good coding practices here?

which tries to call the two private methods, in terms of separation of concerns.

I'm not sure if I understand, sorry 😬
Since it's not really re-usable by other method, I'd vote for merging it, and avoid over-fictionalizing

src/aiida/orm/nodes/data/remote/base.py

tests/orm/nodes/data/test_remote.py

GeigerJ2

Not sure why my replies via the files GH tab appear as a review, but whatever... ^^

Thanks for the review @khsrali. I implemented most of your proposed changes!

As to our in-person discussion if du or lstat is preferred, I think none of the two is ideal... lstat giving the actual byte-sized content, which is neat and all, but won't correspond to the disk space that will actually be occupied locally by a file, due to the use of blocks for the file system. And du giving the actual occupied disk space on the remote, which, however, might be different from the local file system (due to having a different file system on the local machine, different formatting, different block size, etc.). Hence, the big difference in the file size check in the test. For real-world use cases, with more and larger files, the difference is likely much smaller, and won't matter too much, I think. I'll add a test for a larger file, as well as modify the message that the given size is just an estimate, so that user are aware they should take the value with a grain of salt.

Maybe also @agoscinski with his actual computer science background can weigh in 🫶

src/aiida/orm/nodes/data/remote/base.py

GeigerJ2 · 2024-11-25T10:57:42Z

src/aiida/orm/nodes/data/remote/base.py

+
+        return format_directory_size(size_in_bytes=total_size)
+
+    def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:


I found the design to have just one top-level method get_size_on_disk nice, which tries to call the two private methods, in terms of separation of concerns.

Though, it is true that this pollutes the API of RemoteData somewhat... So I'm also fine of either moving the private methods to some utils module, or merging them. Maybe @unkcpz can comment on good coding practices here?

src/aiida/orm/nodes/data/remote/base.py

khsrali

Thanks a lot @GeigerJ2! Just a few minor comments..

src/aiida/common/utils.py

src/aiida/orm/nodes/data/remote/base.py

tests/orm/nodes/data/test_remote.py

src/aiida/orm/nodes/data/remote/base.py

khsrali · 2024-11-25T15:56:13Z

src/aiida/orm/nodes/data/remote/base.py

+
+        return format_directory_size(size_in_bytes=total_size)
+
+    def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:


which tries to call the two private methods, in terms of separation of concerns.

I'm not sure if I understand, sorry 😬
Since it's not really re-usable by other method, I'd vote for merging it, and avoid over-fictionalizing

agoscinski

Yes seems like we should split the function into two or provide optional arguments. These two concepts seem to be distinguished by the terms "disk usage" and "apparent size" (see https://stackoverflow.com/a/569485). I think the output of du --apparent-size is the same as with lstat, so you do not need to use lstat

src/aiida/orm/nodes/data/remote/base.py

src/aiida/cmdline/commands/cmd_data/cmd_remote.py

src/aiida/orm/nodes/data/remote/base.py

GeigerJ2 · 2024-12-10T15:44:03Z

Thanks for the review, @agoscinski! I'm currently still working on this, will ping you once it's again ready for review.

I think the output of du --apparent-size is the same as with lstat, so you do not need to use lstat

The reason I'm providing lstat as a fallback option is if du is not available (e.g., MacOS, as you mentioned, didn't know that ^^), or if exec_command_wait isn't available, which will be the case for FirecREST.

GeigerJ2 · 2024-12-11T13:08:28Z

OK, this should be ready for a final review, @agoscinski and @khsrali. Also pinging, @mikibonacci, if you want to provide some feedback on the CLI/API for use in AiiDAlab?

khsrali

Thanks @GeigerJ2 ,
Please consider my review, I haven't checked the tests thoroughly. I leave that to @agoscinski

src/aiida/cmdline/commands/cmd_data/cmd_remote.py

src/aiida/orm/nodes/data/remote/base.py

tests/orm/nodes/data/test_remote.py

In addition, extend the tests for `RemoteData` in `test_remote.py` for the methods added in this PR, as well as parametrize them to run on a `RemoteData` via local and ssh transport.

for more information, see https://pre-commit.ci

… nested directory

for more information, see https://pre-commit.ci

GeigerJ2 · 2024-12-12T11:45:37Z

Thanks again for the review, @khsrali, I implemented your proposed changes.

for more information, see https://pre-commit.ci

GeigerJ2 marked this pull request as draft October 15, 2024 16:02

GeigerJ2 mentioned this pull request Oct 15, 2024

Add --also-remote option to verdi process dump #6578

Open

GeigerJ2 changed the title ~~Add get_total_size_on_disk method to RemoteData.~~ Add get_total_size_on_disk method to RemoteData Oct 17, 2024

mikibonacci mentioned this pull request Oct 17, 2024

On clean remote data button and checks aiidalab/aiidalab-qe#857

Open

3 tasks

GeigerJ2 requested a review from khsrali November 21, 2024 14:53

GeigerJ2 force-pushed the feature/remote-data-total-size branch from e801a78 to e0be575 Compare November 21, 2024 14:54

GeigerJ2 marked this pull request as ready for review November 21, 2024 14:54

GeigerJ2 changed the title ~~Add get_total_size_on_disk method to RemoteData~~ Add get_size_on_disk method to RemoteData Nov 21, 2024

GeigerJ2 force-pushed the feature/remote-data-total-size branch from e0be575 to b1a0560 Compare November 21, 2024 15:14

khsrali requested changes Nov 21, 2024

View reviewed changes

GeigerJ2 commented Nov 25, 2024

View reviewed changes

GeigerJ2 force-pushed the feature/remote-data-total-size branch from f2f90f9 to 4fcb1ba Compare November 25, 2024 15:41

khsrali requested changes Nov 25, 2024

View reviewed changes

GeigerJ2 force-pushed the feature/remote-data-total-size branch from f7dfe7e to ecfa53b Compare December 9, 2024 09:36

agoscinski reviewed Dec 10, 2024

View reviewed changes

GeigerJ2 force-pushed the feature/remote-data-total-size branch from 58b3248 to 8bcf0f8 Compare December 11, 2024 10:59

GeigerJ2 changed the title ~~Add get_size_on_disk method to RemoteData~~ ORM: Add get_size_on_disk method to RemoteData Dec 11, 2024

khsrali requested changes Dec 12, 2024

View reviewed changes

GeigerJ2 and others added 7 commits December 12, 2024 12:44

Add get_total_size_on_disk method to RemoteData.

b4eb558

Implement function with du and lstat fallback

1c78059

In addition, extend the tests for `RemoteData` in `test_remote.py` for the methods added in this PR, as well as parametrize them to run on a `RemoteData` via local and ssh transport.

Implement function with du and lstat fallback

87ed105

In addition, extend the tests for `RemoteData` in `test_remote.py` for the methods added in this PR, as well as parametrize them to run on a `RemoteData` via local and ssh transport.

Apply changes from code review by @khsrali

5535af9

[pre-commit.ci] auto fixes from pre-commit.com hooks

493e98d

for more information, see https://pre-commit.ci

Add tests for different file sizes.

917c5d0

Backup to check out other branch.

94e12c5

GeigerJ2 and others added 9 commits December 12, 2024 12:44

Final code cleanup

465a93a

[pre-commit.ci] auto fixes from pre-commit.com hooks

0aa54bb

for more information, see https://pre-commit.ci

Address final minor review comments

b4eaf9f

Expand CLI, method argument, fix recursive bug, expand tests also for…

32d753d

… nested directory

[pre-commit.ci] auto fixes from pre-commit.com hooks

a24d721

for more information, see https://pre-commit.ci

Final polishing.

1a35f3d

Fix docstrings for RTD build.

4ef2fca

Fix docstrings for RTD build... second try.

d91f4f1

Apply changes from code review.

192b0fa

GeigerJ2 force-pushed the feature/remote-data-total-size branch from f0bb1f6 to 192b0fa Compare December 12, 2024 11:44

GeigerJ2 and others added 2 commits December 12, 2024 12:56

Minor changes

ce46202

[pre-commit.ci] auto fixes from pre-commit.com hooks

5aceeb3

for more information, see https://pre-commit.ci

GeigerJ2 mentioned this pull request Dec 12, 2024

Adding get_size_on_remote to the Transport interface #6665

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ORM: Add `get_size_on_disk` method to `RemoteData` #6584

ORM: Add `get_size_on_disk` method to `RemoteData` #6584

GeigerJ2 commented Oct 15, 2024 •

edited

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading

giovannipizzi commented Oct 17, 2024

GeigerJ2 commented Oct 17, 2024 •

edited

Loading

khsrali left a comment

khsrali Nov 21, 2024

GeigerJ2 Nov 25, 2024

khsrali Nov 25, 2024

GeigerJ2 left a comment

GeigerJ2 Nov 25, 2024

khsrali left a comment

khsrali Nov 25, 2024

agoscinski left a comment

GeigerJ2 commented Dec 10, 2024

GeigerJ2 commented Dec 11, 2024

khsrali left a comment

GeigerJ2 commented Dec 12, 2024


		return format_directory_size(size_in_bytes=total_size)

		def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:

ORM: Add get_size_on_disk method to RemoteData #6584

Are you sure you want to change the base?

ORM: Add get_size_on_disk method to RemoteData #6584

Conversation

GeigerJ2 commented Oct 15, 2024 • edited Loading

codecov bot commented Oct 15, 2024 • edited Loading

Codecov Report

giovannipizzi commented Oct 17, 2024

GeigerJ2 commented Oct 17, 2024 • edited Loading

khsrali left a comment

Choose a reason for hiding this comment

khsrali Nov 21, 2024

Choose a reason for hiding this comment

GeigerJ2 Nov 25, 2024

Choose a reason for hiding this comment

khsrali Nov 25, 2024

Choose a reason for hiding this comment

GeigerJ2 left a comment

Choose a reason for hiding this comment

GeigerJ2 Nov 25, 2024

Choose a reason for hiding this comment

khsrali left a comment

Choose a reason for hiding this comment

khsrali Nov 25, 2024

Choose a reason for hiding this comment

agoscinski left a comment

Choose a reason for hiding this comment

GeigerJ2 commented Dec 10, 2024

GeigerJ2 commented Dec 11, 2024

khsrali left a comment

Choose a reason for hiding this comment

GeigerJ2 commented Dec 12, 2024

ORM: Add `get_size_on_disk` method to `RemoteData` #6584

ORM: Add `get_size_on_disk` method to `RemoteData` #6584

GeigerJ2 commented Oct 15, 2024 •

edited

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading

GeigerJ2 commented Oct 17, 2024 •

edited

Loading