Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have relative_to return PosixUPath #215

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Have relative_to return PosixUPath #215

wants to merge 1 commit into from

Conversation

CompRhys
Copy link

@CompRhys CompRhys commented Apr 15, 2024 •
edited by ap--
Loading

Close #214

Test that should pass if the local path code worked as expected

ap--
ap-- requested changes Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@ap-- ap-- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for the PR! 🙏 ❤️

I made a few comments. Let me know what you think.

Cheers,
Andreas

upath/tests/implementations/test_local.py Outdated Show resolved Hide resolved
upath/implementations/local.py Outdated Show resolved Hide resolved
upath/implementations/local.py Outdated
@@ -62,6 +63,42 @@ def iterdir(self):
if _LISTDIR_WORKS_ON_FILES and self.is_file():
raise NotADirectoryError(f"{self}")
return super().iterdir()

def relative_to(self, other, /, *_deprecated, walk_up=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not too much work, could you try moving this implementation to the UPath class?

It might just fix the other open relative_to issues too.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that would give a circular dependency is the issue

upath/tests/implementations/test_local.py
@@ -17,6 +17,9 @@ def path(self, local_testdir):
def test_is_LocalPath(self):
assert isinstance(self.path, LocalPath)

def test_relative_to(self):
assert 'file.txt' == UPath("file:///test_bucket/file.txt").relative_to(UPath("file:///test_bucket")).path

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if you could add a test that checks if a ValueError is raised if a UPath of a different protocol is provided as argument.

And a test that checks the cases for the walk_up keyword argument.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a test for s3, not sure what walk_up allows will need more time to look at that to write a test case

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a test for walk_up seems like it's broken for azure

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seemingly its because UPath("az:///other_test_bucket").parents is an empty list which I don't understand

@ap--
Copy link
Collaborator

ap-- commented Apr 16, 2024

You can prevent some of the failing tests, by editing

universal_pathlib/setup.cfg

Lines 22 to 28 in 380144c

[options]
python_requires = >=3.8
zip_safe = False
packages = find:
install_requires=
fsspec>=2022.1.0

to use: fsspec >=2022.1.0,!=2024.3.1

This is due to a incompatibility of UPath's test suite with the newest fsspec.

@CompRhys CompRhys changed the title Add test for local pth Have relative_to return PosixUPath Apr 17, 2024
@bolkedebruin
Copy link

Where are we with this? I would be nice to have this in, because the current behavior is broken :-).

@ap--
Copy link
Collaborator

ap-- commented May 21, 2024

With the new fsspec release, this is ready to be merged. I'm on holiday until the end of May and will prepare a new release when I'm back 😊

@bolkedebruin
Copy link

Great!

@bolkedebruin
Copy link

bolkedebruin commented May 21, 2024 •
edited
Loading

Thinking and testing this a little bit further (from an Airflow perspective, but I don't think that matters). The following does not seem to make sense:

o = ObjectStoragePath("s3://tmp/test")
t = ObjectStoragePath("s3://tmp/test/subdir/bla/bla.txt")

t.relative_to(o).resolve()

prints PosixUPath('/Users/bolke/dev/airflow/subdir/bla/bla.txt')

which doesn't make sense either way. If posix it should resolve to tmp/test/subdir/bla/bla.txt or if s3 (that should make more sense I guess?) to s3://tmp/test/subdir/bla/bla.txt.

Isn't it better to return a PurePath or keep a reference to the original?

@bolkedebruin
Copy link

What I am considering is to have

ObjectStoragePath.relative_to return itself (e.g. an ObjectStoragePath) with a storage_option of parent set to the self.

__str__ then reproduces the relative path by checking if parent was set, but all other mechanisms intact.

@ap-- ap-- mentioned this pull request Jun 14, 2024
Open
16 tasks
@CompRhys
Copy link
Author

coming back to clean up some old PRs, completely no idea what I was attempting to do here as was for an old job but have merged the recent changes and rebased.

Let me know if I can help

@scholtalbers
Copy link

Would be nice to have this resolved. I agree that returning PosixUPath is not ideal, but at least its better than the current state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ap-- ap-- ap-- requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

relative_to doesn't work for FilePath
4 participants
@CompRhys @ap-- @bolkedebruin @scholtalbers