Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: globbing with fsspec #1061

Merged
merged 37 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
7b05aab
feat: add fsspec as required dependency (#1021)
lobis Nov 14, 2023
ab582a9
feat: set fsspec as default source (#1023)
lobis Nov 15, 2023
ba95806
feat: set `fsspec` (`s3fs`) as default handler for s3 paths (#1032)
lobis Nov 17, 2023
6b6c38b
feat: simplify object path split (#1028)
lobis Nov 16, 2023
44d92cf
feat: fsspec for all non-object writing - %-encoded urls no longer de…
lobis Nov 20, 2023
167f3ae
test: improve path object split tests (#1039)
lobis Nov 27, 2023
1d07dad
test: add test for issue 1054 (newer fsspec failing to parse files wi…
lobis Dec 11, 2023
8a5dd52
use fsspec to expand glob
lobis Dec 12, 2023
9a53888
skip root from remote_schemas
lobis Dec 12, 2023
b744d37
test iterate over xrootd
lobis Dec 12, 2023
d03dbef
test
lobis Dec 12, 2023
864822d
add temporary install to ci
lobis Dec 12, 2023
1bc201f
remove ci debug
lobis Dec 12, 2023
9b4bc3a
feat: add fsspec as required dependency (#1021)
lobis Nov 14, 2023
f25e36f
feat: set fsspec as default source (#1023)
lobis Nov 15, 2023
81995ab
feat: set `fsspec` (`s3fs`) as default handler for s3 paths (#1032)
lobis Nov 17, 2023
011db53
feat: simplify object path split (#1028)
lobis Nov 16, 2023
f62798d
feat: fsspec for all non-object writing - %-encoded urls no longer de…
lobis Nov 20, 2023
5c48ba7
test: add test for issue 1054 (newer fsspec failing to parse files wi…
lobis Dec 11, 2023
71ccdbd
test: improve path object split tests (#1039)
lobis Nov 27, 2023
738411c
Merge branch 'main-fsspec' into globbing-fsspec
lobis Dec 12, 2023
cdb70fe
test: add test for issue 1054 (newer fsspec failing to parse files wi…
lobis Dec 11, 2023
d0f4458
try to expand all glob strings if they have the protocol
lobis Dec 12, 2023
9573e39
Merge branch 'main-fsspec' into globbing-fsspec
lobis Dec 12, 2023
50835ae
making it work on windows
lobis Dec 12, 2023
db79ab4
testing globbing for s3
lobis Dec 13, 2023
72f6c09
add failing test for http globbing
lobis Dec 13, 2023
8d8d120
test more handlers, failing test for xrootd (missing files)
lobis Dec 13, 2023
8c4c75e
understanding error
lobis Dec 13, 2023
27772c5
add class method to extract fsspec options
lobis Dec 13, 2023
42a89ed
call super constructor for fsspec source
lobis Dec 13, 2023
fe1681f
pass options to regularize files util
lobis Dec 13, 2023
08dc012
python 3.12 aiohttp test in other PR
lobis Dec 13, 2023
e1fc00d
attempt to hide the ssl destructor error
lobis Dec 13, 2023
771cd44
retry on "expired"
lobis Dec 13, 2023
1baf95b
Merge branch 'main-fsspec' into globbing-fsspec
lobis Dec 13, 2023
1cd7835
style: pre-commit fixes
pre-commit-ci[bot] Dec 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ dependencies = [
"awkward>=2.4.6",
"importlib-metadata;python_version<\"3.8\"",
"numpy",
"fsspec",
"packaging",
"typing_extensions>=4.1.0; python_version < \"3.11\""
]
Expand Down Expand Up @@ -63,7 +64,6 @@ test = [
"zstandard",
"minio",
"aiohttp; python_version<\"3.12\"", # asyncio not available
"fsspec",
"fsspec-xrootd",
"s3fs; python_version<\"3.12\"", # asyncio not available
"paramiko",
Expand Down
2 changes: 1 addition & 1 deletion src/uproot/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@
isort:skip_file
"""


from uproot.version import __version__
import uproot.const
import uproot.extras
Expand All @@ -92,6 +91,7 @@
from uproot.source.xrootd import MultithreadedXRootDSource
from uproot.source.s3 import S3Source
from uproot.source.object import ObjectSource
from uproot.source.fsspec import FSSpecSource
from uproot.source.cursor import Cursor
from uproot.source.futures import TrivialExecutor
from uproot.source.futures import ThreadPoolExecutor
Expand Down
5 changes: 0 additions & 5 deletions src/uproot/_dask.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,11 +142,6 @@ def dask(
Options (type; default):

* handler (:doc:`uproot.source.chunk.Source` class; None)
* file_handler (:doc:`uproot.source.chunk.Source` class; None) (Deprecated: Use `handler` instead. If set, this will take precedence over `handler`)
* xrootd_handler (:doc:`uproot.source.chunk.Source` class; None) (Deprecated: Use `handler` instead. If set, this will take precedence over `handler`)
* s3_handler (:doc:`uproot.source.chunk.Source` class; None) (Deprecated: Use `handler` instead. If set, this will take precedence over `handler`)
* http_handler (:doc:`uproot.source.chunk.Source` class; None) (Deprecated: Use `handler` instead. If set, this will take precedence over `handler`)
* object_handler (:doc:`uproot.source.chunk.Source` class; None) (Deprecated: Use `handler` instead. If set, this will take precedence over `handler`)
* timeout (float for HTTP, int for XRootD; 30)
* max_num_elements (None or int; None)
* num_workers (int; 1)
Expand Down
Loading