Should the adapter be able to connect to a remote data store?

[slobentanzer]

It is within the realm of possibilities that the adapter solution we provide will be used by the community in the compute environment of Open Targets (Google cloud). Thus, we may benefit from an adapter principle (data ingestion package) that can connect to a cloud storage location of the Open Targets pipeline data as opposed to downloading it and processing locally. Streaming data from the original cloud location could bring nice performance improvements if we deploy in this way. Regarding the choice of package for ingesting .parquet, this could be a consideration.

[kpto]

Both duckdb and Polars support HTTPS, S3 and Google Cloud
duckdb supports also Cloudflare R2 while Polars supports also Azure

References:
duckdb: https://duckdb.org/docs/guides/network_cloud_storage/overview
Polars: https://docs.pola.rs/api/python/dev/reference/api/polars.read_parquet.html

Given that a lot of data science packages are designed with cloud storage in mind, this shouldn't be difficult.