Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: support aliyun oss backend. #216

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

divinerapier
Copy link

@divinerapier divinerapier commented Nov 22, 2024

Overview

This PR adds support for using Aliyun Object Storage Service (OSS) as a backend storage option.

Changes

  • Implement OSS storage adapter
  • Add configuration options for OSS credentials and endpoint

@divinerapier divinerapier force-pushed the feat-oss branch 2 times, most recently from abd9610 to 6a14f50 Compare November 22, 2024 10:33
catalog/catalog.go Show resolved Hide resolved
catalog/catalog.go Outdated Show resolved Hide resolved
io/oss.go Outdated Show resolved Hide resolved
@divinerapier
Copy link
Author

@zeroshade Fixed, PTAL

@zeroshade
Copy link
Member

This seems generally good to me. Does Aliyun have something similar to how MinIO works for S3 that can be added to the integration tests to have CI testing the backend? i.e. is there any way to use a docker image to run something that simulates the Aliyun backend for testing?

@divinerapier
Copy link
Author

@zeroshade

Automated integration testing is indeed crucial. However, unfortunately, I haven't yet found a reasonable and convenient way to access OSS services in the integration testing environment.

I've tried the following approaches:

  • Contacted Aliyun's customer service to inquire about recommended solutions. They suggested using OSS cloud services directly, but this raises questions about who would provide the account and access permissions.
  • Attempted to simulate OSS using MinIO, but unfortunately, whether using v1 or v4 signatures, it always results in 400 errors. We suspect that OSS's signature scheme is incompatible with MinIO's.
  • MinIO previously offered a project called minio-gateway-oss, but it was deprecated in 2022.

Therefore, regarding OSS integration testing implementation, I haven't been able to identify other viable solutions.

Since this modification is being used in our company's actual project, the good news is that we haven't encountered any issues in production so far.

@zeroshade
Copy link
Member

the Java iceberg impl has some mocking and test setups for Aliyun as seen here would it make sense to create a similar utility in Go? or is it not worth that effort there?

Is there any sort of testing we could add here for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants