Minio s3 support

[gesaleh]

I have installed gravitino to support mysql, postgres and other database i want to connect to s3 also using the icebeg connection but it looks like empty where i'm doing it wong

# THE CONFIGURATION FOR AUXILIARY SERVICE
# Auxiliary service names, separate by ','
gravitino.auxService.names = iceberg-rest
# Iceberg REST service classpath
gravitino.auxService.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf
# Iceberg REST service host
gravitino.auxService.iceberg-rest.host = 0.0.0.0
# Iceberg REST service http port
gravitino.auxService.iceberg-rest.httpPort = 9001
gravitino.auxService.iceberg-rest.catalog-backend = jdbc
gravitino.auxService.iceberg-rest.uri = jdbc:mysql://mysql:3306/db
gravitino.auxService.iceberg-rest.warehouse = hdfs://hive:9000/user/iceberg/warehouse/
gravitino.auxService.iceberg-rest.jdbc.user = mysql
gravitino.auxService.iceberg-rest.jdbc.password = mysql
gravitino.auxService.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.credential-provider-type = s3-secret-key
gravitino.iceberg-rest.s3-access-key-id = xxxxxxxxxxxxxxx
gravitino.iceberg-rest.s3-secret-access-key = xxxxxxxxxxxxyyyyyyyyyyyzzzzzzzzzz
gravitino.iceberg-rest.s3-endpoint = minio:9000

how i can test and see schema of bucket of files in my s3 ?

[gesaleh]

to catalog s3 should i use iceberg or hadoop fileset ? both i see no metadata from the s3 local (minio)

[FANNG1]

I'm a little confused about your scenarios, could you provide more context?
noticed you add configuration for Gravitino Iceberg REST server which could show Iceberg data stored in S3, gravitino.auxService.iceberg-rest.warehouse should start with s3 schema like s3://xx and the iceberg table metadata should stored in your MySQL tables (Iceberg JDBC catalog backend). You could use spark, flink, trino to explore Iceberg data, please refer to https://gravitino.apache.org/docs/0.7.0-incubating/iceberg-rest-service
I you want to explore s3 raw files, suggest using Hadoop fileset which manages a mapping for the logical fileset to physical S3 location. Please refer to https://gravitino.apache.org/docs/0.7.0-incubating/how-to-use-gvfs.

[gesaleh]

thanks for the clarifications
so yes what i need is the fileset option as you mention to explore s3 raw files, suggest using Hadoop fileset which manages a mapping for the logical fileset to physical S3 location.
i did the configuration with th ui for haddop I'm using the playground repo, so hadoop is not part of it ,
if my understanding is correct, to have the fileset part of the metalake i need hadoop, and conifurations as mentions in the url,

in the ui of gravitino there is an option to create a new catalog type fileset (apache hadoop) i added the following option

location
s3-endpoint
s3-access-key-id 
s3-secret-access-key 
filesystem-providers

this doesn't connect to s3 ? fileset
I also saw that one jar file need to be downloaded is it the case also for the playground ?

i used this url https://gravitino.apache.org/docs/0.7.0-incubating/manage-fileset-metadata-using-gravitino/

# create a S3 catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "FILESET",
  "comment": "comment",
  "provider": "hadoop",
  "properties": {
    "location": "s3a://bucket/root",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key",
    "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "filesystem-providers": "s3"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

[yuqi1129]

Yeah, if you encounter any issues, please provide detailed information like the stack trace here.

[gesaleh]

Hi @yuqi1129 is it possbile to have access and events without moving files around, let me explain if i have datalake that is based on s3 minio and other paths that are not in the s3 can gravitino, trino manage these too ? how to scrap the files using gvfs ? how to listen to add,update delete ?
what happened if a file is replaced or updated ?

[gesaleh]

hello
I have deployed gravitino playground along side with minio
I tried to perfrom the follwoing
in my minio i have the bucket work

# create a S3 catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "FILESET",
  "comment": "comment",
  "provider": "hadoop",
  "properties": {
    "location": "s3a://work/",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key",
    "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "filesystem-providers": "s3"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs 

this works but when i try to create scheme it just hangs

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_schema",
  "comment": "comment",
  "properties": {
    "location": "s3a://work/schemaS3"
  }
}' http://localhost:8090/api/metalakes/metalake_demo/catalogs/catalog_s3/schemas 

I get dead lock probably I'm doing wrong i'm suspecting the location value

2025-01-07 15:00:56 2025-01-07 14:00:56.133 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=metalake_demo.catalog_s3}=1736258146570, ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-53,5,main], ident=metalake_demo.catalog_s3}=1736258145174, ThreadIdentifier{thread=Thread[Gravitino-webserver-71,5,main], ident=metalake_demo.catalog_s3}=1736258388375, ThreadIdentifier{thread=Thread[Gravitino-webserver-52,5,main], ident=metalake_demo.catalog_s3}=1736258343241, ThreadIdentifier{thread=Thread[Gravitino-webserver-59,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-60,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-68,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3}=1736258336696, ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3}=1736258343242} 
2025-01-07 15:00:56 2025-01-07 14:00:56.133 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=metalake_demo.catalog_s3}=1736258146570, ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-53,5,main], ident=metalake_demo.catalog_s3}=1736258145174, ThreadIdentifier{thread=Thread[Gravitino-webserver-71,5,main], ident=metalake_demo.catalog_s3}=1736258388375, ThreadIdentifier{thread=Thread[Gravitino-webserver-52,5,main], ident=metalake_demo.catalog_s3}=1736258343241, ThreadIdentifier{thread=Thread[Gravitino-webserver-59,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-60,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-68,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3}=1736258336696, ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3}=1736258343242} 

i added the jar file aws-bundle-0.7.0-incubating.jar to the root/gravitino/catalogs/hadoop/libs i didn't change any value in the conf files

is there a need ? why it is going to deadlock ?

[gesaleh]

I also saw this

2025-01-07 13:55:45.209 INFO [Gravitino-webserver-53] [org.apache.gravitino.gcp.shaded.com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.<clinit>(GoogleHadoopFileSystemBase.java:611)] - GHFS version: 1.9.4-hadoop3
2025-01-07 13:55:45.308 INFO [Gravitino-webserver-53] [org.apache.commons.beanutils.FluentPropertyBeanIntrospector.introspect(FluentPropertyBeanIntrospector.java:147)] - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
2025-01-07 13:55:45.313 WARN [Gravitino-webserver-53] [org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:134)] - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties

root@7c2e4f9a4c6e:~/gravitino/conf# curl -I http://minio:9000//minio/health/live
HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 261
Content-Type: application/xml
Server: MinIO
Vary: Origin
Date: Tue, 07 Jan 2025 17:26:33 GMT

root@7c2e4f9a4c6e:~/gravitino/conf# curl -I minio:9000//minio/health/live
HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 261
Content-Type: application/xml
Server: MinIO
Vary: Origin
Date: Tue, 07 Jan 2025 17:26:40 GMT

[yuqi1129]

Please see #6135 (comment)