Unable to append to iceberg table with Polaris Open Catalog

[sfc-gh-adlee]

Describe the bug

Unable to update a table with the write_iceberg method. Received an error AWS Error UNKNOWN (HTTP status 301) during CreateMultipartUpload operation: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

tried to do a written_df = df_write.write_iceberg(table, mode="append", io_config=[daft.io](http://daft.io/).IOConfig(s3=[daft.io](http://daft.io/).S3Config(region_name="us-west-2"))) but this didnt work

To Reproduce

from pyiceberg.catalog import load_catalog
import pyarrow.parquet as pq
catalog = load_catalog('demo_catalog',
**{'uri': 'https://xxxxxxxxx.snowflakecomputing.com/polaris/api/catalog',
'warehouse': 'demo_catalog',
'credential': 'sasasasas',
'scope':'PRINCIPAL_ROLE:ALL',
'client.region':'us-west-2'})
table = catalog.load_table("taxi.taxi_dataset")
df = daft.read_iceberg(table)

df_write = daft.sql("select * from df limit 1")
written_df = df_write.write_iceberg(df, mode="append") --> throws error unspecified endpoint

Tried

written_df = df_write.write_iceberg(table, mode="append", io_config=daft.io.IOConfig(s3=daft.io.S3Config(region_name="us-west-2"))) --> TypeError: got an unexpected keyword argument 'io_config'

Expected behavior

Supposed to append the new rows to the dataframe

Component(s)

Python Runner

Additional context

No response

[raunakab]

Hey @sfc-gh-adlee, taking a look into this now. Will get back to you asap.

[raunakab]

Oh looks like @jaychia and you were talking on our open Slack channel. Let me follow up with him.