Unable to Upload to S3 via IAM Role in Hue with has_iam_detection=true – Forbidden Error

[pbuaradhana]

I am experiencing an issue with Hue’s integration with AWS IAM roles for accessing S3. I have configured the hue.ini file to detect and use the IAM role assigned to the service account in Kubernetes, but I receive a 403 Forbidden error specifically when attempting to upload a file to S3 via the Hue file browser.

Steps to Reproduce:

1. Configure hue.ini as follows to use IAM role detection:

[aws]
[[aws_accounts]]
[[[default]]]
has_iam_detection=true

2. Annotate the Kubernetes service account with the necessary roleArn for AWS IAM role usage.
3. Confirm the role has s3:PutObject, s3:GetObject, and other necessary S3 permissions. (I verified this by successfully using the role for similar S3 operations in another application.)
4. Open the Hue file browser, navigate to S3, and attempt to upload a file.

Observed Behavior:

While read and list operations work as expected with has_iam_detection=true, the file upload fails with a 403 Forbidden error:

aws.s3.s3fs.S3FileSystemException: Failed to access path "": User is not authorized to access path "".

When I provide AWS access keys and secrets directly in the hue.ini configuration, the upload works without issue. This suggests a potential difference in how Boto (or another underlying component) is configured to handle IAM role-based access compared to direct access keys.

Additional Details:

Role Verification: The IAM role used has been validated to have the necessary permissions (including s3:PutObject) in another application, and write operations work fine there.
Environment: Hue is running within a Kubernetes cluster, with the service account annotated with the IAM role.
Boto Library: Hue appears to be using Boto 2.x for S3 operations. This could be relevant since Boto 2.x and Boto3 handle IAM role-based access differently.
Hue Version: 4.11

Possible Areas to Investigate:

Boto IAM Role Support: Is there a known limitation with Boto 2.x for IAM role usage in Hue? If so, are there any recommended workarounds?
Configuration Differences: Are there any additional configuration steps needed to ensure full IAM role compatibility for upload operations in Hue?

Expected Behavior:

When using has_iam_detection=true, Hue should detect and utilize the IAM role for all S3 operations, including file uploads, without requiring access keys in the configuration file.

Logs:

Here are the relevant log entries from an upload attempt:

[06/Nov/2024 00:48:02 -0800] decorators ERROR Error running guess_format
...
aws.s3.s3fs.S3FileSystemException: Failed to access path "s3a://myBucket/...": User is not authorized to access path at "s3a://myBucket/...".

Request:

Could the maintainers provide insights or guidance on whether this is a known issue or if there are specific configurations needed to fully support IAM roles with has_iam_detection=true for S3 uploads? If there are any recommended versions of Boto or specific patches for this use case, please advise.
cc: @romainr @bjornalm @Harshg999 @agl29