Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Side effect from region override to credentials[["region"]] #419

Open
yilun11 opened this issue Jun 9, 2022 · 1 comment
Open

Side effect from region override to credentials[["region"]] #419

yilun11 opened this issue Jun 9, 2022 · 1 comment

Comments

@yilun11
Copy link

yilun11 commented Jun 9, 2022

This is related to #340 since the fix is not ideal and has side effects. I'd like opinions on best way to resolve. I have AWS_S3_ENDPOINT environment variable set and region set in my .aws/credentials file. Even if region isn't set, aws.signature returns us-east-1 by default.

The fix is line 62 from s3HTTP.R:

# allow region="" to override any config - the only way to use 3rd party URLs without region
region <- if (length(region) && !nzchar(region)) region else credentials[["region"]]

Before the fix, region is NULL but after, it is set to credentials[["region"]]. Later, in setup_s3_url(), this change prepends region to my AWS_S3_ENDPOINT, which is not what I want and is a fatal error. See line 282-292:

# handle S3-compatible storage URLs
if (base_url != "s3.amazonaws.com") {
    if (isTRUE(verbose)) {
        message("Non-AWS base URL requested.")
    }
    accelerate <- FALSE
    dualstack <- FALSE
    if (!is.null(region) && region != "") {
         base_url <- paste0(region, ".", base_url)
    }
}

What are recommendations to resolve? Only solution I have so far is to fork code and take out line 62.

@yilun11
Copy link
Author

yilun11 commented Jun 9, 2022

I just saw #371 and #363, which seem to be related, and the suggestion to try arrow. This issue can be closed unless its helpful for others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant