Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NSFS | S3 throwing error for empty header and default port for STS #8579

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

NSFS | S3 throwing error for empty header and default port for STS #8579

wants to merge 1 commit into from

Conversation

naveenpaul1
Copy link
Contributor

@naveenpaul1 naveenpaul1 commented Dec 5, 2024 •
edited
Loading

Explain the changes

  1. Added check for header before spliting the host url
  2. set default port for STS only if its not NC

Issues: Fixed #xxx / Gap #xxx

  1. core.endpoint.sts.sts_rest:: STS ERROR , anonymous account and log4j error messages in s3_rest.js  #8509

Testing Instructions:

  1. send s3 request without header/body and verify the logs
  • Doc added/updated
  • Tests added

@naveenpaul1 naveenpaul1 requested a review from romayalon December 5, 2024 06:50
romayalon
romayalon reviewed Dec 5, 2024
View reviewed changes
src/endpoint/s3/s3_rest.js
@@ -334,7 +334,7 @@ function _get_arn_from_req_path(req) {
// we will reintreduce it together with bucket site support.
function parse_bucket_and_key(req) {
const { url, headers, virtual_hosts } = req;
const host = headers.host.split(':')[0]; // cutting off port
const host = headers?.host?.split(':')[0]; // cutting off port

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 do we need to throw here an error about this missing host header?
how AWS react to this missing host header?

Copy link
Contributor Author

@naveenpaul1 naveenpaul1 Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I am not sure how to send the request to AWS s3 without host header, In doc I could see host header is required property for AWS S3. doc
For testing this fix I have to remove the host header inside the code, with curl and httpie are setting minimum header.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this point is still open, if you decide on something could you please update it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I sent https request to AWS S3 url {bucket-name}.s3.amazonaws.com without any header and got AccessDenied error. Do we need to return the same error if header is missing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 if you attach the host header, do you have access to the bucket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I send the request from JS, to make sure no headers.

        const res = await http_utils.make_https_request({
            hostname: '{bucket-name}.s3.amazonaws.com',
            method: 'GET'
        });

Not 100% sure this will send request without any default header, since we cant check in AWS S3 side. It dont have any access credentials.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 we print headers on our logs, try this code directed to NooBaa and check the logs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon tried the same even this request have header info
Anonymous request: GET / { host: 'localhost:6443', connection: 'close' }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

curl 127.0.0.1:6001/ --http1.0 -H 'Host:'
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/</Resource><RequestId>m4if1b6o-dvqhe-12x3</RequestId></Error>%

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I tried with AWS S3 url and its I didnt get any response for the request.

root:noobaa-core# curl https://{bucket-name}.s3.amazonaws.com --http1.0 -H 'Host:'
root:noobaa-core#

romayalon
romayalon reviewed Dec 5, 2024
View reviewed changes
src/endpoint/endpoint.js Outdated Show resolved Hide resolved
@naveenpaul1 naveenpaul1 linked an issue Dec 6, 2024 that may be closed by this pull request
core.endpoint.sts.sts_rest:: STS ERROR , anonymous account and log4j error messages in s3_rest.js #8509
Open
shirady
shirady reviewed Dec 10, 2024
View reviewed changes
Copy link
Contributor

@shirady shirady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few comments

config.js
@@ -896,7 +896,7 @@ config.NSFS_NC_CONFIG_DIR_BACKEND = '';
config.NSFS_NC_STORAGE_BACKEND = '';
config.ENDPOINT_PORT = Number(process.env.ENDPOINT_PORT) || 6001;
config.ENDPOINT_SSL_PORT = Number(process.env.ENDPOINT_SSL_PORT) || 6443;
config.ENDPOINT_SSL_STS_PORT = Number(process.env.ENDPOINT_SSL_STS_PORT) || -1;
config.ENDPOINT_SSL_STS_PORT = Number(process.env.ENDPOINT_SSL_STS_PORT) || (process.env.NC_NSFS_NO_DB_ENV === 'true' ? -1 : 7443);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just add a comment for us that when we will support STS in NSFS we can remove this condition.

src/endpoint/endpoint.js
const https_port_iam = options.https_port_iam || config.ENDPOINT_SSL_IAM_PORT;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can add back the newline, I think we wanted a separation between the port's definitions and the calls to the functions.

src/endpoint/s3/s3_rest.js
@@ -112,7 +112,7 @@ async function handle_request(req, res) {
http_utils.set_cors_headers_s3(req, res, cors);

if (req.method === 'OPTIONS') {
dbg.log1('OPTIONS!');
dbg.log1('s3_rest : S3 request method is ', req.method);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting to add the function name in the logs

Suggested change
dbg.log1('s3_rest : S3 request method is ', req.method);
dbg.log1('s3_rest handle_request : S3 request method is ', req.method);

src/endpoint/s3/s3_rest.js
@@ -334,7 +334,7 @@ function _get_arn_from_req_path(req) {
// we will reintreduce it together with bucket site support.
function parse_bucket_and_key(req) {
const { url, headers, virtual_hosts } = req;
const host = headers.host.split(':')[0]; // cutting off port
const host = headers?.host?.split(':')[0]; // cutting off port

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this point is still open, if you decide on something could you please update it here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shirady shirady shirady left review comments

@romayalon romayalon romayalon left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
size/XS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

core.endpoint.sts.sts_rest:: STS ERROR , anonymous account and log4j error messages in s3_rest.js
3 participants
@naveenpaul1 @romayalon @shirady