Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NSFS | S3 throwing error for empty header and default port for STS #8579

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

NSFS | S3 throwing error for empty header and default port for STS #8579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -896,7 +896,7 @@ config.NSFS_NC_CONFIG_DIR_BACKEND = '';
config.NSFS_NC_STORAGE_BACKEND = '';
config.ENDPOINT_PORT = Number(process.env.ENDPOINT_PORT) || 6001;
config.ENDPOINT_SSL_PORT = Number(process.env.ENDPOINT_SSL_PORT) || 6443;
config.ENDPOINT_SSL_STS_PORT = Number(process.env.ENDPOINT_SSL_STS_PORT) || -1;
config.ENDPOINT_SSL_STS_PORT = Number(process.env.ENDPOINT_SSL_STS_PORT) || (process.env.NC_NSFS_NO_DB_ENV === 'true' ? -1 : 7443);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just add a comment for us that when we will support STS in NSFS we can remove this condition.

config.ENDPOINT_SSL_IAM_PORT = Number(process.env.ENDPOINT_SSL_IAM_PORT) || -1;
config.ALLOW_HTTP = false;
// config files should allow access to the owner of the files
Expand Down
3 changes: 1 addition & 2 deletions src/endpoint/endpoint.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,8 @@ async function main(options = {}) {
// START S3, STS & IAM SERVERS & CERTS
const http_port_s3 = options.http_port || config.ENDPOINT_PORT;
const https_port_s3 = options.https_port || config.ENDPOINT_SSL_PORT;
const https_port_sts = options.https_port_sts || Number(process.env.ENDPOINT_SSL_PORT_STS) || 7443; // || (process.env.NC_NSFS_NO_DB_ENV === 'true' ? -1 : 7443);
const https_port_sts = options.https_port_sts || config.ENDPOINT_SSL_STS_PORT;
const https_port_iam = options.https_port_iam || config.ENDPOINT_SSL_IAM_PORT;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can add back the newline, I think we wanted a separation between the port's definitions and the calls to the functions.

await start_server_and_cert(SERVICES_TYPES_ENUM.S3, init_request_sdk,
{ ...options, https_port: https_port_s3, http_port: http_port_s3, virtual_hosts, bucket_logger, notification_logger });
await start_server_and_cert(SERVICES_TYPES_ENUM.STS, init_request_sdk, { https_port: https_port_sts, virtual_hosts });
Expand Down
4 changes: 2 additions & 2 deletions src/endpoint/s3/s3_rest.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ async function handle_request(req, res) {
http_utils.set_cors_headers_s3(req, res, cors);

if (req.method === 'OPTIONS') {
dbg.log1('OPTIONS!');
dbg.log1('s3_rest : S3 request method is ', req.method);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting to add the function name in the logs

Suggested change
dbg.log1('s3_rest : S3 request method is ', req.method);
dbg.log1('s3_rest handle_request : S3 request method is ', req.method);

const error_code = req.headers.origin && req.headers['access-control-request-method'] ? 403 : 400;
const res_headers = res.getHeaders(); // We will check if we found a matching rule - if no we will return error_code
res.statusCode = res_headers['access-control-allow-origin'] && res_headers['access-control-allow-methods'] ? 200 : error_code;
Expand Down Expand Up @@ -334,7 +334,7 @@ function _get_arn_from_req_path(req) {
// we will reintreduce it together with bucket site support.
function parse_bucket_and_key(req) {
const { url, headers, virtual_hosts } = req;
const host = headers.host.split(':')[0]; // cutting off port
const host = headers?.host?.split(':')[0]; // cutting off port

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 do we need to throw here an error about this missing host header?
how AWS react to this missing host header?

Copy link
Contributor Author

@naveenpaul1 naveenpaul1 Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I am not sure how to send the request to AWS s3 without host header, In doc I could see host header is required property for AWS S3. doc
For testing this fix I have to remove the host header inside the code, with curl and httpie are setting minimum header.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this point is still open, if you decide on something could you please update it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I sent https request to AWS S3 url {bucket-name}.s3.amazonaws.com without any header and got AccessDenied error. Do we need to return the same error if header is missing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 if you attach the host header, do you have access to the bucket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I send the request from JS, to make sure no headers.

        const res = await http_utils.make_https_request({
            hostname: '{bucket-name}.s3.amazonaws.com',
            method: 'GET'
        });

Not 100% sure this will send request without any default header, since we cant check in AWS S3 side. It dont have any access credentials.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@naveenpaul1 we print headers on our logs, try this code directed to NooBaa and check the logs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon tried the same even this request have header info
Anonymous request: GET / { host: 'localhost:6443', connection: 'close' }

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

curl 127.0.0.1:6001/ --http1.0 -H 'Host:'
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/</Resource><RequestId>m4if1b6o-dvqhe-12x3</RequestId></Error>%

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romayalon I tried with AWS S3 url and its I didnt get any response for the request.

root:noobaa-core# curl https://{bucket-name}.s3.amazonaws.com --http1.0 -H 'Host:'
root:noobaa-core#

let virtual_host = null;
if (host && host !== 'localhost' && !net.isIP(host)) {
Expand Down
Loading