Getting the infamous (98: Address already in use) error #3063

walidabualafia · 2023-09-21T15:40:40Z

Hi, folks,

I am currently in the works of deploying an OnDemand instance at my institution. I got OnDemand installed on a VM host, and configured authentication with our Azure AD. Now, whenever I browse the webpage, AD prompts my credentials, I sign in, auth succeeds, and then I see this:

Error -- nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()

Now I am certain that the issue is with the name mapping, but I'm struggling to find the fix. I have tried several approaches from Discourse and issues here in GitHub, but could not find anything to fix my issue.

I turned on Lua debug to see what the username being mapped is, and I got the following:

[Thu Sep 21 10:23:39.625087 2023] [lua:debug] [pid 2100128:tid 139873696593664] @/opt/ood/mod_ood_proxy/lib/ood/user_map.lua(14): [client 10.42.59.64:52676] Mapped '[email protected]' => '[email protected]' [80.185 ms], referer: https://adfs.company.org/

I tried stripping the username using regex in ood_portal.yml using variable user_map_match = '([^@]+)', but with no success.

Any help would be greatly appreciated. :)

Cheers,
Walid

The text was updated successfully, but these errors were encountered:

johrstrom · 2023-09-21T15:44:00Z

Can you try something more like this?

user_map_match: '^([^@]+)@company.org$'

walidabualafia · 2023-09-21T16:03:12Z

I added this change, and now it's not mapping my user. I'm getting this back:

Error -- failed to map user ([email protected])

FYI, I'm running RHEL 8.8 (Ootpa).

johrstrom · 2023-09-21T18:27:53Z

The linux user is [email protected]? I think the @ and . characters are likely throwing you off.

walidabualafia · 2023-09-21T20:17:48Z

The Linux user is wabuala. If I run ls /var/run/ondemand-nginx I get wabuala. If I run ls /var/run/ondemand-nginx/wabuala I get passenger.pid passenger.sock.

I believe the username should just be the name without the @company.org extension. When I don't perform regex ops, or try user_map_match = '([^@]+)', I get the original error

Error -- nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/ondemand-nginx/wabuala/passenger.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()

Do you think it might be related to something other than username mapping?

I'd be happy to clean up and post any logs as well, if that helps.

walidabualafia · 2023-09-21T20:32:40Z

Sorry for the spam, thought this could be helpful. I ran irb in /opt/ood/nginx_stage to see what user.name resolves to. This is the output:

[wabuala@svltondemand01 nginx_stage]$ irb
irb(main):001:0> require_relative 'lib/nginx_stage/user'
=> true
irb(main):002:0> require 'etc'
=> true
irb(main):003:0> u = NginxStage::User.new('wabuala')
=> #<NginxStage::User:0x000055cc8e9db178 @passwd=#<struct Etc::Passwd name="wabuala", passwd="*", ... >
irb(main):004:0> u.name
=> "wabuala"
irb(main):005:0> u = NginxStage::User.new('[email protected]')
=> #<NginxStage::User:0x000055cc8e8330f0 @passwd=#<struct Etc::Passwd name="wabuala", passwd="*", ... >
irb(main):006:0> u.name
=> "wabuala"
irb(main):007:0> n = NginxStage::User.new('[email protected]')
=> #<NginxStage::User:0x000055cc8e9f2378 @passwd=#<struct Etc::Passwd name="wabualaadmin", passwd="*", ... >
irb(main):008:0> n.name
=> "wabualaadmin"
irb(main):009:0> exit()

johrstrom · 2023-09-21T20:36:23Z

Got it - I see that regex should work.

Lua 5.3.6  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> string.match('[email protected]', '([^@]+)')
jeff

So from what I can recall from the other topics (infamous is a bit right!) there's an issue somewhere in your SSSD stack.

You're able to boot the PUN, but somehow, you're unable to recognize that the file belongs to the wabuala user. Are you running SELinux?

That siad - this line is concerning

Mapped '[email protected]' => '[email protected]' [80.185 ms], referer: https://adfs.company.org/

It should read

Mapped '[email protected]' => 'wabuala' [80.185 ms], referer: https://adfs.company.org/

that's not spam at all - that's a real clue!

You have 2 different usernames mapping to the same user?!

irb(main):003:0> u = NginxStage::User.new('wabuala')
=> #<NginxStage::User:0x000055cc8e9db178 @passwd=#<struct Etc::Passwd name="wabuala", passwd="*", ... >
irb(main):004:0> u.name
=> "wabuala"
irb(main):005:0> u = NginxStage::User.new('[email protected]')
=> #<NginxStage::User:0x000055cc8e8330f0 @passwd=#<struct Etc::Passwd name="wabuala", passwd="*", ... >

What's the output of id wabuala and id [email protected]? They can't have the same UID or be the same user right?

Mappings need to be unique - either [email protected] the email maps to wabuala or it maps to [email protected] but not both.

johrstrom · 2023-09-21T20:45:48Z

Looking at this again - you should not be able to do this at all. What version of OnDemand are you running?

[email protected] and [email protected] should both be hitting this error.

ondemand/nginx_stage/lib/nginx_stage/user.rb

Lines 40 to 50 in 04ee5fb

    
                 if name.to_s != user.to_s 
        
                   err_msg = <<~HEREDOC 
        
                     Username '#{user}' is being mapped to '#{name}' in SSSD and they don't match. 
        
                     Users with domain names cannot be mapped correctly. If '#{name}' still has the 
        
                     domain in it you'll need to set SSSD's full_name_format to '%1$s'. 
        
                     See https://github.com/OSC/ondemand/issues/1759 for more details. 
        
                   HEREDOC 
        
                   raise StandardError, err_msg 
        
                 end

walidabualafia · 2023-09-21T21:13:00Z

I'm running ondemand-1.8.20-1.el8.x86_64. I got it from RHEL8 AppStreams.

Your concern is in its place. Both wabuala and [email protected] map to the same user. Both share uid=96916.

I'm not running SELinux. Could that be the reason? I can go ahead and add SELinux if necessary.

Also, you're right in that it's supposed to read

Mapped '[email protected]' => 'wabuala' [80.185 ms], referer: https://adfs.company.org/

I'm not sure why it's mapping to the full email.

johrstrom · 2023-09-21T21:26:08Z

I'm running ondemand-1.8.20-1.el8.x86_64.

That's why you don't see that message! Well I think that message is directly applicable to you and if you were running 2.0 or higher you'd see it. I would try to set SSSD's config full_name_format '%1$s' and see if that resolves.

I got it from RHEL8 AppStreams.

We host on yum.osc.edu I'd be interested to know if you installed this package from some other yum repo.

I'm not running SELinux. Could that be the reason? I can go ahead and add SELinux if necessary.

No I think your issue is definitly the same as the issue referenced in the error message.

I'm not sure why it's mapping to the full email.

Let me check the documentation for the same. What I said there in lua code is 2.0+ and user_map_match is a 2.0+ thing too so I guess I'd ask if you're actually using user_map_cmd? (that would explain why it takes so long. 80 ms seemed high to me).

user_map_cmd: "/opt/ood/ood_auth_map/bin/ood_auth_map.regex --regex='^(\\w+)@example.com'"

walidabualafia · 2023-09-21T21:42:14Z

It worked! Thank you, Jeff!

I was using user_map_cmd earlier. It only worked when I added both the regex in the user_map_cmd and the full_name_format in sssd.conf.

We host on yum.osc.edu I'd be interested to know if you installed this package from some other yum repo.

I had to get it through my institution's internal repo satellite. I believe it is slightly out of date. Is there any documentation for unpacking/installing tarballs?

walidabualafia · 2023-09-21T22:00:11Z

Completely unrelated error, and I can move it to another issue if necessary. I'm trying to spawn a shell now, and I'm getting this error:

Warning: unable to write to /tmp/passenger.spawn.XXXX01ieaF/response/steps/subprocess_exec_wrapper/end_time: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1695333010.76)
Warning: unable to write to /tmp/passenger.spawn.XXXX01ieaF/response/steps/subprocess_wrapper_preparation/begin_time: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1695333010.763)
Warning: unable to write to /tmp/passenger.spawn.XXXX01ieaF/response/steps/subprocess_wrapper_preparation/begin_time: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1695333010.784)
Warning: unable to write to /tmp/passenger.spawn.XXXX01ieaF/response/steps/subprocess_wrapper_preparation/end_time: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1695333010.785)
Warning: unable to write to /tmp/passenger.spawn.XXXX01ieaF/response/steps/subprocess_app_load_or_exec/begin_time: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1695333010.785)
innerError Error: Cannot find module '../build/Debug/pty.node'
Require stack:
- /var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js
- /var/www/ood/apps/sys/shell/node_modules/node-pty/lib/index.js
- /var/www/ood/apps/sys/shell/app.js
- /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/node-loader.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:931:15)
    at Function.Module._load (internal/modules/cjs/loader.js:774:27)
    at Module.require (internal/modules/cjs/loader.js:1003:19)
    at Module.require (/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/node-loader.js:80:25)
    at require (internal/modules/cjs/helpers.js:107:18)
    at Object.<anonymous> (/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:30:15)
    at Module._compile (internal/modules/cjs/loader.js:1114:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:979:32)
    at Function.Module._load (internal/modules/cjs/loader.js:819:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js',
    '/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/index.js',
    '/var/www/ood/apps/sys/shell/app.js',
    '/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/node-loader.js'
  ]
}
/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:35
        throw outerError;
        ^

Error: The module '/var/www/ood/apps/sys/shell/node_modules/node-pty/build/Release/pty.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 64. This version of Node.js requires
NODE_MODULE_VERSION 83. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1173:18)
    at Module.load (internal/modules/cjs/loader.js:979:32)
    at Function.Module._load (internal/modules/cjs/loader.js:819:12)
    at Module.require (internal/modules/cjs/loader.js:1003:19)
    at Module.require (/opt/ood/ondemand/root/usr/share/passenger/helper-scripts/node-loader.js:80:25)
    at require (internal/modules/cjs/helpers.js:107:18)
    at Object.<anonymous> (/var/www/ood/apps/sys/shell/node_modules/node-pty/lib/unixTerminal.js:26:11)
    at Module._compile (internal/modules/cjs/loader.js:1114:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:979:32) {
  code: 'ERR_DLOPEN_FAILED'
}

Seems like a whole can of worms to npm rebuild all the node modules. Do you have any suggestions for this? A quick run of npm --version shows 6.14.18.

johrstrom · 2023-09-22T13:43:37Z

Seems like a whole can of worms to npm rebuild all the node modules. Do you have any suggestions for this? A quick run of npm --version shows 6.14.18.

That is an extremely old version of nodejs. I want to say 1.8 shipped with node 12, but would have to confirm - the RPM should have resolved the correct nodejs.

I had to get it through my institution's internal repo satellite. I believe it is slightly out of date. Is there any documentation for unpacking/installing tarballs?

No - IDK how satellites work. Seems like it should just mirror yum.osc.edu or you could upload RPMs manually I guess?

treydock · 2023-09-22T19:54:31Z

Satellite , at least modern versions like 6.x, can sync any yum repo. OSC Satellite syncs both RedHat repos as well as EPEL, but if we wanted we could add other repos like OnDemand, etc. In Satellite you add a new product and point at the desired path for repo that contains the repodata directory, so like https://yum.osc.edu/ondemand/3.0/web/el8/x86_64/

johrstrom · 2023-10-11T16:44:44Z

I feel like this ticket is resolved (at least the initial issue), so i'm going to close this. @walidabualafia if you continue to have issues, just open more issues here or on discourse.

osc-bot added this to the Backlog milestone Sep 21, 2023

johrstrom closed this as completed Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting the infamous (98: Address already in use) error #3063

Getting the infamous (98: Address already in use) error #3063

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023 •

edited

Loading

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 22, 2023

treydock commented Sep 22, 2023

johrstrom commented Oct 11, 2023

Getting the infamous (98: Address already in use) error #3063

Getting the infamous (98: Address already in use) error #3063

Comments

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023 • edited Loading

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

walidabualafia commented Sep 21, 2023

johrstrom commented Sep 22, 2023

treydock commented Sep 22, 2023

johrstrom commented Oct 11, 2023

walidabualafia commented Sep 21, 2023 •

edited

Loading