Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recreate listener if error is occured #236

Merged
merged 3 commits into from
Jan 10, 2025

Conversation

clyang82
Copy link
Contributor

@clyang82 clyang82 commented Dec 18, 2024

recreate the listener when the notify is nil or there is an error for Ping()
here is the document for notify is nil

A notification channel will remain open until Unlisten is called, though connection loss might
result in some notifications being lost. To solve this problem, Listener sends
a nil pointer over the Notify channel any time the connection is re-established
following a connection loss. The application can get information about the
state of the underlying connection by setting an event callback in the call to
NewListener.

@clyang82 clyang82 changed the title [WIP] Recreate listener if error is occured Recreate listener if error is occured Jan 3, 2025
func (f *Default) NewListener(ctx context.Context, channel string, callback func(id string)) *pq.Listener {
logger := ocmlogger.NewOCMLogger(ctx)

listener := newListener(ctx, f.config, channel)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about new the listener in the goroutine waitForNotification? we can maintain the listener in one place, may like

func waitForNotification() {
   listener := newListener(ctx, f.config, channel)
   for {
     		// ...	
		// recreate the listener
		listener = newListener(ctx, dbConfig, channel)
        }
    }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if encapsulate the listener into waitForNotification, we cannot get the listener in the integration test. without the listener we cannot close the listener to generate an errListenerClosed to test the listener re-creation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok ...

may add some comments, the returned listener only for test purpose, it cannot be shared by others?

Signed-off-by: clyang82 <[email protected]>
@clyang82
Copy link
Contributor Author

clyang82 commented Jan 6, 2025

/retest

Signed-off-by: clyang82 <[email protected]>
logger.Infof("recreate the listener due to %s", err.Error())
l.Close()
// recreate the listener
l = newListener(ctx, dbConfig, channel)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this be a problem? it seems we are changing a shared variable without lock. What happened when another is using the listener, and it is changed here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if reaching here, it means the listener has problem. it cannot be used anymore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens when a caller calls the listener at the same time, will it panic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the listener is initialized when the maestro server is starting.

go env().Database.SessionFactory.NewListener(ctx, "events", s.KindControllerManager.AddEvent)

so there is no new caller to call the listener. we leverage LISTEN/NOTIFY mechanism to sends a notification event together with an optional “payload” string to listener that has previously executed LISTEN channel.

@clyang82 clyang82 merged commit 2833e34 into openshift-online:main Jan 10, 2025
7 checks passed
@clyang82 clyang82 deleted the recreate_listener branch January 10, 2025 01:49
machi1990 added a commit to Azure/ARO-HCP that referenced this pull request Jan 13, 2025
…e6d51af797e6d4b

This include the following notable changes;
- purge status events (openshift-online/maestro#232)
- update sdk to support using two topics for kafka (openshift-online/maestro#238)
- multiple instances support for grpc broker. (openshift-online/maestro#235)
- Recreate listener if error is occured (openshift-online/maestro#236)
- fix maestro resource status hash getter. (openshift-online/maestro#245)

The last two contain fixes needed by ARO-HCP as well.
machi1990 added a commit to Azure/ARO-HCP that referenced this pull request Jan 13, 2025
…e6d51af797e6d4b

This include the following notable changes;
- purge status events (openshift-online/maestro#232)
- update sdk to support using two topics for kafka (openshift-online/maestro#238)
- multiple instances support for grpc broker. (openshift-online/maestro#235)
- Recreate listener if error is occured (openshift-online/maestro#236)
- fix maestro resource status hash getter. (openshift-online/maestro#245)

The last two contain fixes needed by ARO-HCP as well.
geoberle pushed a commit to Azure/ARO-HCP that referenced this pull request Jan 13, 2025
…e6d51af797e6d4b (#1087)

This include the following notable changes;
- purge status events (openshift-online/maestro#232)
- update sdk to support using two topics for kafka (openshift-online/maestro#238)
- multiple instances support for grpc broker. (openshift-online/maestro#235)
- Recreate listener if error is occured (openshift-online/maestro#236)
- fix maestro resource status hash getter. (openshift-online/maestro#245)

The last two contain fixes needed by ARO-HCP as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants