How to Check Connection Health

[wtatum]

In the process of switching from "out of the box" Spring Boot support for AMQP to using reactor-rabbitmq to more easily tie in to an existing reactor/webflux powered backend. The project reactor programming model gives me a lot of power, but I'm loosing a features that spring-amqp brings that I'm curious the best way to bring back.

In particular, I want to tie in to the health-check mechanism of Spring Boot by implementing a Reactive Health Indicator, but I'm having trouble figuring out the best way to check the status of a Sender's underlying connection. The transparent reconnect being invisible to senders is great, but I'd like the health endpoint to reflect even transient connection drops, but I'm not sure how to implement it.

Any feedback would be hugely appreciated.

[acogoluegnes]

One way to check connection health would be to mimic the management plugin aliveness test, that is declaring a health queue, publishing and consuming a message from this queue. The queue could be exclusive to the connection, as this is a client-oriented test.

There's everything needed in the Sender to do that, so you can come up with our own check for now. But both Sender and Receiver could contain such a check, if we find something generic enough. It could use a Spring Boot-like API like you mentioned, e.g. Mono<Health>.

[wtatum]

So, if I understand correctly this involves:

• Declare a temporary queue. Probably requires some unique factor in the queue name in case multiple connected services implement a similar check? Could just use the connection ID...
• Put a sort of ping/pong in place. Can this be done with sender alone? How do actually train from the queue? Just give them a really short TTL?
• Possibly have this repeat periodically a cache in between polls? I think most Spring ReactiveHealthIndicators just check on demand so maybe that's not needed
• If this is to be included in reactor-rabbitmq, implement a Mono<Something> type that is comparable to Mono<Health> but doesn't introduce a Spring dependency
• Implement an adapter from Mono<Something> to Mono<Health>

Sounds straightforward enough, but maybe not as simple as I'd like. Given a Connection object is there any reliable way to tell if the connection is currently open? Would that be sufficient as a health indicator, given that the connection would eventually close if connectivity were lost?

[acogoluegnes]

• Declare a temporary queue. Probably requires some unique factor in the queue name in case multiple connected services implement a similar check? Could just use the connection ID...

It could be a UUID or indeed something related to the connection name.

• Put a sort of ping/pong in place. Can this be done with sender alone? How do actually train from the queue? Just give them a really short TTL?

Pretty much, quite easy with a CountDownLatch for example.

• Possibly have this repeat periodically a cache in between polls? I think most Spring ReactiveHealthIndicators just check on demand so maybe that's not needed

Yes, I was thinking applications would call this health check, just like external monitoring systems can query Spring Boot HTTP health check endpoints.

• If this is to be included in reactor-rabbitmq, implement a Mono<Something> type that is comparable to Mono<Health> but doesn't introduce a Spring dependency

Yes.

• Implement an adapter from Mono<Something> to Mono<Health>

Not in Reactor RabbitMQ, to avoid an hard dependency on Spring.

Sounds straightforward enough, but maybe not as simple as I'd like. Given a Connection object is there any reliable way to tell if the connection is currently open? Would that be sufficient as a health indicator, given that the connection would eventually close if connectivity were lost?

There's Connection#isOpen (it could be the first step of the ping-pong health check). It can be considered enough, but the connection itself can take some time to notice it's been disconnected, the ping-pong thing is more thorough and could trigger an error immediately.