Health Check in Microservices

Health Check Definition

A health check in microservices is a mechanism that ensures each service is functioning correctly and is available. It typically involves periodically checking the status of various components of a service and reporting their health.

• You can refer to Microservice Health Check at my Linked In for more details.

Use Cases of Health Check

Service Availability Monitoring:

• Scenario: Ensuring that each microservice is up and running.
• Benefit: Helps in quickly identifying and addressing service outages.

Dependency Checking

• Scenario: Verifying that all dependencies of a service are available and functioning.
• Benefit: Ensures the entire application stack is healthy and operational.

Deployment Validation

• Scenario: Checking the health of services post-deployment to ensure they are functioning as expected.
• Benefit: Detects deployment issues early, preventing faulty services from affecting the system.

Load Balancing

• Scenario: Directing traffic only to healthy instances of a service.
• Benefit: Ensures reliable service delivery by avoiding unhealthy instances.

Auto-scaling

• Scenario: Scaling up or down based on the health and load of the services.
• Benefit: Optimizes resource usage and cost efficiency.

Implementation

Tools

• Spring Boot Actuator, AWS Elastic Load Balancer, Kubernetes liveness and readiness probes

Endpoints:

• Health check endpoints (e.g., /health, /status) that return the health status of the service.

API Design:

• Request: GET /health
• Response:

  {
    "status": "DOWN",
    "details": {
      "sql": {
        "status": "DOWN",
        "data": {
          "error": "pq: database 'demo' does not exist"
        }
      },
      "firestore": {
        "status": "UP"
      },
      "kafka": {
        "status": "UP"
      }
    }
  }

Health Status

• UP: Indicates that the application is functioning normally and all health checks have passed.
• DOWN: Indicates that the application is experiencing issues, and one or more health checks have failed.

Implementation of core-go/health

Core Library

• Purpose: Provides basic health check functionalities
• Features:

  • Define standard health check interfaces.

    • Model Health

      package health
      
      type Health struct {
        Status  string                 `json:"status,omitempty"`
        Data    map[string]interface{} `json:"data,omitempty"`
        Details map[string]Health      `json:"details,omitempty"`
      }

  • Allow custom health checks with this standard interface Checker:

package health

import "context"

type Checker interface {
  Name() string
  Check(ctx context.Context) (map[string]interface{}, error)
  Build(ctx context.Context, data map[string]interface{}, err error) map[string]interface{}
}

• Build the response JSON from many custom health checks by this GO function Check
  • This function can be called by http handler (gin, echo, mux, go-chi)
• Implement basic checks
  • CPU, memory, disk space: not yet implemented.
  • Cache (Redis, Memcached)
  • Databases: sql, mongo, dynamodb, firestore, elasticsearch, cassandra, hive
  • Message Queue
  • External Service Health
• Integration with Existing Systems, by supporting these Go libraries: gin, echo, mux, go-chi

External Service Health Check Library

• Purpose: Monitors the availability of external services.
• Features:
  • Check HTTP/HTTPS endpoints for expected responses.
    • http client. The sample is at go-sql-hexagonal-architecture-sample.
  • Measure response time and reliability.

Cache Health Check Library

• Purpose: Verifies the status of cache services.
• Features:
  • Check connectivity to cache servers (Redis, Memcached).
    • Redis:
      • go-redis/redis to support redis/go-redis. The sample is at go-admin.
      • garyburd/redigo to support gomodule/redigo.
  • Validate cache hit/miss ratio and performance metrics.

Database Health Check Libraries

• Purpose: Monitors the health of database connections
• Features:
  • Check connectivity and response time for various databases (SQL, NoSQL).
    • sql. The sample is at go-sql-sample.
    • mongo to support mongo-driver/mongo. The sample is at go-mongo-sample.
    • dynamodb to support aws/aws-sdk-go/service/dynamodb. The sample is at go-dynamodb-tutorial.
    • firestore to support go/firestore. The sample is at go-firestore-sample.
    • elasticsearch to support elastic/go-elasticsearch. The sample is at go-elasticsearch-sample.
    • cassandra to support apache/cassandra-gocql-driver. The sample is at go-cassandra-sample.
    • hive to support beltran/gohive. The sample is at go-hive-sample.
  • Provide detailed status messages and error handling.

Message Queue Health Check Libraries

• Purpose: Ensures message queues are operational.
• Features:
  • Check connectivity and queue depth for different message brokers.
    • Amazon SQS: support aws-sdk-go/service/sqs. The sample is at go-amazon-sqs-sample.
    • Google Pub/Sub: support go/pubsub. The sample is at go-pubsub-sample.
    • Kafka: support 3 GO libraries: segmentio/kafka-go, IBM/sarama and confluent. The sample is at go-kafka-sample.
    • NATS: support nats.go. The sample is at go-nats-sample.
    • Active MQ: support go-stomp. The sample is at go-active-mq-sample.
    • RabbitMQ: support rabbitmq/amqp091-go. The sample is at go-rabbit-mq-sample.
    • IBM MQ: support ibmmq. The sample is at go-ibm-mq-sample.
  • Monitor message lag and processing time (Not yet implemented)

Future Libraries to develop

Cluster Health Check Library

• Purpose: Ensures the health of the microservices cluster.
• Features:
  • Check node status, CPU, and memory usage across the cluster.
  • Integrate with orchestration platforms like Kubernetes for liveness and readiness probes.

Metrics and Monitoring Integration Library

• Purpose: Integrates health checks with monitoring tools.
• Features:
  • Export health check results to monitoring systems (Prometheus, Grafana, ELK stack).
  • Provide detailed dashboards and alerting mechanisms.

Notification and Alerting Library

• Purpose: Sends alerts based on health check results.
• Features:
  • Integrate with notification systems (Slack, PagerDuty, email).
  • Provide configurable thresholds and alerting rules.

Integration with Existing Systems

• Designed to integrate seamlessly with existing Go libraries: Gorilla mux, Go-chi, Echo and Gin.
  • handler, to support Gorilla mux and Go-chi. The sample is at go-sql-sample.
  • echo handler to support Echo. The sample is at go-sql-echo-sample.
  • gin handler to support Gin. The sample is at go-sql-gin-sample.

Benefits and Challenges:

Benefits:

• Improved Reliability: Continuous monitoring of services ensures quick detection and resolution of issues.
• Scalability: Supports growth by maintaining health across distributed systems.
• Better Resource Management: Helps optimize resource usage and prevent failures.

Challenges:

• Overhead: Adding health checks can introduce some performance overhead.
• Complexity: Requires careful planning and integration to avoid false positives and negatives.
• Maintenance: Health check implementations need regular updates to stay relevant and effective.

Appendix

Health Check for nodejs

Database Health Check Libraries

• Redis: redis-plus, to support redis.
• Mongo: mongodb-extension, to support mongodb. The sample is at mongo-modular-sample.
• My SQL: mysql2-core, to support mysql2. The sample is at sql-modular-sample.
• Oracle: oracle-core, to support oracledb.
• Postgres: pg-extension, to support pg.
• MS SQL: mssql-core, to support mssql.
• SQLite: sqlite3-core to support sqlite3.

Message Queue Health Check Libraries

• Kafka: kafka-plus, to support kafkajs. The sample is at kafka-sample.
• RabbitMQ: rabbitmq-ext, to support amqplib. The sample is at rabbitmq-sample.
• Google Pub/Sub: google-pubsub, to support @google-cloud/pubsub. The sample is at pubsub-sample
• IBM MQ: ibmmq-plus, to support ibmmq. The sample is at ibmmq-sample.
• Active MQ: activemq, to support amqplib. The sample is at activemq-sample.
• NATS: nats-plus, to support nats. The sample is at nats-sample.

Integration with Existing Systems

• for nodejs, we have express-ext to integrate with express. The sample is at is at mongo-modular-sample.

Installation

Please make sure to initialize a Go module before installing core-go/health:

go get -u github.com/core-go/health

Import:

import "github.com/core-go/health"