Skip to content

Latest commit

 

History

History
73 lines (42 loc) · 5.16 KB

egov-logging-setup.md

File metadata and controls

73 lines (42 loc) · 5.16 KB
description
This tutorial will walk you through How to Setup Logging in eGov

eGov Logging Setup

Pre-reads

Know about fluent-bit https://github.com/fluent/fluent-bit
Know about es-curator https://github.com/elastic/curator

Pre-requisites

  • DIGIT uses golang (required v1.13.3) automated scripts to deploy the builds onto Kubernetes - Linux or Windows or Mac
  • All DIGIT services are packaged using helm charts Installing Helm
  • kubectl is a CLI to connect to the kubernetes cluster from your machine
  • Install Visualstudio IDE Code for better code/configuration editing capabilities
  • Git

Logging Architecture:

Logging Architecture Dig

Logging Deployment Steps:

  1. Clone the following DIGIT-DevOps repo (If not already done as part of Infra setup), you may need to install git and then run git clone it to your machine.

    git clone -b release https://github.com/egovernments/DIGIT-DevOps

  2. Implement the kafka-v2-infra and elastic search infra setup into the existing cluster

  3. Deploy the fluent-bit, kafka-connect-infra, and es-curator into your cluster, either using Jenkins deployment Jobs or go lang deployer

    go run main.go deploy -e <environment_name> 'fluent-bit,kafka-connect-infra,es-curator'

  4. Create Elasticsearch Service Sink Connector. You can run the below command in playground pods, make sure curl is installed before running any curl commands

    1. Delete the Kafka infra sink connector if already exists with the Kafka connection, using the below command

      1. Use the below command to check Kafka infra sink connector

        curl http://kafka-connect-infra.kafka-cluster:8083/connectors/

      2. To delete the connector

        curl -X DELETE http://kafka-connect-infra.kafka-cluster:8083/connectors/egov-services-logs-to-es

    2. The Kafka Connect Elasticsearch Service Sink connector moves data from Kafka-v2-infra to Elasticsearch infra. It writes data from a topic in Kafka-v2-infra to an index in Elasticsearch infra.

      curl -X POST http://kafka-connect-infra.kafka-cluster:8083/connectors/ -H 'Content-Type: application/json' -H 'Cookie: SESSIONID=f1349448-761e-4ebc-a8bb-f6799e756185' -H 'Postman-Token: adabf0e8-0599-4ac9-a591-920586ff4d50' -H 'cache-control: no-cache' -d '{ "name": "egov-services-logs-to-es", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "connection.url": "http://elasticsearch-data-infra-v1.es-cluster-infra:9200", "type.name": "general", "topics": "egov-services-logs", "key.ignore": "true", "schema.ignore": true, "value.converter.schemas.enable": false, "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "transforms": "TopicNameRouter", "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.TopicNameRouter.regex": ".*", "transforms.TopicNameRouter.replacement": "egov-services-logs", "batch.size": 50, "max.buffered.records": 500, "flush.timeout.ms": 600000, "retry.backoff.ms": 5000, "read.timout.ms": 10000, "linger.ms": 100, "max.in.flight.requests": 2, "errors.log.enable": true, "errors.deadletterqueue.topic.name": "egov-services-logs-to-es-failed", "tasks.max": 1 } }'

    3. You can verify sink Connector by using the below command

      curl http://kafka-connect-infra.kafka-cluster:8083/connectors/

  5. Deploy the kibana-infra to query the elasticseach infra egov-services-logs indexes data.

    go run main.go deploy -e <environment_name> 'kibana-infra'

  6. You can access the logging to https://<sub-domain_name>/kibana-infra

Troubleshooting:

  1. If the data is not receiving to elasticsearch infra's egov-services-logs index from kafka-v2-infra topic egov-services-logs.

    1. Ensure that the elasticsearch sink connector is available, use the below command to check

      curl http://kafka-connect-infra.kafka-cluster:8083/connectors/

    2. Also, make sure kafka-connect-infra is running without errors

      kubectl logs -f deployments/kafka-connect-infra -n kafka-cluster

    3. Ensure elasticsearch infra is running without errors

  2. In the event that none of the above services are having issues, take a look at the fluent-bit logs and restart it if necessary.