Centralised logging with EFK Stack

[torrespro]

It is difficult to get an overview of what is going on in a system landscape of microservices when each microservice instance writes log records to its local filesystem. We need a component that can collect the log records from the microservice's local filesystem and store them in a central database for analysis, search, and visualization. A popular open source-based solution for this builds on the following tools:

• Elasticsearch, a distributed database with great capabilities for the search and analysis of large datasets
• Fluentd, a data collector that can be used to collect log records from various sources, filter and transform the collected information, and finally send it to various consumers, for example, Elasticsearch
• Kibana, a graphical frontend to Elasticsearch that can be used to visualize search results and run analyses of the collected log records

The idea is to show how to collect and store log records from microservice instances, as well as how to search and analyze log records.