Skip to content

Commit

Permalink
feature: add elastic db
Browse files Browse the repository at this point in the history
- add suport to elastic db
  • Loading branch information
fonteeboa committed Dec 4, 2024
1 parent f61a9fe commit 56611f9
Show file tree
Hide file tree
Showing 9 changed files with 400 additions and 31 deletions.
26 changes: 20 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ O Log Guardian é uma biblioteca desenvolvida para padronizar e gerenciar logs d

🔹 Configuração Flexível: Permite a fácil integração com diferentes sistemas, possibilitando a customização e configuração dos logs de acordo com as necessidades específicas.

🔹 Conexão com Banco de Dados: Além da gestão dos logs, o Log Guardian pode se integrar a diferentes tipos de banco de dados, como PostgreSQL, MySQL, SQLite e MongoDB. A configuração é simples, utilizando variáveis de ambiente para especificar os detalhes de conexão.
🔹 Conexão com Banco de Dados: Além da gestão dos logs, o Log Guardian pode se integrar a diferentes tipos de banco de dados, como PostgreSQL, MySQL, SQLite, ElasticSearch e MongoDB. A configuração é simples, utilizando variáveis de ambiente para especificar os detalhes de conexão.

🔹 Inserção Automática de Logs: Quando configurado corretamente com variáveis de ambiente, o Log Guardian é capaz de inserir automaticamente os logs no banco de dados especificado.

Expand All @@ -40,37 +40,49 @@ O Log Guardian utiliza variáveis de ambiente para configurar suas operações d
<h4 align="center"> <strong>Banco de Dados Relacional</strong> </h4>

#### PostgreSQL
```

```textplein
POSTGRES_HOST: Define o endereço do host para o PostgreSQL.
POSTGRES_EXTERNAL_PORT: Especifica a porta externa para o PostgreSQL.
POSTGRES_USER: Nome de usuário para autenticação no PostgreSQL.
POSTGRES_PASSWORD: Senha para autenticação no PostgreSQL.
POSTGRES_DB: Nome do banco de dados PostgreSQL a ser utilizado.
```

#### MySQL
```

```textplein
MYSQL_HOST: Define o endereço do host para o MySQL.
MYSQL_PORT: Especifica a porta para o MySQL.
MYSQL_USER: Nome de usuário para autenticação no MySQL.
MYSQL_PASSWORD: Senha para autenticação no MySQL.
MYSQL_DBNAME: Nome do banco de dados MySQL a ser utilizado.
```

#### SQLite
```

```textplein
SQLITE_PATH: Caminho do arquivo SQLite, se for o banco de dados escolhido.
```

<h4 align="center"> <strong>Banco de Dados NoSQL</strong> </h4>

#### MongoDB
```
```textplein
MONGODB_URI: Define o URI de conexão para o MongoDB.
MONGODB_DBNAME: Nome do banco de dados MongoDB a ser utilizado.
```

<h4 align="center"> <strong>Configuração Geral</strong> </h4>
#### ElasticSearch

```textplein
ELASTIC_URI: Define o URI de conexão para o ElasticSearch.
DATABASE_TYPE: Deve ser configurado como "elastic" para utilizar o ElasticSearch como destino dos logs.
```

<h4 align="center"> <strong>Configuração Geral</strong> </h4>

```textplein
DATABASE_TYPE: Especifica o tipo de banco de dados a ser utilizado pelo Log Guardian (Valores: sqlite, postgres, mysql, mongodb).
```

Expand All @@ -79,3 +91,5 @@ DATABASE_TYPE: Especifica o tipo de banco de dados a ser utilizado pelo Log Guar
Para utilizar as funções automáticas do go-log-guardian, é obrigatório o uso da variável DATABASE_TYPE, pois algumas validações são executadas com base nesta variável antes de chamar as rotinas de inserção.

Certifique-se de fornecer valores válidos e corretos para cada uma dessas variáveis de ambiente. Isso garante uma conexão adequada e o funcionamento correto do Log Guardian com o banco de dados desejado.

⚠️ O arquivo testMain.go foi criado para testar funcionalidades de gerenciamento de logs. Para executar esta aplicação, é necessário configurar o ambiente utilizando o Docker Compose que pode ser encontrado no repo [docker-build-library](https://github.com/fonteeboa/docker-build-library/tree/master/golang/go-log-guardian)
27 changes: 21 additions & 6 deletions README_en_us.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Log Guardian is a library developed to standardize and efficiently manage logs i

🔹 Flexible Configuration: Allows easy integration with different systems, enabling customization and configuration of logs according to specific needs.

🔹 Database Connection: In addition to log management, Log Guardian can integrate with different types of databases such as PostgreSQL, MySQL, SQLite, and MongoDB. Configuration is simple, using environment variables to specify connection details.
🔹 Database Connection: In addition to log management, Log Guardian can integrate with different types of databases such as PostgreSQL, MySQL, SQLite, ElasticSearch and MongoDB. Configuration is simple, using environment variables to specify connection details.

🔹 Automatic Log Insertion: When correctly configured with environment variables, Log Guardian can automatically insert logs into the specified database.

Expand All @@ -39,37 +39,50 @@ Log Guardian uses environment variables to configure its database operations, in
<h4 align="center"><strong>Relational Database</strong></h4>

#### PostgreSQL
```

```textplein
POSTGRES_HOST: Defines the host address for PostgreSQL.
POSTGRES_EXTERNAL_PORT: Specifies the external port for PostgreSQL.
POSTGRES_USER: Username for authentication in PostgreSQL.
POSTGRES_PASSWORD: Password for authentication in PostgreSQL.
POSTGRES_DB: Name of the PostgreSQL database to be used.
```

#### MySQL
```

```textplein
MYSQL_HOST: Defines the host address for MySQL.
MYSQL_PORT: Specifies the port for MySQL.
MYSQL_USER: Username for authentication in MySQL.
MYSQL_PASSWORD: Password for authentication in MySQL.
MYSQL_DBNAME: Name of the MySQL database to be used.
```

#### SQLite
```

```textplein
SQLITE_PATH: Path to the SQLite file if it is the chosen database.
```

<h4 align="center"><strong>NoSQL Database</strong></h4>

#### MongoDB
```

```textplein
MONGODB_URI: Defines the connection URI for MongoDB.
MONGODB_DBNAME: Name of the MongoDB database to be used.
```

<h4 align="center"><strong>General Configuration</strong></h4>
#### ElasticSearch

```textpleintextplein
ELASTIC_URI: Specifies the connection URI for ElasticSearch.
DATABASE_TYPE: Must be set to "elastic" to use ElasticSearch as the log destination.
```

<h4 align="center"><strong>General Configuration</strong></h4>

```textplein
DATABASE_TYPE: Specifies the type of database to be used by Log Guardian (Values: sqlite, postgres, mysql, mongodb).
```

Expand All @@ -78,3 +91,5 @@ DATABASE_TYPE: Specifies the type of database to be used by Log Guardian (Values
To utilize the automatic functions of go-log-guardian, using the DATABASE_TYPE variable is mandatory, as some validations are performed based on this variable before calling the insertion routines.

Ensure to provide valid and correct values for each of these environment variables. This ensures proper connection and functioning of Log Guardian with the desired database.

⚠️ The testMain.go file was created to test log management functionality. To execute this application, you need to configure the environment using Docker Compose, which can be found in the [docker-build-library]((https://github.com/fonteeboa/docker-build-library/tree/master/golang/go-log-guardian)) repository.
13 changes: 11 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
module github.com/fonteeBoa/go-log-guardian

go 1.21
go 1.22

toolchain go1.22.3

replace github.com/fonteeBoa/go-log-guardian => github.com/fonteeBoa/go-log-guardian v1.0.3

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/elastic/elastic-transport-go/v8 v8.6.0 // indirect
github.com/elastic/go-elasticsearch/v8 v8.16.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-sql-driver/mysql v1.7.0 // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
Expand All @@ -17,12 +23,15 @@ require (
github.com/mattn/go-sqlite3 v1.14.17 // indirect
github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/stretchr/testify v1.8.4 // indirect
github.com/stretchr/testify v1.9.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.2 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
go.mongodb.org/mongo-driver v1.13.0 // indirect
go.opentelemetry.io/otel v1.28.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
golang.org/x/crypto v0.14.0 // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/text v0.13.0 // indirect
Expand Down
16 changes: 16 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/elastic/elastic-transport-go/v8 v8.6.0 h1:Y2S/FBjx1LlCv5m6pWAF2kDJAHoSjSRSJCApolgfthA=
github.com/elastic/elastic-transport-go/v8 v8.6.0/go.mod h1:YLHer5cj0csTzNFXoNQ8qhtGY1GTvSqPnKWKaqQE3Hk=
github.com/elastic/go-elasticsearch/v8 v8.16.0 h1:f7bR+iBz8GTAVhwyFO3hm4ixsz2eMaEy0QroYnXV3jE=
github.com/elastic/go-elasticsearch/v8 v8.16.0/go.mod h1:lGMlgKIbYoRvay3xWBeKahAiJOgmFDsjZC39nmO3H64=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-sql-driver/mysql v1.7.0 h1:ueSltNNllEqE3qcWBTD0iQd3IpL/6U+mJxLkazJ7YPc=
github.com/go-sql-driver/mysql v1.7.0/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
Expand Down Expand Up @@ -29,6 +38,7 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
Expand All @@ -40,6 +50,12 @@ github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7Jul
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.mongodb.org/mongo-driver v1.13.0 h1:67DgFFjYOCMWdtTEmKFpV3ffWlFnh+CYZ8ZS/tXWUfY=
go.mongodb.org/mongo-driver v1.13.0/go.mod h1:/rGBTebI3XYboVmgz+Wv3Bcbl3aD0QF9zl6kDDw18rQ=
go.opentelemetry.io/otel v1.28.0 h1:/SqNcYk+idO0CxKEUOtKQClMK/MimZihKYMruSMViUo=
go.opentelemetry.io/otel v1.28.0/go.mod h1:q68ijF8Fc8CnMHKyzqL6akLO46ePnjkgfIMIjUIX9z4=
go.opentelemetry.io/otel/metric v1.28.0 h1:f0HGvSl1KRAU1DLgLGFjrwVyismPlnuU6JD6bOeuA5Q=
go.opentelemetry.io/otel/metric v1.28.0/go.mod h1:Fb1eVBFZmLVTMb6PPohq3TO9IIhUisDsbJoL/+uQW4s=
go.opentelemetry.io/otel/trace v1.28.0 h1:GhQ9cUuQGmNDd5BTCP2dAvv75RdMxEfTmYejp+lkx9g=
go.opentelemetry.io/otel/trace v1.28.0/go.mod h1:jPyXzNPg6da9+38HEwElrQiHlVMTnVfM3/yv2OlIHaI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
Expand Down
26 changes: 17 additions & 9 deletions internal/database/dbhandler/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ import (
"fmt"
"os"

"github.com/elastic/go-elasticsearch/v8"
"go.mongodb.org/mongo-driver/mongo"
"gorm.io/gorm"

"github.com/fonteeBoa/go-log-guardian/internal/database/elastic"
"github.com/fonteeBoa/go-log-guardian/internal/database/mongodb"
"github.com/fonteeBoa/go-log-guardian/internal/database/mysql"
"github.com/fonteeBoa/go-log-guardian/internal/database/postgres"
Expand All @@ -16,42 +18,48 @@ import (

// GetConnection retrieves the database connection based on the DATABASE_TYPE environment variable.
//
// It returns the *gorm.DB and *mongo.Client connections and an error.
func GetConnection() (*gorm.DB, *mongo.Client, error) {
// It returns the *gorm.DB, *mongo.Client, *elasticsearch.Client connections, and an error.
func GetConnection() (*gorm.DB, *mongo.Client, *elasticsearch.Client, error) {
dbType := os.Getenv("DATABASE_TYPE")
if dbType == "" {
fmt.Println("DATABASE_TYPE not set")
return nil, nil, errors.New("database type not specified")
return nil, nil, nil, errors.New("database type not specified")
}

var gormDB *gorm.DB
var mongoClient *mongo.Client
var elasticClient *elasticsearch.Client
var err error

switch dbType {
case "postgres":
gormDB, err = postgres.Connect()
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
case "mysql":
gormDB, err = mysql.Connect()
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
case "sqlite":
gormDB, err = sqlite.Connect()
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
case "mongodb":
mongoClient, err = mongodb.Connect()
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
case "elastic":
elasticClient, err = elastic.Connect()
if err != nil {
return nil, nil, nil, err
}
default:
return nil, nil, errors.New("unsupported database type")
return nil, nil, nil, errors.New("unsupported database type")
}

return gormDB, mongoClient, nil
return gormDB, mongoClient, elasticClient, nil
}
38 changes: 38 additions & 0 deletions internal/database/elastic/connect.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package elastic

import (
"log"
"os"

"github.com/elastic/go-elasticsearch/v8"
)

// Connect establishes a connection to the ElasticSearch cluster.
//
// It reads the ElasticSearch URI from the environment variable `ELASTIC_URI`.
//
// Returns:
// - A pointer to an elasticsearch.Client instance.
// - An error, if the connection fails.
func Connect() (*elasticsearch.Client, error) {
cfg := elasticsearch.Config{
Addresses: []string{
os.Getenv("ELASTIC_URI"),
},
}

client, err := elasticsearch.NewClient(cfg)
if err != nil {
return nil, err
}

// Test connection
res, err := client.Info()
if err != nil {
return nil, err
}
defer res.Body.Close()

log.Println("ElasticSearch connection established")
return client, nil
}
Loading

0 comments on commit 56611f9

Please sign in to comment.