Bug: Separating UTs & ITs for better visibility on coverage

[narmidm]

📜 Description

In our current development process, we are facing a challenge in terms of visibility and tracking of test coverage. To address this issue, we propose the separation of Unit Tests (UTs) and Integration Tests (ITs) into distinct categories within our testing framework.

👟 Reproduction steps

make test
make test-docker

👍 Expected behavior

should work on different command for UTs & ITs

👎 Actual Behavior

make test gives combined results.

🐚 Relevant log output

NA

Version

1.1.1

🖥️ What operating system are you seeing the problem on?

Windows

✅ Proposed Solution

Create two distinct categories within our testing framework - one for Unit Tests and another for Integration Tests. This categorization will provide clarity on the purpose and scope of each test, making it easier for developers and QA teams to select and run the appropriate tests during different phases of development and testing.

Separate Unit Tests from Integration Tests:

Unit Tests (UTs): These tests will remain closely integrated with the service layer code they are meant to test. Each service or component will have its associated Unit Tests placed within the same directory as the service code.
The Unit Tests for each service or component will be organized in a subdirectory within the service's main directory. For example:
/service_name
├── src
│ ├── service_code_files
│ └── ...
└── tests
├── unit
│ ├── unit_test_1.py
│ ├── unit_test_2.py
│ └── ...
└── integration
├── integration_test_1.py
├── integration_test_2.py
└── ...

Integration Tests (ITs): A dedicated directory or folder structure will be created for Integration Tests. This separation will make it clear which tests are focused on individual components and which are designed to test the interaction between multiple components or services.
ITs will reside in a separate directory, for instance:
/integration_tests
├── integration_test_suite_1.py
├── integration_test_suite_2.py
└── ...

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find any similar issue

Code of Conduct

• I agree to follow this project's Code of Conduct

[Bhargav-InfraCloud]

I think unit tests in Go are placed in the same directory as the code. I'm not sure if moving the unit test to a sub-directory gonna add any benefit.

abc
├── abc.go
└── abc_test.go

And if black-box testing is the requirement, the tests may use package abc_test instead of package abc, but in the same directory.

About integration tests, either of the ways are fine:

1. Having a separate directory like the tests (where currently, most of the tests exist).
2. Files similar to unit tests but named as *_integration_test.go with a build tag //go:build integration to skip it from running with unit tests and coverage.

And coverage is evaluated only from unit tests.

WDYT?

[Bhargav-InfraCloud]

Unit tests in the same directory as the code, and a separate directory (say tests) in the repo's root for any other tests, is a common pattern. I can see it in many Go-based OSS projects, kubernetes, etcd, keda to name a few. I'll give it a try with this approach. Please let me know if you have any concerns/suggestions.

/assign

[narmidm]

The approach looks good to me. I think we need to have a way to run both types of Tests separately. For example, we can use different commands in the makefile for Uts and ITs.