feat: fastapi schema-registry module with DI and E2E tests

[nosahama]

About this change - What it does

Some changes based off the work from the jjaakola-aiven-fastapi branch, majorly:

• integrations tests which are failing need to be checked and resolved
  • some of the tests can be moved to the e2e folder
  • it'd be nice to stop spinning up all these kafka processes
  • it'd be nice to move this into the docker tests environment
  • i am not making any changes to the integration tests setup, so it's clear that we aren't changing logic that should verify the operability of our services
  • the integration tests fail now with issues spinning up kafka, so we can check that now or after we merge this work
• unit tests now run within a docker compose environment
• e2e tests were introduced, which also run within a docker compose environment
  • the plan is to extract some test out of the integration tests folder slowly into the e2e tests folder
  • e2e tests run against the karapace docker compose services: i.e. schema-registry/rest rather than local processes
• karapace schema registry is now it's own module which houses all the FastAPI related things
• karapace rest proxy still runs on rapu
• we add dependency injection to the project
  • wiring the dependencies for both sr and rest
  • config is loaded on app startup
  • dependencies are injected into route handlers and other modules
  • modules with dependencies are wired on app startup
• config is now based on pydantic-settings
  • settings are loaded via an .env file
  • base config is added as a yaml and packaged with the built python library
  • base config loads the reference to the .env file needed to populate the rest of the config
  • we can override the settings values directly via environment variables or via a helper method on the settings object itself
  • previously some other values were set, i.e. sentry config was added, we need to revisit this.
• commits show progress
• initial draft PR schema_registry: standalone module for SR with basic dependency injection #994 was rushed and contained more changes, this is a split out of that PR which will be closed at some point
• we would need to check the coverage mechanism now with the new module and split

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/karapace
auth.py					214, 226, 305
config.py					265, 273-290, 295-296
container.py
karapace_all.py					7-15, 20-35, 44-46
logging_setup.py					6-46
typing.py
src/karapace/protobuf
io.py					100
src/schema_registry
__main__.py					5-49
container.py
factory.py					5-59
schema_registry_apis.py
user.py					6-30
src/schema_registry/http_handlers
__init__.py					6-28
src/schema_registry/middlewares
__init__.py					6-33
src/schema_registry/routers
compatibility.py					6-37
config.py					6-121
errors.py
health.py					6-10, 18-20, 38
master_availability.py					6-13, 37-38
metrics.py					6-24
mode.py					6-47
requests.py
schemas.py					6-13, 24, 41, 70, 87
setup.py					6-27
subjects.py					6-17, 32, 47, 69, 93, 119, 134, 150, 175, 190
Project Total

The report is truncated to 25 files out of 28. To see the full report, please visit the workflow summary page.

This report was generated by python-coverage-comment-action

[jjaakola-aiven]

LGTM, getting rather large and preferable to merge. Most integration tests pass, forwarding tests have some issues. Linting and other issues can be resolved in subsequent PRs.