DRAFT: Using the buf.build toolchain for generating language bindings

[johnsabath]

Improves UX of building and consuming api-interfaces

• Leverages buf's remote builds to avoid having to install most system dependencies on the host machine
• Generates python, go, and typescript web bindings that are valid installable dependencies into a gen directory
• Generates correct python import paths, avoiding the need for import path hacks

There are no breaking changes to the gRPC interface as a result of this change, although some imports may need to be updated.

Changes

Generation Command

yarn generate

Python SDK

Before:

# setup.py
    install_requires=[
        'grpcio==1.48.1',
        'grpcio-tools==1.48.1',
        'protobuf==3.19.5'
    ],

# some python file that wants to use grpc
thisPath = pathlib.Path(__file__).parent.parent.resolve()
genPath = thisPath / "src/stability_sdk/interfaces/gooseai/generation"
tensPath = thisPath / "src/stability_sdk/interfaces/src/tensorizer/tensors"
assert genPath.exists()
assert tensPath.exists()

sys.path.extend([str(genPath), str(tensPath)])

import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
import stability_sdk.interfaces.gooseai.generation.generation_pb2_grpc as generation_grpc

After:

# setup.py
    install_requires=[
        'grpcio==1.51.1',
        'grpcio-tools==1.51.1',
        'protobuf==4.21.6',
        'stability-api-interfaces @ git+https://github.com/stability-ai/api-interfaces.git@js/buf-spike#subdirectory=gen/proto/python',
    ],

# some python file that wants to use grpc
import stability_api_interfaces.gooseai.generation_pb2 as generation
import stability_api_interfaces.gooseai.generation_pb2_grpc as generation_grpc

Full Diff

Go

Before:

import (
  "github.com/stability-ai/api-interfaces/gooseai/engines"
  "github.com/stability-ai/api-interfaces/gooseai/generation"
  "github.com/stability-ai/api-interfaces/gooseai/dashboard"
)

After:

import (
  "github.com/stability-ai/api-interfaces/gen/proto/go/gooseai"
)

---

Testing status:

• Verified python bindings with Python SDK
  • Stability-AI/stability-sdk@main...js/buf-spike
  • All tests pass
  • CLI works locally
• Verified python bindings with generator service
• Verified go bindings with REST and GRPC APIs
  • REST commit: https://github.com/Stability-AI/api-bridge/commit/7305f4b3a8e9c3a985c9c9521e60c41275a39be9
• Verified js bindings on JS SDK / DreamStudio
  • Grrr, this is trickier because while npm and yarn do support installing a package from a git repo, they don't support installing from a subdirectory of a git repo. Can likely be worked around via a git submodule, but still annoying.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1261 Code Smells

No Coverage information
2.4% Duplication