Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using latest protobuf version, fixing protoc-gen-js install... #30

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Using latest protobuf version, fixing protoc-gen-js install... #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@ ARG NODE_IMAGE_TAG=16-bullseye
FROM node:${NODE_IMAGE_TAG} as builder
ARG GOLANG_VERSION=1.18.6
ARG GOLANG_PACKAGE=https://go.dev/dl/go${GOLANG_VERSION}.linux-amd64.tar.gz
ARG GRPC_VERSION=v1.48.2
ARG PROTOC_GEN_GO_VERSION=v1.28.1
ARG PROTOC_GEN_GO_GRPC_VERSION=v1.1.0
ARG GRPC_VERSION=v1.50.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this impact the Python build? There's specific reasons why we selected this gRPC version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to say no since the generated python output hasn't changed, but I could verify this with local testing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If no, then ok. Do verify quickly, though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version of protoc built here is not used, see here:
#27

# In newer versions of gRPC, the protoc-gen-js plugin has been moved to a separate repository
# https://github.com/protocolbuffers/protobuf-javascript/releases
ARG PROTOBUF_JAVASCRIPT_VERSION=3.21.2

# Install python and build tools from apt
RUN apt-get update && apt-get install -y cmake git build-essential python3-venv wget && rm -rf /var/lib/apt/lists/*
Expand All @@ -16,12 +19,20 @@ ENV PATH $GOPATH/bin:$PATH
RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
RUN mkdir -p /build

# Build GRPC; note this issue before updating: https://github.com/protocolbuffers/protobuf-javascript/issues/127
# Build gRPC
WORKDIR /build
RUN git clone --recurse-submodules -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc
WORKDIR /build/grpc
RUN mkdir -p cmake/build; cd cmake/build; cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=/usr/local ../.. && make -j 8 && make install

# Install protoc-gen-js
WORKDIR /build/protoc-gen-js
RUN wget -O protobuf-javascript.zip https://github.com/protocolbuffers/protobuf-javascript/releases/download/v${PROTOBUF_JAVASCRIPT_VERSION}/protobuf-javascript-${PROTOBUF_JAVASCRIPT_VERSION}-linux-x86_64.zip && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap these super-long lines please.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would seem to me that doing a git checkout to a release branch is a lot less brittle than this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would seem to me that doing a git checkout to a release branch is a lot less brittle than this.

To be clear, are you suggesting that we do a git checkout of the release branch and build the binary in docker instead of using the pre-built binary?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. Maybe? It would allow the protobufs to be built on non-x86-64 containers, but I'm not sure if this is relevant.

unzip protobuf-javascript.zip -d ./protobuf-javascript && \
mv ./protobuf-javascript/bin/protoc-gen-js /usr/local/bin && \
rm -rf ./protobuf-javascript && \
rm protobuf-javascript.zip

# Build api-interfaces
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@${PROTOC_GEN_GO_VERSION}
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@${PROTOC_GEN_GO_GRPC_VERSION}
Expand All @@ -34,5 +45,5 @@ WORKDIR /build/api-interfaces
RUN cmake . && make clean && cmake --build .

# Copy output to a bare container
FROM debian:bullseye-slim
FROM scratch
wbrown marked this conversation as resolved.
Show resolved Hide resolved
COPY --from=builder /build/api-interfaces/gooseai /gooseai
9 changes: 9 additions & 0 deletions build-with-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

set -euo pipefail
scriptdir=$(cd $(dirname $0) && pwd)

# Enable docker buildkit in order to use the --output flag
export DOCKER_BUILDKIT=1

docker build --output type=local,dest=$scriptdir $scriptdir
2 changes: 1 addition & 1 deletion gooseai/completion/completion.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions gooseai/completion/completion_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@

var jspb = require('google-protobuf');
var goog = jspb;
var global = (function() {
if (this) { return this; }
if (typeof window !== 'undefined') { return window; }
if (typeof global !== 'undefined') { return global; }
if (typeof self !== 'undefined') { return self; }
return Function('return this')();
}.call(null));
var global =
(typeof globalThis !== 'undefined' && globalThis) ||
(typeof window !== 'undefined' && window) ||
(typeof global !== 'undefined' && global) ||
(typeof self !== 'undefined' && self) ||
(function () { return this; }).call(null) ||
Function('return this')();

goog.exportSymbol('proto.gooseai.Answer', null, global);
goog.exportSymbol('proto.gooseai.AnswerMeta', null, global);
Expand Down
2 changes: 1 addition & 1 deletion gooseai/dashboard/dashboard.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion gooseai/dashboard/dashboard_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,13 @@

var jspb = require('google-protobuf');
var goog = jspb;
var global = (function() { return this || window || global || self || Function('return this')(); }).call(null);
var global =
(typeof globalThis !== 'undefined' && globalThis) ||
(typeof window !== 'undefined' && window) ||
(typeof global !== 'undefined' && global) ||
(typeof self !== 'undefined' && self) ||
(function () { return this; }).call(null) ||
Function('return this')();

goog.exportSymbol('proto.gooseai.APIKey', null, global);
goog.exportSymbol('proto.gooseai.APIKeyFindRequest', null, global);
Expand Down
2 changes: 1 addition & 1 deletion gooseai/engines/engines.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions gooseai/engines/engines_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@

var jspb = require('google-protobuf');
var goog = jspb;
var global = (function() {
if (this) { return this; }
if (typeof window !== 'undefined') { return window; }
if (typeof global !== 'undefined') { return global; }
if (typeof self !== 'undefined') { return self; }
return Function('return this')();
}.call(null));
var global =
(typeof globalThis !== 'undefined' && globalThis) ||
(typeof window !== 'undefined' && window) ||
(typeof global !== 'undefined' && global) ||
(typeof self !== 'undefined' && self) ||
(function () { return this; }).call(null) ||
Function('return this')();

goog.exportSymbol('proto.gooseai.EngineInfo', null, global);
goog.exportSymbol('proto.gooseai.EngineTokenizer', null, global);
Expand Down
2 changes: 1 addition & 1 deletion gooseai/generation/generation.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions gooseai/generation/generation_pb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@

var jspb = require('google-protobuf');
var goog = jspb;
var global = (function() {
if (this) { return this; }
if (typeof window !== 'undefined') { return window; }
if (typeof global !== 'undefined') { return global; }
if (typeof self !== 'undefined') { return self; }
return Function('return this')();
}.call(null));
var global =
(typeof globalThis !== 'undefined' && globalThis) ||
(typeof window !== 'undefined' && window) ||
(typeof global !== 'undefined' && global) ||
(typeof self !== 'undefined' && self) ||
(function () { return this; }).call(null) ||
Function('return this')();

goog.exportSymbol('proto.gooseai.Action', null, global);
goog.exportSymbol('proto.gooseai.Answer', null, global);
Expand Down
2 changes: 1 addition & 1 deletion gooseai/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/stability-ai/api-interfaces/gooseai
go 1.18

require (
google.golang.org/grpc v1.49.0
google.golang.org/grpc v1.50.1
google.golang.org/protobuf v1.28.1
)

Expand Down
4 changes: 2 additions & 2 deletions gooseai/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEY
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.49.0 h1:WTLtQzmQori5FUH25Pq4WT22oCsv8USpQ+F6rqtsmxw=
google.golang.org/grpc v1.49.0/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI=
google.golang.org/grpc v1.50.1 h1:DS/BukOZWp8s6p4Dt/tOaJaTQyPyOoCcrjroHuCeLzY=
google.golang.org/grpc v1.50.1/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
Expand Down