Use protobuf and GRPC

[damip]

RFC: Use protobuf and GRPC

Protobuf and GRPC

Protocol Buffers also called "proto" or "protobuf" is a schema-based binary serialization/deserialization system. It is designed to be fast and compact.

The way it works is that the serialization format is described in a ".proto" file, and a generator tool is called to generate the described structures and serialization/deserialization code for many supported languages, including go, python and rust.

On top of that, GRPC is a binary RPC protocol that uses protobuf as the underlying serialization format. It allows describing endpoints in a single ".proto" file. A generator tool is then used to produce the RPC code implementation for various supported languages, including go, python and rust.

Note: GRPC and protobuf support streaming and versioning as well.

Why it is interesting for Massa

• We could describe all our transferable structures using .proto files, and allow them to be transparently handled in many languages without manually implementing and maintaining ser/deser for each one.
• We could use GRPC instead of JSON-RPC for the API because it is better fit for transferring binary data, faster, more compact, and can reuse the proto-described structures transparently to fit many languages.

Questions

• is proto/GRPC secure against untrusted inputs ? eg. declaring a huge array to cause a 1000GB memory pre-allocation
• would it be a good idea to use GRPC for inter-node communication as well ?

[BatiGencho]

On note here: GRPC is great but not for Frontend libs. We could expose grpc endpoints for uni and bi-directional streaming, as well as unary calls which can be a second alternative to json-rpc and that could allow ppl to interact with the node from within backends more efficiently. We def need to keep json-rpc. Grpc is also good for interporotocl communication. A good alternative is CaptainProto which is smaller and more compact.

[damip]

@BatiGencho thanks for the info, indeed it looks like GRPC is unpractical for Thyra-front communication. But it might be a good fit for the Thyra-node link.

Note however that I found this:
https://grpc.io/blog/state-of-grpc-web/
https://github.com/grpc/grpc-web

[aoudiamoncef]

IMHO, gRPC is not a replacement for JsonRPC and WebSockets, but it could be a better solution when we'll split Massa node, It's ideal for inter services communication instead of WebSockets.

bchd is a Go implementation of Bitcoin cash node which supports both gRPC and JsonRPC.

Here are just a few benefits of gRPC:

• No more hunting down API documentation or dealing with poorly documented APIs .
• No more hand-crafted JSON call objects.  All requests and responses are strongly typed and code-generated, with hints available in the IDE.
• No more dealing with methods, headers, body, and low-level networking. Everything is handled by gRPC.
• No more second-guessing the meaning of error codes. gRPC status codes are a canonical way of representing issues in APIs.
• No more one-off, server-side request handlers to avoid concurrent connections. gRPC is based on HTTP2, which multiplexes - multiple streams over the same connection. So no more head-of-line blocking.
• No more problems streaming data from a server. gRPC supports both one-to-one RPCs and one-to-many streaming requests. No more awkward REST / WebSocket combo APIs.
• No more data parse errors when rolling out new binaries. gRPC enables backwards and forwards-compatibility of requests and responses.
• No need to write any client code as the protoc compiler will autogenerate client libraries - including wire serialization and deserialization code - for just about every major language. The compiler writes thousands of lines of code so you don’t have to.

grpc-vs-websocket
grpc-vs-websocket-when-is-it-better-to-use

[aoudiamoncef]

Articels && tutorials:

• https://www.diem.com/en-us/blog/new-json-rpc/
• https://github.com/hyperium/tonic/blob/master/examples/routeguide-tutorial.md
• https://blog.logrocket.com/rust-and-grpc-a-complete-guide/
• https://www.thorsten-hans.com/grpc-services-in-rust-with-tonic/

Libraries && tools:

• https://github.com/google/gnostic
• https://github.com/hyperium/tonic
• https://github.com/tokio-rs/prost
• https://github.com/stepancheg/rust-protobuf
• https://github.com/tafia/quick-protobuf
• https://github.com/nytimes/openapi2proto

[gregLibert]

IMHO, gRPC is not a replacement for JsonRPC and WebSockets, but it could be a better solution when we'll split Massa node, It's ideal for inter services communication instead of WebSockets.

I wonder if this couldn't be an alternative. Obviously, the browser will not be able to connect directly to the gRPC endpoints, but it could use a proxy.
Thus, publishing the node API would be as simple as writing the proxy configuration.
Finally, using a proxy would have other advantages:

• health monitoring if we split the node into more than one component
• fine-grained access management (limiting, authorization)
• visibility on network calls
• ...

[Thykof]

There is this Protobuf lib in assemblyscript https://github.com/piotr-oles/as-proto

[gregLibert]

And that means we could also use protobuf in our smart contract to :

• serialize/deserialize function arguments/returned values
• serialize/deserialize the byte-encoded data in the datastore.

This would offer great flexibility as all programming languages could call our smart contracts or directly read the datastore as long as they have the proto files.

To make this easier, we could standardize sending systematically the .proto files (one for the exported functions, one for the datastore) with the SC at deployment time.

[Thykof]

Also for return values.

But it might not be ideal because sc-developers would need to write a proto file for each function (and a second one for each return values) and for each datastore record. It could be great if we can create a transformer to automate this.

[aoudiamoncef]

I found this interesting feature which creates a binding between gRPC and classic RESTFull APIs. We don't have any reason to maintain both gRPC and JsonRPC APIs.

gRPC transcoding
gRPC Http

[aoudiamoncef]

Questions

• is proto/GRPC secure against untrusted inputs ? eg. declaring a huge array to cause a 1000GB memory pre-allocation

In general, security issues comes from implementations of protobuf and transport layers, we could find CVEs in protobuf Github Security Advisories only for officially supported languages*.

As we are using Rust, I had to see security issues in protobuf + gRPC crates. The most mature stack is Prost + Tonic. I didn't found any CVEs, may be Rust is safer ?! (Joke)

In order to give a beginning of response to the question above, as gRPC uses HTTP as transport layer, we could limit the request body size (frame); More information links below:

• Allow setting a message max size hyperium/tonic#264
• tonic: add max http2 frame size to server. hyperium/tonic#529

I pinged Tonic dev in this related issue.

• would it be a good idea to use GRPC for inter-node communication as well ?

I would say yes, It seems to be the most mature all in one solution:

• Bi-directional streaming
• High performance async io
• Interoperability
• TLS backed by rustls
• Load balancing
• Custom metadata
• Authentication
• Health Checking

[damip]

Thanks @aoudiamoncef for this breakdown. Yes, limiting the size of the proto message is an option. But I'm worried about the following: what if I hack a protobuf message and change the encoded size of say a string (the size is encoded as u64 varint) and set it to 10 terabytes, even though the message is small and passes the size check ? Does the pre-allocation still happen and crash the receiving computer ?

[gregLibert]

I do hope that the unmarshaling lib is doing a proper input validation by matching the expected size of the string and its content...

[aoudiamoncef]

After a deep dive in the source code of Tonic and Prost:

In Tonic, we could setup a middleware to set the maximum size of http2 frames as a first protection.

1- Tonic handles for us gRPC framing and sends the buffered data of an entire message:

/// Decodes gRPC message types
pub trait Decoder {
	...
    /// Decode a message from the buffer.
    ///
    /// The buffer will contain exactly the bytes of a full message. There
    /// is no need to get the length from the bytes, gRPC framing is handled
    /// for you.
    fn decode(&mut self, src: &mut DecodeBuf<'_>) -> Result<Option<Self::Item>, Self::Error>;
}

2- Tonic calls a function in prost to create a Message struct:

impl<U: Message + Default> Decoder for ProstDecoder<U> {
    type Item = U;
    type Error = Status;

    fn decode(&mut self, buf: &mut DecodeBuf<'_>) -> Result<Option<Self::Item>, Self::Error> {
        let item = Message::decode(buf)
            .map(Option::Some)
            .map_err(from_decode_error)?;

        Ok(item)
    }
}

3 - Prost will consume the buffer in order to create the Message. For Length-Delimited Records, Prost seems to handle the buffer underflow introduced by our wrong string size of 10TB:

    pub(super) fn merge_one_copy<A, B>(
        wire_type: WireType,
        value: &mut A,
        buf: &mut B,
        _ctx: DecodeContext,
    ) -> Result<(), DecodeError>
    where
        A: BytesAdapter,
        B: Buf,
    {
        check_wire_type(WireType::LengthDelimited, wire_type)?;
        let len = decode_varint(buf)?;
        if len > buf.remaining() as u64 {
            return Err(DecodeError::new("buffer underflow"));
        }
        let len = len as usize;

        // If we must copy, make sure to copy only once.
        value.replace_with(buf.take(len));
        Ok(())
    }

4 resources:

• https://www.oreilly.com/library/view/grpc-up-and/9781492058328/ch04.html
• https://developers.google.com/protocol-buffers/docs/encoding#structure
• https://github.com/hyperium/tonic/blob/ff642f9233beab322333745f9edfa9c62ae18ca4/tonic/src/codec/mod.rs#L72-L77
• https://github.com/hyperium/tonic/blob/master/tonic/src/codec/prost.rs#L63-L66
• https://github.com/tokio-rs/prost/blob/b72967aae99c3f185249ebb950238201678868fe/src/encoding.rs#L1002-L1006

[damip]

Very interesting ! So limiting the total size safely limits the size of every array ! Thanks @aoudiamoncef

[damip]

• JSON:

  • • native to the web, no need for libs on the client side
  • • our dev target (fullstack web devs) are very familiar with it
  • • available as libs for all SC languages
  • • reasonnably widely accepted for communication between wasm services
  • • very heavy representation of binary (non-string) data
      • does BSON solve that problem partially ?
  • • high gas usage for parsing and serialization and high storage costs (slightly better with BSON ?)
  • • in its basic, schema-free represenation, it requires maintaining structure representations in every supported language
      • Does JSON Schema and accompanying codegens solve this problem ? (Compatibility with BSON ?)
      • JSON Schema does not auto-generate the host implementation (wasmer/wasmtime side) for functions and interfaces, only the (de-)serialization of function arguments. So they need to be maintained separately.
  • • no support for native multi-argument SC functions

• Protobuf:

  • • grpc available for efficient communication between services (eg. thyra/node)
  • • a single .proto schema needs to be maintained and generators can be used for many languages
  • • available as codegen and libs for many SC languages (incl. AS)
  • • medium gas usage for parsing and serialization, low storage costs
  • • no support for multi-argument SC functions
  • • traditional webdevs are not used to protobuf (and creating schemas in general ?) ; no schema-free alternative
  • • does not auto-generate the host implementation (wasmer/wasmtime side) for functions and interfaces, only the (de-)serialization of function arguments and return types. So they still have to be maintained separately.

• WIT

  • • a single .wit file generates both Host and Guest implementations in many languages, allows multi-argument functions transparently with native types
  • • top performance on Wasm
  • • transparent interoperability with other services written in Wasm with other languages
  • • currently no AS implem (but doable according to our studies)
  • • still experimental
  • • how does a webdev interact with it from a web page ?

[aoudiamoncef]

Apache AVRO

• a single avro JSON schema(external or included in each message) needs to be maintained and generators can be used for many languages
• no SC languages specific support
• medium gas usage for parsing and serialization, low storage costs(but Protobuf wins)
• no support for multi-argument SC functions
• traditional webdevs are not used to Avro (and creating schemas in general ?) ; backend developers does.
• does not auto-generate the host implementation (wasmer/wasmtime side) for functions and interfaces, only the (de-)serialization of function arguments and return types. So they still have to be maintained separately.

In addition:
gRPC:

• support transcoding Protobuf to JSON via proxies(internal or external)
• support transport over Http/1 and WebSockets via proxies(internal or external)

Resources:

• https://www.funnel-labs.io/2022/08/26/data-serialization-apache-avro-vs-google-protobuf
• https://github.com/apache/avro
• https://github.com/grpc-ecosystem/awesome-grpc

[BatiGencho]

One thing I also wanted to add about grpc is that since it uses a tagged schema, one could also partially deserialize the payload if only particular part is needed. That is a big advantage and it makes it quite fast in some cases.

[aoudiamoncef]

After building a small POC, I have some informations to share:

• Schema reflection (introspection). See https://github.com/hyperium/tonic/blob/master/tonic-reflection/tests/server.rs
• Support Http 1.1 transport. See https://github.com/hyperium/tonic/blob/master/tonic-web/README.md
• Limit size of message(s). See Config max request message size to fend off potential malicious attacks hyperium/tonic#1097
• Support gateway annotations. See Support gRPC-gateway annotations hyperium/tonic#332
• Support Json instead of Protobuf. See https://github.com/hyperium/tonic/blob/master/examples/src/json-codec/common.rs
• Support of service health check. See https://github.com/hyperium/tonic/blob/master/tonic-health/README.md
• Support mTLS + auth. See mTLS Authorization hyperium/tonic#511
• Massa gRPC POC. See https://github.com/massalabs/massa/tree/feature/testnet-19-grpc
• Postman gRPC client with Massa collection. See https://www.postman.com/massalabs

We have some missing features in Rust like generating a gateway(proxy), but there is a generator in Go for example and lot of Api gateways support proxing out of the box. If we would serve Json from the node, we have to use the JsonCodec and keep use serde_json.

[damip]

Alternative: Borsh

Quote from @BatiGencho

Borsh and Protobuf are both serialization formats used for encoding data in a compact and efficient way in blockchain systems. While both formats have their advantages and disadvantages, there are a few reasons why modern blockchains tend to prefer Borsh over Protobuf:
Performance: Borsh is generally faster than Protobuf, both in terms of encoding and decoding. This is because Borsh uses a simpler encoding scheme, with fewer overheads and fewer steps in the encoding and decoding process.
Compatibility: Borsh is designed specifically for use in Rust-based blockchain systems, which are becoming increasingly popular in the blockchain space. As such, it is well-suited to work with the unique features and requirements of these systems.
Simplicity: Borsh is generally considered to be easier to use and understand than Protobuf. This is because it uses a simple schema definition language and has fewer options and features to worry about.
Size: Borsh typically produces smaller serialized data than Protobuf, which can be important for blockchain systems with limited storage capacity.
Overall, while both Borsh and Protobuf have their strengths and weaknesses, Borsh is gaining popularity among modern blockchain systems due to its performance, compatibility, simplicity, and size advantages.
The point of this is to research the Borsh Serde and evaluate its fitness for us.

Damir's additions

Borsh is indeed more compact, higher-performance and has a very strong advantage over protobuf: Serialization is deterministic across languages and there is bijectivity between the schema and the serialized data in all languages which is not the case with protobuf ( https://gist.github.com/kchristidis/39c8b310fd9da43d515c4394c3cd9510 ). This makes it possible to create objects like transactions and sign them in other languages than rust, contrary to protobuf which does not guarantee determinism nor bijectivity, making it imparctical for generating signable objects for which the serialized form must be bit-perfect for signature purposes.

Quote from the Protobuf design notes:

The deterministic serialization is, however, NOT canonical across languages; it is also unstable across different builds with schema changes due to unknown fields.

That's why Near uses Borsh: near/nearcore#1166

However, it's not perfect as it has lower support in non-blockchain industries than protobuf and lacks versioning which led Solana people to think about moving from Borsh to Protobuf: coral-xyz/anchor#357

Note that Cosmos are discussing how to make protobuf deterministic here: https://docs.cosmos.network/main/architecture/adr-027-deterministic-protobuf-serialization

Another resource about canonicalizing protobuf: https://github.com/regen-network/canonical-proto3

[gregLibert]

All in all, I think we should explore canonical protobuf, like Cosmos did

I reach the same conclusion on my side. But it's more a run question :-)

[aoudiamoncef]

I could add:

Flatbuffers
https://grpc.io/blog/grpc-flatbuffers/

Capnproto
https://capnproto.org/

Karmem
https://github.com/inkeliz/karmem

IMHO, for the API, gRPC and Protobuf seems to be the most mature ecosystem.

Actually, we are using a mix of Prorobuf and Massa custom serialization in the gRPC endpoint ˋsend_blocks` for example. Switching to full protobuf isn’t mandatory and could be kept partially to support the new chosen Massa core serialization format.

[BatiGencho]

Flatbuffers is bad, used that in the past. Captnproto is good, but not sure how flexible.
I would say for apis, sure, go for grpc, for sc we can explore what cosmos has done e.g.

[aoudiamoncef]

We have an interesting comparaison of binary serialisation formats https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats

[damip]

I agree protobuf is the best option. If we can make protobuf canonical (see cosmos) it would remove the need for a second serialization format and simplify everything