Improved sink query layers

[abourget]

Here's a top-level design draft of new query layers on top of the this postgres sink:

Prior art

The KV Sink has a WASM Query and GenericQuery sink configurations. Here are the configuration structures. Here is an example usage.

1. The postgres sink could get a port of the WASM Query layer from the sink-kv, and expose the pg_query() method to the WASM VM, where results could be packed in a general protobuf for response. (the sink-kv WASM Query is based on gRPC + Connect Web).

This means one could craft any SQL query, and specify the indexes necessary in the sink configuration, to satisfy requests that are baked into the WASM.

In this case, the protocol is (at start) gRPC, but could grow to be GraphQL, REST, or whatever we want to pack in the WASM server literally. We can think of exposing even an http request/response mechanism inside the WASM.. so you'd have your router there in, etc.. packed as a deployable unit (which is critical for permissionless The Graph Network deployment).

2. HasuraQuery layer

would mean we build a proto file that packs all of the configuration for Hasura, and the sink would be able to configure hasura and take care of writing to postgres from a Substreams module.

3. PostgraphileQuery

same thing as with Hasura, but fronted by Postgraphile... with its config packed in an spkg and deployed to the network as a deployable unit.

4. NativePostgresConnection

This would expose the port with native pg protocol allowing you to plug the endpoint directly in Looker or other BI tools. That's obviously database specific

5. RegexpQuery

This could expose a simple well-defined REST interface, that accepts a simple query=SELECT * FROM $1&param=table, filtered through a predefined regexp list of queries specified in the Query Layer config protobuf object (packed into the spkg).

message RegexpQueryService {
  repeated string queries_regexp = 1;
}

[abourget]

message DBTConfig {
   bytes files = 1  [ (sf.substreams.options).zip_from_folder = true ];
   // frequency config?
}

message Service {
  // Containing both create table statements and index creation statements.
  string schema = 1 [ (sf.substreams.options).from_file = true ];
  optional DBTConfig dbt_config = 2;
  bool wire_protocol_access = 3;
  HasuraConfig hasura_frontend = 4;
  PostgraphileConfig postgraphile_frontend = 5;
}

message HasuraConfig {}
message PostgraphileConfig {}