Architecture

The current deployment with docker-compose launches four services:

• NGINX proxy server that forwards all requests to Flask over a socket
• Flask, which exposes the API endpoints
• A Postgres database
• A gRPC service written in golang that searches over indexed data

Component communication

Messages between Flask and the search micro service are sent using protocol buffers. The definitions for these messages are found in the proto/ subfolder, and the Makefile has targets for generating python and golang code from these protobuf definitions.

Search microservice

The search gRPC microservice is written in golang and has an internal database to store indexed, searchable data corresponding to pieces held in the Postgres database. The service accepts a SearchRequest protobuf message consisting of many Note protobuf messages. It then executes a search against all of the indexed data in the database by calling an algorithm through an FFI layer. The algorithm performing this sequential search is written in C, hosted at https://github.com/davidgarfinkle/helsinki-ttwi. It is an implementation of the "Most Compact Partial Occurrence" algorithm described in the following paper, https://www.computer.org/csdl/proceedings-article/icme/2011/06012173/12OmNrAMEPZ

This search algorithm:

• considers only notes and ignores rests
• considers notes as 2-dimensional points of pitch and rhythmic offset (measured in absolute # of quarter notes since the beginning of the piece)
• collapses all voices into a single line
• finds all transpositions of a query
• ignores rhythmic differences

The search microservice sorts the Occurrences by:

• occurrence length (prefer more notes found)
• compactness (prefer less intervening notes)
• piece ID (so that if there is a tie, each search returns the same order of results every time)

The search microservice returns all results according to the above description, and returns Occurrence protobuf messages to the Flask application.

Flask application

The Flask application:

• exposes POST /index, GET /search, GET /excerpt (see flask/app.py)
• filters Occurrences using the query parameters in the GET /search? request (see flask/occurrence.py)
• extracts metadata from Postgres database (see flask/response.py)
• extracts highlighted excerpts (see flask/excerpt.py)
• renders HTML responses with Jinja and returned to the client.

Webclient

• Implements the client-side of the OpenAPI 3.0 spec
• performs queries by writing **kern in a text editor
• not asynchronous; submits a new form on every request
• basic pagination