Sources
Currently, a "Source" in Sossity is an API Gateway where streaming data is received from the "outside world". Right now, this is with REST endpoints powered by App Engine. The codebase for this is cowbox.
All it takes to define a Source is to make an entry in the Sossity config file, like :sources {"hxtrialendpoint" {:type "gce"}. Sossity builds this into the dependency graph, and Terraform deploys the App Engine module.
This module is heavily monitored and autoscales according to latency metrics such as maximum time per request. There is a large amount of ops tools available inside App Engine.
Sossity favors "convention over configuration", so the name you assign the source is important everywhere. The REST endpoint is at https://{project}-dot-{sourcename}.appspot.com/endpoint, e.g. `https://hxtrialendpoint-dot-hx-trial.appspot.com/endpoint'.
An endpoint is deployed and managed with Sossity and Terraform.
Sossity only accepts valid JSON data, one object per request. Any valid JSON object is acceptable.
Cowbox then writes additional JSON metadata along with the payload to a PubSub pipeline.
{"timestamp": unix timestamp when the payload was received by endpoint
"resource_hash": md5 hash of payload
"id": generated UUID
"resource": the original payload}
In addition, fields are added to the envelope any time an exception is thrown (during a cloud dataflow job, in a sink, anywhere):
{"error" : the error message and stacktrace
"errortimestamp": unix timestamp of when the error occurred}
THIS ENVELOPE FORMAT MUST BE MAINTAINED, ALL PIPELINES DEPEND ON ITS CONSISTENCY.
If your JSON is not valid, a 400 Bad Request will be returned, with an error message in the body. If PubSub is unable to be written to, a 503 error is returned. There is no error PubSub pipeline -- all errors should be handled client-side.