progress

README.md

@@ -1,11 +1,19 @@
 # Mindflash Backend Coding Exercise
-Our goal is to give you a small coding challenge that allows you to demonstrate your skills and also gives you a small idea of some of problems that you may encounter at Mindflash. We know you're busy with life, so our aim is to have you spend no more than 2 hours working through this exercise. We don't expect you to finish in 2 hours, so don't worry if you can't. Submit what you have along with some notes on your thoughts and how you would proceed if you have more time. Most importantly, try to have some fun with it!
+Our goal is to give you a small coding challenge that gives you a chance to show off your skills and also gives you a small idea of some of the problems that you may encounter at Mindflash. We know you're busy with life, so our aim is to have you spend no more than 2 hours working through this exercise. We don't expect you to finish in 2 hours, so don't worry if you can't. Submit what you have along with some notes on your thoughts and how you would proceed if you have more time. Most importantly, try to have some fun with it!
+
+**Lastly, your code is yours to keep, publish, delete, or blog about. However, please no mentions of Mindflash should you choose to share or publish it.**
 
 ## Task
-Your task is to build a small dockerized application that filters, transforms, and indexes a stream of events into Elasticsearch (ES). To help you get started, we've included a [docker compose](https://docs.docker.com/compose/overview/) file to set up your local environment.
+Your task is to build a small application using that filters, transforms, and indexes a stream of events from NSQ (a real-time distributed messaging platform) into Elasticsearch (a highly performant and scalable search/aggregation engine).
+
+<p align="center">
+<img src="./architecture.png" align="center" alt="architecture diagram" />
+</p>
+
+This repository includes a number of resources to help you get started, including documentation, code samples, and a [docker compose](https://docs.docker.com/compose/overview/) file that you can use to set up your local/test environment.
 
 ## Overview
-You just got hired to join the *badass* engineering team at *BrainSpark*, the world's leading LMS! The first story in the sprint assigned assigned to you is to build an application that indexes a stream of events into ElasticSearch for use in reporting and general anaytics. For reference, the domain model of our application for the exercise is shown below:
+You just got hired to join the *badass* engineering team at *BrainSpark* (a fictional LMS)! The first story in your sprint backlog is to build an application that indexes a stream of events into ElasticSearch for use in reporting and anaytics. For reference, the domain model of our application for the exercise is shown below:
 
 <p align="center">
 <img src="./schema.png" align="center" alt="schema diagram" />
@@ -17,9 +25,9 @@ At a high level:
 - A user may attempt a course many times
 - Users can perform actions that trigger events
 
-Our focus for this exercise are the domain events. Events are published via an [NSQ Topic](http://nsq.io). The event topic messages represent a single domain event and are serialized as JSON. A sample event is shown below: 
+Our focus for this exercise are the domain events. Events are published on an [NSQ Topic](http://nsq.io). An event message represents a single event and is serialized as JSON. A sample event is shown below: 
 
-**Sample NSQ Event Message:**
+###### Sample Event Message
 ```json
 {
     "id": "5c92de28-14f0-449e-821b-e61e871179c2",
@@ -35,11 +43,11 @@ Our focus for this exercise are the domain events. Events are published via an [
 Prior to indexing the event records into ES, you will need to apply the following transformations:
 - filter out invalid messages (ie messages that do not pass the schema above)
 - filter out events with type `FOO`
-- denormalize the event by *hydrating* (ie *embedding*) the related attempt, course, trainee, and user (if available) using the *BrainSpark* API (see details section below for more info)
+- denormalize the event by *hydrating* (ie *embedding*) the related attempt, course, trainee, and user (if available) using the *BrainSpark* JSON-RPC API (see *resources* section below for more info)
 
 An example of a successful transformation is shown below:
 
-**Sample Transformed ES Document:**
+###### Sample Transformed ElasticSearch Document
 ```json
 {
     "id": "5c92de28-14f0-449e-821b-e61e871179c2",
@@ -79,12 +87,46 @@ An example of a successful transformation is shown below:
 }
 ```
 
-The events need to be indexed into the `trainee-events` index, with type `trainee-event`.
+The events need to be indexed into Elasticsearch using `event` as the document type and index `events-YYYY-MM-DD`, where `YYYY-MM-DD` is replaced using the formatted event `timestamp` attribute. *Note: you do not need to preallocate the indices, they will be created on first index*. 
 
-Details:
+## Resources
+
+- [NSQ Design Overview](http://nsq.io/overview/design.html)
+    - [nsqjs](https://github.com/dudleycarr/nsqjs) node client
+    - [go-nsq](https://github.com/nsqio/go-nsq) go client
+- [Elasticsearch Document API](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs.html)
+    - [elasticsearch.js](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html) node client
+    - [elastic](https://github.com/olivere/elastic) go client
+- [BrainSpark API](./docs/README.md)
+- [JSON RPC](http://www.jsonrpc.org/specification)
+- [a sample json event](./docs/sample-event.json)
+
+## Getting Started
+The only prerequisites for using this repository are [Docker](https://www.docker.com/what-container) & [Compose](https://docs.docker.com/compose/overview/). Installation links are below:
+- [Docker](https://store.docker.com/search?type=edition&offering=community)
+- [Compose](https://docs.docker.com/compose/install/)
+
+To start the local environment:
+```shell
+$ docker-compose up -d
+
+```
+*Note: This command will start NSQ, ElasticSearch, and the BrainSpark container. It may take a few minutes the first time you run it, as it will have to pull the images for NSQ and ElasticSearch and build the BrainSpark image locally. By default, NSQ will be mapped to localhost:4150, ElasticSearch will be mapped to localhost:9200, and the BrainSpark application will be listening on port 8080. You can change these mappings and other configurations by editing the docker-compose.yml file*
 
-Requirements:
+## Requirements
+- you may build your application using `node.js`, `go`, or `.net`
+- you should include a `Dockerfile` in the root of your repository
+- your code should be linted
+    - nodejs: use `eslint` with `eslint-config-airbnb-base`
+    - go: use `golint`
+- your code should include a couple tests
+- your code should include a `README.md` file in the root with instructions for building, running, and testing. It can also include notes on your throught process and any issues you may have run into.
 
-Expectations:
+## Evaluation
+We will evaluate your submission using the following criteria
+- Is your application well organized?
+- Is your code documented?
+- Is your code efficient & performant?
 
-Submission:
+## Submission
+Please upload this repository to Github and submit to @jgiless when complete.

architecture.dot

@@ -0,0 +1,20 @@
+digraph architecture {
+    nsq [label="NSQ" style="filled" fillcolor="skyblue1"]
+    brainspark [label="Brainspark API" shape="egg" style="filled" fillcolor="orange"]
+    es [label="ElasticSearch" shape="egg" style="filled" fillcolor="lightseagreen"]
+
+
+    brainspark->nsq [label="publish\nevents"]
+    app->nsq [label="consume\nevents"]
+    app->es [label="index\n events"]
+    brainspark->es [label="search &\n   aggregate"]
+    app->brainspark [label="query\n  related\ndata"]
+
+    subgraph clusterApp {
+        color="lightgrey"
+        style="filled"
+        app [label="Your App" shape="rect" style="filled" fillcolor="gold"]
+
+        app->app [label="filter &\n   hydrate\n    events"]
+    }
+}

brainspark/config.yml

@@ -0,0 +1,3 @@
+log:
+  format: text
+  level: info

docker-compose.yml

@@ -0,0 +1,51 @@
+version: '3'
+services:
+  brainspark:
+    build:
+      context: .
+      dockerfile: ./brainspark/Dockerfile
+    command: --jsonrpc --seed
+    depends_on:
+      - nsqd
+    ports:
+      - "8080:8080"
+    volumes:
+      - ./test.yml:/etc/brainspark/config.yml
+    environment:
+      BRAINSPARK_NSQ_HOST: nsqd:4150
+
+  make:
+    build:
+      context: .
+      dockerfile: ./Dockerfile.test
+    command: --jsonrpc --seed
+    depends_on:
+      - nsqd
+    ports:
+      - "8080:8080"
+    environment:
+      BRAINSPARK_NSQ_HOST: nsqd:4150
+
+  nsqlookupd:
+    image: nsqio/nsq
+    command: /nsqlookupd
+    ports:
+      - "4160"
+      - "4161"
+
+  nsqd:
+    image: nsqio/nsq
+    command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
+    depends_on:
+      - nsqlookupd
+    ports:
+      - "4150:4150"
+      - "4151:4151"
+
+  nsqadmin:
+    image: nsqio/nsq
+    command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
+    depends_on:
+      - nsqlookupd  
+    ports:
+      - "4171:4171"

docs/README.md

@@ -0,0 +1,44 @@
+# Brainspark API
+The *BrainSpark* API is a JSON RPC 2.0 exposed over HTTP.
+
+All operations:
+- must use path `/`
+- must use http method `POST`
+- must include header `Content-Type: application/json`
+- must include a valid [JSON RPC 2.0](http://www.jsonrpc.org/specification) payload
+
+###### Example Request
+```http
+POST / HTTP/1.1
+Host: {brainspark_host}
+Accept: application/json
+Content-Type: application/json
+
+{
+    "jsonrpc": "2.0",
+    "method": "{service}.{method}",
+    "id": "{request_id}",
+    "params": {
+        "ids": [
+            "599a7dba-fb3a-11e7-93d8-a7d489958da0"
+        ]
+    }
+}
+```
+
+## Services
+See below for a list of API methods grouped by service.
+
+### Account
+- [Account.BatchGetAccount](account-batch-get-account.md)
+- [Account.BatchGetUser](account-batch-get-user.md)
+- [Account.GetAccount](account-get-account.md)
+- [Account.GetUser](account-get-user.md)
+
+### Content
+- [Content.BatchGetCourse](content-batch-get-course.md)
+- [Content.GetCourse](content-get-course.md)
+
+### Participation
+- [Participation.BatchGetAttempt](participation-batch-get-attempt.md)
+- [Participation.GetAttempt](participation-get-attempt.md)

docs/account-batch-get-account.md

@@ -0,0 +1,56 @@
+[Back](README.md)
+
+### Batch Get Accounts
+
+Retrieves a set of 1 to 200 accounts by id. Missing accounts will be designated with a `null` value
+
+###### Method
+```
+Account.BatchGetAccount
+```
+
+###### Example Request
+```http
+POST / HTTP/1.1
+Host: {brainsparkHost}
+Content-Type: application/json
+
+{
+    "jsonrpc": "2.0",
+    "method": "Account.BatchGetAccount",
+    "params": {
+        "ids": [
+            "599a7dba-fb3a-11e7-93d8-a7d489958da0",
+            "bec784b6-fb3b-11e7-9b5b-0bc91e981e31"
+        ]
+    },
+    "id": "c4e0bf8a-fb3f-11e7-8241-0faf4421c718",
+}
+```
+
+###### Example Response
+```http
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+{
+    "jsonrpc": "2.0",
+    "result": {
+        "data": [
+            {
+                "id": "599a7dba-fb3a-11e7-93d8-a7d489958da0",
+                "name": "foo",
+                "domain": "foo.mindflash.com"
+            },
+            {
+                "id": "bec784b6-fb3b-11e7-9b5b-0bc91e981e31",
+                "name": "bar",
+                "domain": "bar.mindflash.com"
+            }
+        ]
+    },
+    "id": "c4e0bf8a-fb3f-11e7-8241-0faf4421c718"
+}
+```
+
+[Back](README.md)

docs/account-batch-get-user.md

@@ -0,0 +1,60 @@
+[Back](README.md)
+
+### Batch Get Users
+
+Retrieves a set of 1 to 200 users by id. Missing users will be designated with a `null` value
+
+###### Method
+```
+Account.BatchGetUser
+```
+
+###### Example Request
+```http
+POST / HTTP/1.1
+Host: {brainsparkHost}
+Content-Type: application/json
+
+{
+	"jsonrpc": "2.0",
+	"method": "Account.BatchGetUser",
+	"params": {
+		"ids": [
+			"853629b4-fb40-11e7-a26c-4b5026d382d8",
+			"b6feec9c-fb40-11e7-89ce-374290f50084"
+		]
+	},
+	"id": "c73931e4-fb40-11e7-a95f-fb1e838d3f17"
+}
+```
+
+###### Example Response
+```http
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+{
+	"jsonrpc": "2.0",
+	"result": {
+		"data": [
+			{
+				"id": "853629b4-fb40-11e7-a26c-4b5026d382d8",
+				"account_id": "21f8c490-2e26-42d0-bbc1-4d4efab6cc4d",
+				"email": "[email protected]",
+				"first_name": "Ashley",
+				"last_name": "Gordon"
+			},
+			{
+				"id": "b6feec9c-fb40-11e7-89ce-374290f50084",
+				"account_id": "b91e575c-8ab0-4114-b11b-d7a06cdbba4f",
+				"email": "[email protected]",
+				"first_name": "Frances",
+				"last_name": "Larson"
+			}
+		]
+	},
+	"id": "c73931e4-fb40-11e7-a95f-fb1e838d3f17"
+}
+```
+
+[Back](README.md)

docs/account-get-account.md

@@ -0,0 +1,46 @@
+[Back](README.md)
+
+### Get Account
+
+Retrieves a single account by id. A missing account will be designated with a `null` value
+
+###### Method
+```
+Account.GetAccount
+```
+
+###### Example Request
+```http
+POST / HTTP/1.1
+Host: {brainsparkHost}
+Content-Type: application/json
+
+{
+	"jsonrpc": "2.0",
+	"method": "Account.GetAccount",
+	"params": {
+		"id": "21f8c490-2e26-42d0-bbc1-4d4efab6cc4d"
+	},
+	"id": "00c56456-fb40-11e7-9ffe-4bb811edbd26"
+}
+```
+
+###### Example Response
+```http
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+{
+	"jsonrpc": "2.0",
+	"result": {
+		"data": {
+			"id": "21f8c490-2e26-42d0-bbc1-4d4efab6cc4d",
+			"created_at": "2013-05-29T08:24:48Z",
+			"name": "Topicblab"
+		}
+	},
+	"id": "00c56456-fb40-11e7-9ffe-4bb811edbd26"
+}
+```
+
+[Back](README.md)