@@ -51,62 +53,62 @@ The mermaid source is converted into an png under
graph LR
style L fill:none,stroke-dasharray: 5 5
subgraph L["`*Legend*`"]
- style ss height:150px
- subgraph ss["Standard Search"]
- direction LR
-
- style ln1a fill:blue
- ln1a[ ]--->ln1b[ ];
- end
- subgraph ubi-leg["UBI data flow"]
- direction LR
-
- ln2a[ ].->|"`**UBI interaction**`"|ln2b[ ];
- style ln1c fill:red
- ln1c[ ]-->|query_id flow|ln1d[ ];
- end
+ style ss height:150px
+ subgraph ss["Standard Search"]
+ direction LR
+
+ style ln1a fill:blue
+ ln1a[ ]--->ln1b[ ];
+ end
+ subgraph ubi-leg["UBI data flow"]
+ direction LR
+
+ ln2a[ ].->|"`**UBI interaction**`"|ln2b[ ];
+ style ln1c fill:red
+ ln1c[ ]-->|query_id flow|ln1d[ ];
+ end
end
linkStyle 0 stroke-width:2px,stroke:#0A1CCF
linkStyle 2 stroke-width:2px,stroke:red
```
```mermaid
%%{init: {
- "flowchart": {"htmlLabels": false},
+ "flowchart": {"htmlLabels": false},
- }
+ }
}%%
graph TB
-User--1) raw search string-->Search;
+User--1) raw search string-->Search;
Search--2) search string-->Docs
style OS stroke-width:2px, stroke:#0A1CCF, fill:#62affb, opacity:.5
subgraph OS[OpenSearch Cluster fa:fa-database]
- style E stroke-width:1px,stroke:red
- E[( UBI Events )]
- style Docs stroke-width:1px,stroke:#0A1CCF
- style Q stroke-width:1px,stroke:red
- Docs[(Document Index)] -."3) {DSL...} & [object_id's,...]".-> Q[( UBI Queries )];
- Q -.4) query_id.-> Docs ;
+ style E stroke-width:1px,stroke:red
+ E[( UBI Events )]
+ style Docs stroke-width:1px,stroke:#0A1CCF
+ style Q stroke-width:1px,stroke:red
+ Docs[(Document Index)] -."3) {DSL...} & [object_id's,...]".-> Q[( UBI Queries )];
+ Q -.4) query_id.-> Docs ;
end
Docs -- "5) return both query_id & [objects,...]" --->Search ;
-Search-.6) query_id.->U;
+Search-.6) query_id.->U;
Search --7) [results, ...]--> User
style *client-side* stroke-width:1px, stroke:#D35400
subgraph "`*client-side*`"
- style User stroke-width:4px, stroke:#EC636
- User["`**User**`" fa:fa-user]
- App
- Search
- U
- style App fill:#D35400,opacity:.35, stroke:#0A1CCF, stroke-width:2px
- subgraph App[ UserApp fa:fa-store]
- style Search stroke-width:2px, stroke:#0A1CCF
- Search( Search Client )
- style U stroke-width:1px,stroke:red
- U( UBI Client )
- end
+ style User stroke-width:4px, stroke:#EC636
+ User["`**User**`" fa:fa-user]
+ App
+ Search
+ U
+ style App fill:#D35400,opacity:.35, stroke:#0A1CCF, stroke-width:2px
+ subgraph App[ UserApp fa:fa-store]
+ style Search stroke-width:2px, stroke:#0A1CCF
+ Search( Search Client )
+ style U stroke-width:1px,stroke:red
+ U( UBI Client )
+ end
end
User -.8) selects object_id:123.->U;
@@ -121,22 +123,22 @@ linkStyle 3,4,5,8 stroke-width:2px,fill:none,stroke:red
There are 2 separate stores for UBI:
### 1) **UBI queries**
All underlying query information and results ([`object_id`](#object_id)'s) are stored in the **UBI Queries** store, and remains largely invisible in the background.
-The only obvious difference will be in the `ubi` stanze of the json response, *which could cause index bloat if one forgets that this is enabled*.
+The only obvious difference will be in the `ubi` stanza of the JSON response, *which could cause index bloat if one forgets that this is enabled*.
**UBI Queries** [schema](https://github.com/o19s/opensearch-ubi/tree/2.14.0/src/main/resources/queries-mapping.json):
Since UBI manages the **UBI Queries** store, the developer should never have to write directly to this store (except for importing data).
-- `timestamp` (events & queries)
+- `timestamp` (events and queries)
A UNIX timestamp of when the query was received
-- `query_id` (events & queries)
- A unique ID of the query provided by the client or generated automatically. The same query text issued multiple times would generate different `query_id`.
+- `query_id` (events and queries)
+ A unique ID of the query provided by the client or generated automatically. The same query text issued multiple times would generate different `query_id`.
- `client_id` (events)
A user/client ID provided by the client application
-- `query_response_objects_ids` (queries)
- This is an array of the `object_id`'s. This *could* be the same id as the `_id` but is meant to be the externally valid id of document/item/product.
+- `query_response_objects_ids` (queries)
+ This is an array of the `object_id`'s. This *could* be the same id as the `_id` but is meant to be the externally valid id of document/item/product.
@@ -144,84 +146,84 @@ Since UBI manages the **UBI Queries** store, the developer should never have to
This is the event store that the client side directly indexes events to, linking the event [`action_name`](#action_name), [`object_id`](#object_id)'s and [`query_id`](#query_id)'s together with any other important event information.
Since this schema is dynamic, the developer can add any new fields and structures (such as *user* information, *geo-location* information) at index time that are not in the current **UBI Events** [schema](https://github.com/o19s/opensearch-ubi/tree/2.14.0/src/main/resources/events-mapping.json):
- `application`
- +
- (size 100) - name of the application tracking UBI events (e.g. `amazon-shop`, `ABC-microservice`) + (size 100) - name of the application tracking UBI events (e.g. `amazon-shop`, `ABC-microservice`) - `action_name` -
+
- (size 100) - any name you want to call your event such as `click`, `watch`, `purchase`, and `add_to_cart`, but one could map these to any common *JavaScript* events, or debugging events. -_TODO: How to formalize? A list of standard ones and then custom ones._ + (size 100) - any name you want to call your event such as `click`, `watch`, `purchase`, and `add_to_cart`, but one could map these to any common *JavaScript* events, or debugging events. +_TODO: How to formalize? A list of standard ones and then custom ones._ -- `query_id` -
+- `query_id` +
- (size 100) - ID for some query. Either the client provides this, or the `query_id` is generated at index time by the **UBI Plugin**. - + (size 100) - ID for some query. Either the client provides this, or the `query_id` is generated at index time by the **UBI Plugin**. + The `client_id` must be consistent in both the **UBI Queries** and **UBI Events** stores. - `timestamp`: - UTC-based, UNIX epoch time. + UTC-based, UNIX epoch time. -- `message_type` - - (size 100) - originally thought of in terms of ERROR, INFO, WARN, but could be anything else useful such as `QUERY` or `CONVERSION`. - Can be used to group `action_name` together in logical bins. _Thinking this should be backend logic in analysis_ +- `message_type` + + (size 100) - originally thought of in terms of ERROR, INFO, WARN, but could be anything else useful such as `QUERY` or `CONVERSION`. + Can be used to group `action_name` together in logical bins. _Thinking this should be backend logic in analysis_ -- `message` - - (size 256) - optional text message for the log entry. For example, with a `message_type` of `INFO`, people might expect an informational or debug type text for this field, but a `message_type` of `QUERY`, we would expect the text to be more about what the user is searching on. +- `message` + + (size 256) - optional text message for the log entry. For example, with a `message_type` of `INFO`, people might expect an informational or debug type text for this field, but a `message_type` of `QUERY`, we would expect the text to be more about what the user is searching on. `event_attributes` has dynamic mapping, meaning if events are indexed with many custom fields, the index could bloat quickly with many new fields. -{: .warning} +{: .warning} -- `event_attributes`'s structure that describes any important context about the event. Within it, it has 2 primary structures `position` and `object`, as well as being extensible to add anymore relevant, custom, information about the event can be stored such as timing informaiton, individual user or session information, etc. +- `event_attributes`'s structure that describes any important context about the event. Within it, it has 2 primary structures `position` and `object`, as well as being extensible to add anymore relevant, custom, information about the event can be stored such as timing informaiton, individual user or session information, etc. - The two primary structures in the `event_attributes`: - - **`event_attributes.position`** - structure that contains information on the location of the event origin, such as screen *x,y* coordinates, or the *n-th* object out of 10 results, .... + The two primary structures in the `event_attributes`: + - **`event_attributes.position`** - structure that contains information on the location of the event origin, such as screen *x,y* coordinates, or the *n-th* object out of 10 results, .... + + - `event_attributes.position.ordinal` - - `event_attributes.position.ordinal` - - tracks the *n*th item within a list that a user could select, click (i.e. selecting the 3rd element could be event{`onClick, results[4]`}) + tracks the *n*th item within a list that a user could select, click (i.e. selecting the 3rd element could be event{`onClick, results[4]`}) - - `event_attributes.position.{x,y}` - - tracks x and y values, that the client defines + - `event_attributes.position.{x,y}` + + tracks x and y values, that the client defines - - `event_attributes.position.page_depth` - - tracks page depth of results + - `event_attributes.position.page_depth` + + tracks page depth of results - - `event_attributes.position.scroll_depth` - - tracks scroll depth of page results + - `event_attributes.position.scroll_depth` + + tracks scroll depth of page results - - `event_attributes.position.trail` - - text field for tracking the path/trail that a user took to get to this location - -
+ - `event_attributes.position.trail` + + text field for tracking the path/trail that a user took to get to this location + +
- - **`event_attributes.object`**, which contains identifying information of the object returned from the query that the user interacts with (i.e.: a book, a product, a post). - The `object` structure has two ways to refer to the object, with `object_id` being the id that links prior queries to this object: - - - `event_attributes.object.internal_id` is a unique id that OpenSearch can use to internally to index the object, think the `_id` field in the indexes. - - `event_attributes.object.object_id` - is the id that a user could look up amd find the object instance within the **document corpus**. Examples include: `ssn`, `isbn`, `ean`. Variants need to be incorporated in the `object_id`, so for a t-shirt that is red, you would need SKU level as the `object_id`. - Initializing UBI requires mapping from the **Document Index**'s primary key to this `object_id` - - - `event_attributes.object.object_id_field` - - indicates the type/class of object _and_ the ID field of the search index. - - - `event_attributes.object.description` - - optional description of the object - - - - `event_attributes.object.object_detail` - - optional text for further data object details - - - *extensible fields*: any new fields by any other names in the `object` that one indexes will dynamically expand this schema to that use-case. -{: .warning} + - **`event_attributes.object`**, which contains identifying information of the object returned from the query that the user interacts with (i.e.: a book, a product, a post). + The `object` structure has two ways to refer to the object, with `object_id` being the id that links prior queries to this object: + + - `event_attributes.object.internal_id` is a unique id that OpenSearch can use to internally to index the object, think the `_id` field in the indexes. + - `event_attributes.object.object_id` + is the id that a user could look up and find the object instance within the **document corpus**. Examples include: `ssn`, `isbn`, `ean`. Variants need to be incorporated in the `object_id`, so for a t-shirt that is red, you would need SKU level as the `object_id`. + Initializing UBI requires mapping from the **Document Index**'s primary key to this `object_id` + + - `event_attributes.object.object_id_field` + + indicates the type/class of object _and_ the ID field of the search index. + + - `event_attributes.object.description` + + optional description of the object + + + - `event_attributes.object.object_detail` + + optional text for further data object details + + - *extensible fields*: any new fields by any other names in the `object` that one indexes will dynamically expand this schema to that use-case. +{: .warning}