Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UBI docs consolidation #1

Merged
merged 25 commits into from
May 31, 2024
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d100735
gathering potential documentation attempts
RasonJ May 15, 2024
0596e47
considering the dashboard tutorial
RasonJ May 16, 2024
555d72b
place holder for js data structure usage
RasonJ May 16, 2024
a89ae40
data-structures placeholder
RasonJ May 16, 2024
19d1516
Updating index links
RasonJ May 20, 2024
c1085c1
adding old doc to be merged
RasonJ May 20, 2024
cf9e128
Starting to link things together
RasonJ May 22, 2024
e26b8e5
fix broken link
RasonJ May 22, 2024
92d84b1
respond to vale
epugh May 22, 2024
84f9395
more vale violations
epugh May 22, 2024
e390422
name files consistently with docs site and fix links.
epugh May 22, 2024
46cc0d9
vale
epugh May 22, 2024
4e2a617
Minor tweaks. Moved Ubi under SEARCH.
RasonJ May 22, 2024
c251237
add label for versining of spec and OS version
epugh May 23, 2024
2a39c1d
try to sort out vale error
epugh May 23, 2024
982e334
Converting mermaid diagrams to png's
RasonJ May 28, 2024
9402dbd
Updating query_id mermaid code
RasonJ May 28, 2024
db2546e
Better way to ignore the mermaid scripts in the md files
RasonJ May 28, 2024
cec25c9
description updates
RasonJ May 29, 2024
1f0fa78
schema.md updating (still need to update the mermaid diagram)
RasonJ May 30, 2024
ad1b92a
schema updates
RasonJ May 30, 2024
6f08485
chase some more vale warnings.
epugh May 30, 2024
c9edc86
updates
RasonJ May 31, 2024
6626bb1
Merge branch 'ubi-docs-consolidation' of https://github.com/o19s/docu…
RasonJ May 31, 2024
736bf69
Getting there. TODO's indicate a few loose ends to tie
RasonJ May 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/vale/styles/Vocab/OpenSearch/Products/accept.txt
Original file line number Diff line number Diff line change
@@ -88,6 +88,7 @@ Simple Schema for Observability
Tableau
TorchScript
Tribuo
User Behavior Insights
VisBuilder
Winlogbeat
Zstandard
2 changes: 2 additions & 0 deletions _search-plugins/index.md
Original file line number Diff line number Diff line change
@@ -70,6 +70,8 @@ OpenSearch provides the following search relevance features:

- [Querqy]({{site.url}}{{site.baseurl}}/search-plugins/querqy/): Offers query rewriting capability.

- [User Behavior Insights]({{site.url}}{{site.baseurl}}/search-plugins/ubi/): Link user application behavior to their queries for improving search relevance insights.

## Search results

OpenSearch supports the following commonly used operations on search results:
179 changes: 179 additions & 0 deletions _search-plugins/ubi/data-structures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
---
layout: default
title: UBI client data structures
parent: User behavior insights
has_children: false
nav_order: 7
---

# Sample client data structures
The client data structures can be used to create events that follow the [UBI event schema specification](https://github.com/o19s/opensearch-ubi),
which is described in further detail [here]({{site.url}}{{site.baseurl}}/search-plugins/ubi/schemas/).

The developer provides an implementation for the following functions:
- `getClientId()`
- `getQueryId()`

_Optionally_:
- `getSessionId()`
- `getPageId()`

<!-- Not needed with this page: Other sample implementations can be found [here](#TODO-clients-link). -->

```js
/*********************************************************************************************
* Ubi Event data structures
* The following structures help ensure adherence to the UBI event schema
*********************************************************************************************/



export class UbiEventData {
constructor(object_type, id=null, description=null, details=null) {
this.object_id_field = object_type;
this.object_id = id;
this.description = description;
this.object_detail = details;
}
}
export class UbiPosition{
constructor({ordinal=null, x=null, y=null, trail=null}={}) {
this.ordinal = ordinal;
this.x = x;
this.y = y;
if(trail)
this.trail = trail;
else {
const trail = getTrail();
if(trail && trail.length > 0)
this.trail = trail;
}
}
}


export class UbiEventAttributes {
/**
* Tries to prepopulate common event attributes
* The developer can add an `object` that the user interacted with and
* the site `position` information relevant to the event
*
* Attributes, other than `object` or `position` can be added in the form:
* attributes['item1'] = 1
* attributes['item2'] = '2'
*
* @param {*} attributes: object with general event attributes
* @param {*} object: the data object the user interacted with
* @param {*} position: the site position information
*/
constructor({attributes={}, object=null, position=null}={}) {
if(attributes != null){
Object.assign(this, attributes);
}
if(object != null && Object.keys(object).length > 0){
this.object = object;
}
if(position != null && Object.keys(position).length > 0){
this.position = position;
}
this.setDefaultValues();
}

setDefaultValues(){
try{
if(!this.hasOwnProperty('dwell_time') && typeof TimeMe !== 'undefined'){
this.dwell_time = TimeMe.getTimeOnPageInSeconds(window.location.pathname);
}

if(!this.hasOwnProperty('browser')){
this.browser = window.navigator.userAgent;
}

if(!this.hasOwnProperty('page_id')){
this.page_id = window.location.pathname;
}
if(!this.hasOwnProperty('session_id')){
this.session_id = getSessionId();
}

if(!this.hasOwnProperty('page_id')){
this.page_id = getPageId();
}

if(!this.hasOwnProperty('position') || this.position == null){
const trail = getTrail();
if(trail.length > 0){
this.position = new UbiPosition({trail:trail});
}
}
// ToDo: set IP
}
catch(error){
console.log(error);
}
}
}



export class UbiEvent {
constructor(action_name, {message_type='INFO', message=null, event_attributes={}, data_object={}}={}) {
this.action_name = action_name;
this.client_id = getClientId();
this.query_id = getQueryId();
this.timestamp = Date.now();

this.message_type = message_type;
if( message )
this.message = message;

this.event_attributes = new UbiEventAttributes({attributes:event_attributes, object:data_object});
}

/**
* Use to suppress null objects in the json output
* @param key
* @param value
* @returns
*/
static replacer(key, value){
if(value == null ||
(value.constructor == Object && Object.keys(value).length === 0)) {
return undefined;
}
return value;
}

/**
*
* @returns json string
*/
toJson() {
return JSON.stringify(this, UbiEvent.replacer);
}
}
```
{% include copy.html %}

# Sample usage

```js
export async function logUbiMessage(event_type, message_type, message){
let e = new UbiEvent(event_type, {
message_type:message_type,
message:message
});
logEvent(e);
}

export async function logDwellTime(action_name, page, seconds){
console.log(`${page} => ${seconds}`);
let e = new UbiEvent(action_name, {
message:`On page ${page} for ${seconds} seconds`,
event_attributes:{dwell_seconds:seconds},
data_object:TimeMe
});
logEvent(e);
}
```
{% include copy.html %}
48 changes: 48 additions & 0 deletions _search-plugins/ubi/documentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
layout: default
title: UBI plugin management
parent: User behavior insights
has_children: false
nav_order: 2
---


# OpenSearch User Behavior Insights

This *repository* contains the OpenSearch plugin for the User Behavior Insights (UBI) capability. This plugin
facilitates persisting client-side events (e.g. item clicks, scroll depth) and OpenSearch queries for the purpose of analyzing the data
to improve search relevance and user experience.

[Link to repo plugin's documentation](https://github.com/o19s/opensearch-ubi)

## Quick start

We need a Quick Start!!!

## UBI store

The plugin has a concept of a "store", which is a logical collection of the events and queries. A store consists of two indexes.
One index is used to store events, and the other index is for storing queries.

### OpenSearch data mappings
UBI has 2 primary indexes:
- **UBI Queries** stores all queries and results.
- **UBI Events** store that the UBI client writes events to.
*Follow the [schema deep dive]({{site.url}}{{site.baseurl}}/search-plugins/ubi/schemas/) to understand how these two indexes make UBI into a causal framework for search.*

## Plugin API


### Associating a query with client-side events

Check failure on line 36 in _search-plugins/ubi/documentation.md

GitHub Actions / vale

[vale] _search-plugins/ubi/documentation.md#L36

[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.
Raw output
{"message": "[OpenSearch.StackedHeadings] Do not stack headings. Insert an introductory sentence between headings.", "location": {"path": "_search-plugins/ubi/documentation.md", "range": {"start": {"line": 36, "column": 1}}}, "severity": "ERROR"}

The plugin passively listens to query requests passing through OpenSearch. Without any extra information,
the plugin cannot associate a query with the client-side events associated with the query. (What user clicked on what to make this query?)

To make this association, queries need to have a header value that indicates the user ID.


### Example queries

[Sample SQL queries]({{site.url}}{{site.baseurl}}/search-plugins/ubi/sql-queries/)

[Sample OpenSearch queries]({{site.url}}{{site.baseurl}}/search-plugins/ubi/dsl-queries/)
97 changes: 97 additions & 0 deletions _search-plugins/ubi/dsl-queries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
layout: default
title: UBI queries using OpenSearch
parent: User behavior insights
has_children: false
nav_order: 8
---

# Sample UBI using OpenSearch DSL queries



```json
GET .ubi_log_events/_search
{
"size":0,
"aggs":{
"event_types":{
"terms": {
"field":"action_name",
"size":10
}
}
}
}
```
{% include copy.html %}

returns
```json
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": null,
"hits": []
},
"aggregations": {
"event_types": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "brand_filter",
"doc_count": 3084
},
{
"key": "product_hover",
"doc_count": 3068
},
{
"key": "button_click",
"doc_count": 3054
},
{
"key": "product_sort",
"doc_count": 3012
},
{
"key": "on_search",
"doc_count": 3010
},
{
"key": "type_filter",
"doc_count": 2925
},
{
"key": "login",
"doc_count": 2433
},
{
"key": "logout",
"doc_count": 1447
},
{
"key": "new_user_entry",
"doc_count": 207
}
]
}
}
}
```
{% include copy.html %}

These can be performed on the OpenSearch Dashboards/Query Workbench:
[http://chorus-opensearch-edition.dev.o19s.com:5601/app/opensearch-query-workbench](http://chorus-opensearch-edition.dev.o19s.com:5601/app/opensearch-query-workbench)

Check failure on line 97 in _search-plugins/ubi/dsl-queries.md

GitHub Actions / vale

[vale] _search-plugins/ubi/dsl-queries.md#L97

[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.
Raw output
{"message": "[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.", "location": {"path": "_search-plugins/ubi/dsl-queries.md", "range": {"start": {"line": 97, "column": 16}}}, "severity": "ERROR"}

Check failure on line 97 in _search-plugins/ubi/dsl-queries.md

GitHub Actions / vale

[vale] _search-plugins/ubi/dsl-queries.md#L97

[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.
Raw output
{"message": "[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.", "location": {"path": "_search-plugins/ubi/dsl-queries.md", "range": {"start": {"line": 97, "column": 57}}}, "severity": "ERROR"}
45 changes: 45 additions & 0 deletions _search-plugins/ubi/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
layout: default
title: User behavior insights
has_children: true
nav_order: 90
redirect_from:
- /search-plugins/ubi/
---
# Overview

**Introduced 2.15**
{: .label .label-purple }

**References UBI Draft Specification X.Y.Z**
{: .label .label-purple }

User Behavior Insights, or UBI, is a community plugin for capturing client-side events and queries for the purposes of improving search relevance and user experience.
It is a causal system, linking a user's query to all subsequent user interactions with your application until they perform another search.


* An machine readable [schema](https://github.com/o19s/ubi) that faciliates interoperablity of the UBI specification.
* An OpenSearch [plugin](https://github.com/o19s/opensearch-ubi) that facilitates the storage of client-side events and queries.
* A client-side JavaScript [ example ]({{site.url}}{{site.baseurl}}/search-plugins/ubi/data-structures/) reference implementation that shows how to capture events and send those events to the OpenSearch UBI plugin.

<!-- vale off -->

| Explanation & Reference | Description
| :--------- | :------- |
| [UBI Request/Response Specification](https://github.com/o19s/ubi/) <br/> **References UBI Draft Specification X.Y.Z** | Schema standard for making UBI requests and responses |
| [UBI OpenSearch Schema Documentation]({{site.url}}{{site.baseurl}}/search-plugins/ubi/schemas/) | Documentation on the individual Query and Event stores for OpenSearch |
| `query_id` Data Flow <!-- ({{site.url}}{{site.baseurl}}/search-plugins/ubi/query_id/) --> | To remove? |


| Tutorials & How-to Guides | Description
| :--------- | :------- |
| [UBI Plugin Admin]({{site.url}}{{site.baseurl}}/search-plugins/ubi/documentation/) | How to install and use the UBI Plugin |
| [ JavaScript client structures ]({{site.url}}{{site.baseurl}}/search-plugins/ubi/data-structures/) | Sample JavaScript structures for populating the Event store |
| [UBI SQL queries ]({{site.url}}{{site.baseurl}}/search-plugins/ubi/sql-queries/) | How to write analytic queries for UBI data in SQL |
| [UBI Dashboard Tutorial]({{site.url}}{{site.baseurl}}/search-plugins/ubi/ubi-dashboard-tutorial/) | Teaches you how to build an OpenSearch dashboard with UBI data |
| ... | teaches how to do something |

<!-- vale on -->
Documentation adapted using concepts from [Diátaxis](https://diataxis.fr/)
{: .tip }

Loading