(WIP) Enabling of Knowledge Layer

[chrizmc]

This PR is intended to commit the first basic components that enable a knowledge layer in the CDSP

[slawr]

Hi @claireqiu, thinking to the future it would be good to get a little more information into the commit messages. "add files" forces someone to look into the commit itself to find the nature of the change.

Something like "json-rdf-connector: add data struct and utils headers" or "add data struct and utils headers to json-rdf-connector" would be more informative.

Update: I notice there is good information in the PR comment.

[slawr]

Great to see this work becoming public and new functionality being added to the Playground itself.

When we first discussed the Knowledge Layer Connector in the May 21st call I remember that we debated whether the Information Layer part should be parallel to the Knowledge Layer part or under it. One point of concern was that in parallel it might be interpreted as an abstraction layer and that whilst it can do that, it would have potentially high performance consequences for latency, high frequency data and ability to access advanced DB queries.

To be honest I do not recall and it is not recorded in the minutes if we came to a conclusion. I do recall saying that one possible approach would be to keep it as presented here and be careful in documenting the circumstances for its use. Also that we might not limit ourselves to it whilst developing the connector as performance was an attribute we were interested in.

So what am I saying? I think we need to be clear on how we are approaching this and the planning should reflect it.

[chrizmc]

I understand what you mean. One disadvantage of a structure like this
-knowledge layer
-- information layer
could be that you might think that the information layer can only be used together with the knowledge layer. But in the case of RealmDB (maybe also for IoTDB or other DBs), the websocket IF on the information layer can also be used by a simple UI app or something, without "starting" and using the knowledge layer.

But yes, I agree, we should be clear about how we want to do it....

[slawr]

I understand what you mean. One disadvantage of a structure like this -knowledge layer -- information layer could be that you might think that the information layer can only be used together with the knowledge layer. But in the case of RealmDB (maybe also for IoTDB or other DBs), the websocket IF on the information layer can also be used by a simple UI app or something, without "starting" and using the knowledge layer.

Agreed. I think you would do something like that if you were communicating that at the current state of development it is specified for the demands of the knowledge layer connector. Moving it later as it evolves. However as you go onto say that might confuse people who would like to use it for other purposes.

I understand that point. Also of course in terms of having a workaround for the fact that Realm is an Embedded Application Database normally bound into a specific application.

But yes, I agree, we should be clear about how we want to do it....

I think for me it is a possible communications issue around explaining the playground and avoidance of people grossly under utilising the tool-set.

I think it's great that this PR already contains some documentation.
Of course doc will always be a challenge.
When I read the root readme for the info layer it authoritatively states the components for the playground info layer are the database handlers and router. It goes on to say data access is through this web socket server and that Realm is the single database supported.
From the viewpoint of the existing BMW UX app development and in the context of the connector I think this is good documentation but it doesn't put it in context for the more general case.
For example:
Does this WS server handle a broad range of use cases? Is it simple single key/value get/set only for example?
How does it handles complex queries?
The Playground presented at the Autumn AMM discussed in part the possibilities of TS DBs, then if this was merged as it is now people could be confused by discussion of data access through this WS and IoTDB is not supported, but a new DB Realm is..

If I'm someone from one of the COVESA BoFs what do I understand about data access in the Playground in reading this?

Of course a balance in doc needs to be struck. We are not a supplier and people need to put some effort in.

I've not thought about it deeply, but one approach could be a large info box at the top putting the rest of the doc in context. This was developed for X,Y, Z requirements, if your requirements are different then you may consider using other means such as directly accessing a component.

Does that make sense?

[slawr]

It is not a requirement for merging but for the actual code modules you might consider adding SPDX information in a header to allow automation of license handling.

Here is an example: https://github.com/slawr/cdsp/blob/iotdb-udf-grafana/docker/iotdb/Dockerfile

[chrizmc]

We will have a look into SPDX, thank's for the hint.

[slawr]

Some initial review comments

[slawr]

'hier' should be 'here'?

[slawr]

'If you the handler'?

[slawr]

Flagging an early review comment as the impact is somewhat large

[slawr]

An early review observation is that in your IoTDB implementation you have chosen to replace VSS dot notation in VSS node/leaf names with underscores to (I assume) integrate and avoid conflicts with the dot notation of the IoTDB data model. Whilst a valid approach (I considered the same when creating the Playground) it is different to the one I choose which quotes the VSS name and which is used in various places in the Playground codebase. For example, the RemotiveLabs feeder, the VISSR integration, examples and in the online documentation. See https://covesa.github.io/cdsp/manuals/apache-iotdb/#integrating-vss-data-into-the-iotdb-data-model

So we need to have an alignment discussion to resolve this.