docs

gisaia · Feb 5, 2024 · 8a8a485 · 8a8a485
1 parent d806808
commit 8a8a485
Show file tree

Hide file tree

Showing 27 changed files with 1,087 additions and 54 deletions.
diff --git a/README.md b/README.md
@@ -1,7 +1,6 @@
 # arlas_cli
 
-
-ARLAS Command Line is for managing collections and indices. You can list them, get some stats, get their structure, add or delete one.
+ARLAS Command Line is for managing collections and indices.
 
 ## Prerequisite
 
@@ -19,7 +18,7 @@ In a new terminal, you should be able to see the version:
 arlas_cli
 ```
 
-## Configuration
+### Initial Cconfiguration
 The confiuration file `~/.arlas/cli/configuration.yaml` must contain the different ARLAS/elasticsearch endpoints you want to interact with. [One is automatically created for your convinience at the first launch](configuration.yaml). It contains the demo endpoint and the localhost endpoint.
 
 The configuration must contain references to collection models for creating collections. A default one is provided for ARLAS EO. A reference can be an http url or a path to a local file.
@@ -33,100 +32,189 @@ arlas_cli --help
 
 ## Examples
 
-### Listing collections
+In the following examples, you will see how to:
+- generate an elasticsearch mapping based on json objects
+- add the mapping in elasticsearch
+- list the elasticsearch indices
+- add (index) data in the elasticsearch index
+- get the structure of the mapping
+- add a collection in arlas
+- list the arlas collection
+- get the structure of a arlas collection
+- delete a collection from arlas
+- delete a mapping from elasticsearch
+
+... with the `arlas_cli` command line only!
+
+We suppose you hava an elasticsearch and arlas server running.
+
+### Generate the elasticsearch mapping
+
+Writing the elasticsearch mapping for an index can be laborious. `arlas_cli` does it for you. `arlas_cli` can inspect a NDJSON file (one json object per line) and generate the corresponding elasticsearch mapping file.
+
+Let's get a sample data file:
 ```shell
-arlas_cli demo collections list
+curl -X GET https://raw.githubusercontent.com/gisaia/arlas-cli/master/tests/sample.json -o sample.json
 ```
+
+Now we can generate the mapping file based on that sample:
 ```shell
-+------------------------------+------------------------------+
-| name                         | index                        |
-+------------------------------+------------------------------+
-| demo_ais_course              | demo_ais_course              |
-| demo_ais_flow                | demo_ais_flow                |
-...
-| demo_wpi                     | demo_wpi                     |
-| metacollection               | .demo_arlas                  |
-+------------------------------+------------------------------+
+arlas_cli local indices \
+   mapping sample.json \
+   --field-mapping track.timestamps.center:date-epoch_second \
+   --field-mapping track.timestamps.start:date-epoch_second \
+   --field-mapping track.timestamps.end:date-epoch_second \
+   --push-on courses
+```
+
+The `--push-on` option registers the mapping in the specified index.
+
+Note that the three timestamps are not identified as datetimes by `arlas_cli`. The `--field-mapping` allows to overwrite the detected type.
+
+### Generate the elasticsearch mapping
+
+To add a specific mapping, it is possible to use the `create`` command:
+
+```shell
+arlas_cli local indices \
+   create courses \
+   --mapping mapping.json 
 ```
 
-### Listing indices
+### List indices
 ```shell
 arlas_cli local indices list
 ```
+
+returns:
+
 ```shell
 +----------+--------+-------+--------+
 | name     | status | count | size   |
 +----------+--------+-------+--------+
 | .arlas   | open   | 4     | 11.9kb |
-| arlas_eo | open   | 0     | 249b   |
+| courses  | open   | 0     | 249b   |
 +----------+--------+-------+--------+
 ```
 
-### Describing a collection
+### Add data
+```shell
+arlas_cli local indices \
+   data courses sample.json
+```
+
+### Describe an index
 
 ```shell
-arlas_cli demo collections describe demo_ais_course
+arlas_cli local indices describe courses
 ```
+
+returns:
+
 ```shell
 +----------------------------------------------------+-----------+
 | field name                                         | type      |
 +----------------------------------------------------+-----------+
-| arrival.address.country                            | KEYWORD   |
-| arrival.address.port                               | KEYWORD   |
-| arrival.location                                   | GEO_POINT |
-| arrival.stop_after.duration_s                      | LONG      |
-| arrival.stop_after.location                        | GEO_POINT |
-| arrival.stop_after.location_precision.geometry     | GEO_SHAPE |
+| arrival.address.country                            | keyword   |
+| arrival.address.port                               | keyword   |
+| arrival.location                                   | geo_point |
 ...
-| track.timestamps.start                             | DATE      |
-| track.trail                                        | GEO_SHAPE |
-| track.trail_geohashes_6                            | GEO_POINT |
-| track.visibility.proportion                        | FLOAT     |
+| track.timestamps.start                             | date      |
+| track.trail                                        | geo_shape |
+| track.trail_geohashes_6                            | geo_point |
+| track.visibility.proportion                        | double    |
 +----------------------------------------------------+-----------+
 ```
 
-### Describing an index
-
+### Add a collection
 ```shell
-arlas_cli local indices describe arlas_eo
-```
-```shell
-+---------------------------------------------------------+-----------+
-| field name                                              | type      |
-+---------------------------------------------------------+-----------+
-| assets                                                  | object    |
-| bbox                                                    | float     |
-| catalog                                                 | keyword   |
-...
-| properties.view__sun_elevation                          | float     |
-| properties.water_coverage                               | float     |
-+---------------------------------------------------------+-----------+
+arlas_cli local collections \
+    create courses \
+    --index courses --display-name courses \
+    --id-path track.id \
+    --centroid-path track.location \
+    --geometry-path track.trail \
+    --date-path track.timestamps.center
 ```
 
-### Adding an index
+
+### List collections
 ```shell
-arlas_cli local indices create arlas_eo --mapping arlas_eo
+arlas_cli local collections list
 ```
 
-### Adding a collection
+returns:
+
 ```shell
-arlas_cli local collections create arlas_eo --model arlas_eo --index arlas_eo
++---------+---------+
+| name    | index   |
++---------+---------+
+| courses | courses |
++---------+---------+
 ```
 
-### Deleting an index
+### Describe a collection
+
 ```shell
-arlas_cli local indices delete arlas_eo
+arlas_cli local collections describe courses
 ```
 
-Note: by default, it is not allowed to delete an index for a given configuration. To allow deleting, edit the configuration file and set `allow_delete` to `True`.
+returns:
 
+```shell
++----------------------------------------------------+-----------+
+| field name                                         | type      |
++----------------------------------------------------+-----------+
+| arrival.address.country                            | KEYWORD   |
+| arrival.address.port                               | KEYWORD   |
+| arrival.location                                   | GEO_POINT |
+| arrival.stop_after.duration_s                      | LONG      |
+...
+| track.timestamps.start                             | DATE      |
+| track.trail                                        | GEO_SHAPE |
+| track.trail_geohashes_6                            | GEO_POINT |
+| track.visibility.proportion                        | FLOAT     |
++----------------------------------------------------+-----------+
+```
 
-### Deleting a collection
+### Delete a collection
 ```shell
-arlas_cli local collections delete arlas_eo
+arlas_cli local collections delete courses
 ```
 
-### Adding data
+### Delete an index
 ```shell
-arlas_cli local indices data arlas_eo --help
+arlas_cli local indices delete courses
 ```
+
+Note: by default, it is not allowed to delete an index for a given configuration. To allow deleting, edit the configuration file and set `allow_delete` to `True`.
+
+## Cconfiguration
+
+The command line uses the `${HOME}/.arlas/cli/configuration.yaml` configuration file:
+
+```yaml
+arlas:
+  local:
+    allow_delete: true
+    elastic:
+      headers:
+        Content-Type: application/json
+      location: http://localhost:9200
+    server:
+      headers:
+        Content-Type: application/json
+      location: http://localhost:9999/arlas
+mappings:
+  arlas_eo:
+    headers: null
+    location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/mapping.json
+models:
+  arlas_eo:
+    headers: null
+    location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/collection.json
+```
+
+The `arlas` section contains the different deployment configurations. The mapping section lists the mapping template that you can use.
+ Finaly, the models are the templates for the collections. A [detaild description](docs/model/README.md) of the configuration structure is provided.
+
diff --git a/arlas/cli/settings.py b/arlas/cli/settings.py
@@ -1,5 +1,7 @@
 from pydantic import BaseModel, Field
 import yaml
+import textwrap
+import json
 
 
 class Resource(BaseModel):
@@ -34,3 +36,24 @@ def init(configuration_file: str) -> Settings:
         with open(configuration_file, 'r') as file:
             data = yaml.safe_load(file)
             Configuration.settings = Settings.parse_obj(data)
+
+
+def __short_titles(o):
+    if type(o) is dict:
+        d = {}
+        for key in o:
+            if key == "title" and isinstance(o[key], str):
+                d[key] = textwrap.shorten(o[key], 220)
+            else:
+                d[key] = __short_titles(o[key])                   
+        return d
+    if type(o) is list:
+        return list(map(lambda elt: __short_titles(elt), o))
+    else:
+        return o
+
+
+if __name__ == '__main__':
+    model = __short_titles(Settings.model_json_schema())
+    model["$id"] = "airs_model"
+    print(json.dumps(model))
diff --git a/docs/model/README.md b/docs/model/README.md
@@ -0,0 +1,24 @@
+# README
+
+## Top-level Schemas
+
+*   [Settings](./model.md) – `airs_model`
+
+## Other Schemas
+
+### Objects
+
+*   [ARLAS](./model-defs-arlas.md) – `airs_model#/$defs/ARLAS`
+
+*   [Resource](./model-defs-resource.md) – `airs_model#/$defs/Resource`
+
+*   [Untitled object in Settings](./model-defs-resource-properties-list-of-headers-if-needed-for-https-requests-anyof-0.md) – `airs_model#/$defs/Resource/properties/headers/anyOf/0`
+
+*   [dictionary of name/arlas configurations](./model-properties-dictionary-of-namearlas-configurations.md) – `airs_model#/properties/arlas`
+
+*   [dictionary of name/mapping resources](./model-properties-dictionary-of-namemapping-resources.md) – `airs_model#/properties/mappings`
+
+*   [dictionary of name/model resources](./model-properties-dictionary-of-namemodel-resources.md) – `airs_model#/properties/models`
+
+### Arrays
+
diff --git a/docs/model/model-defs-arlas-properties-arlas-iam-url-anyof-1.md b/docs/model/model-defs-arlas-properties-arlas-iam-url-anyof-1.md
@@ -0,0 +1,15 @@
+# Untitled null in Settings Schema
+
+```txt
+airs_model#/$defs/ARLAS/properties/iam/anyOf/1
+```
+
+
+
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                                |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------------ |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [model.schema.json\*](../../out/model.schema.json "open original schema") |
+
+## 1 Type
+
+`null`, the value must be null
diff --git a/docs/model/model-defs-arlas-properties-arlas-iam-url.md b/docs/model/model-defs-arlas-properties-arlas-iam-url.md
@@ -0,0 +1,21 @@
+# ARLAS IAM URL Schema
+
+```txt
+airs_model#/$defs/ARLAS/properties/iam
+```
+
+
+
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                                |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------------ |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [model.schema.json\*](../../out/model.schema.json "open original schema") |
+
+## iam Type
+
+merged type ([ARLAS IAM URL](model-defs-arlas-properties-arlas-iam-url.md))
+
+any of
+
+*   [Resource](model-defs-resource.md "check type definition")
+
+*   [Untitled null in Settings](model-defs-arlas-properties-arlas-iam-url-anyof-1.md "check type definition")
diff --git a/docs/model/model-defs-arlas-properties-arlas-server.md b/docs/model/model-defs-arlas-properties-arlas-server.md
@@ -0,0 +1,19 @@
+# ARLAS Server Schema
+
+```txt
+airs_model#/$defs/ARLAS/properties/server
+```
+
+
+
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                                |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------------ |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [model.schema.json\*](../../out/model.schema.json "open original schema") |
+
+## server Type
+
+merged type ([ARLAS Server](model-defs-arlas-properties-arlas-server.md))
+
+all of
+
+*   [Resource](model-defs-resource.md "check type definition")
diff --git a/docs/model/model-defs-arlas-properties-dictionary-of-namees-resources-anyof-1.md b/docs/model/model-defs-arlas-properties-dictionary-of-namees-resources-anyof-1.md
@@ -0,0 +1,15 @@
+# Untitled null in Settings Schema
+
+```txt
+airs_model#/$defs/ARLAS/properties/elastic/anyOf/1
+```
+
+
+
+| Abstract            | Extensible | Status         | Identifiable            | Custom Properties | Additional Properties | Access Restrictions | Defined In                                                                |
+| :------------------ | :--------- | :------------- | :---------------------- | :---------------- | :-------------------- | :------------------ | :------------------------------------------------------------------------ |
+| Can be instantiated | No         | Unknown status | Unknown identifiability | Forbidden         | Allowed               | none                | [model.schema.json\*](../../out/model.schema.json "open original schema") |
+
+## 1 Type
+
+`null`, the value must be null