diff --git a/docs/docs/collections.md b/docs/docs/collections.md index 2daed73..979cc10 100644 --- a/docs/docs/collections.md +++ b/docs/docs/collections.md @@ -10,8 +10,7 @@ ARLAS collection are built on top on ES index. They describe yhe basic data stru ```shell > arlas_cli collections --help - Usage: python -m arlas.cli.cli collections [OPTIONS] COMMAND - [ARGS]... + Usage: arlas_cli collections [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ * --config TEXT Name of the ARLAS configuration to use │ @@ -49,8 +48,7 @@ The command line options let you specify how the index should be used by the col ```shell > arlas_cli collections --config local create --help - Usage: python -m arlas.cli.cli collections create [OPTIONS] - COLLECTION + Usage: arlas_cli collections create [OPTIONS] COLLECTION Create a collection @@ -166,17 +164,45 @@ To create the collection, run the following command: ```shell > arlas_cli collections --config local create --help - Usage: python -m arlas.cli.cli collections create [OPTIONS] - COLLECTION + Usage: arlas_cli collections create [OPTIONS] COLLECTION Create a collection ╭─ Arguments ────────────────────────────────────────────────────────╮ │ * collection TEXT Collection's name [default: None] │ │ [required] │ - ╰────────────────────────────────────────────────────────────────────╯ ╭─ Options ──────────────────────────────────────────────────────────╮ +│ --model TEXT Name of the model within │ +│ your configuration, or URL │ +│ or file path │ +│ [default: None] │ +│ --index TEXT Name of the index │ +│ referenced by the │ +│ collection │ +│ [default: None] │ +│ --display-name TEXT Display name of the │ +│ collection │ +│ [default: None] │ +│ --public --no-public Whether the collection is │ +│ public or not │ +│ [default: no-public] │ +│ --owner TEXT Organisation's owner │ +│ [default: None] │ +│ --orgs TEXT List of organisations │ +│ accessing the collection │ +│ --id-path TEXT Overide the JSON path to │ +│ the id field. │ +│ [default: None] │ +│ --centroid-path TEXT Overide the JSON path to │ +│ the centroid field. │ +│ [default: None] │ +│ --geometry-path TEXT Overide the JSON path to │ +│ the geometry field. │ +│ [default: None] │ +│ --date-path TEXT Overide the JSON path to │ +│ the date field. │ +│ [default: None] │ │ --help Show this message and │ │ exit. │ ╰────────────────────────────────────────────────────────────────────╯ @@ -201,8 +227,7 @@ This command line provides a description of the collection structure (fields) an ```shell > arlas_cli collections --config local describe --help - Usage: python -m arlas.cli.cli collections describe [OPTIONS] - COLLECTION + Usage: arlas_cli collections describe [OPTIONS] COLLECTION Describe a collection diff --git a/docs/docs/collections.md.template b/docs/docs/collections.md.template new file mode 100644 index 0000000..86da4e7 --- /dev/null +++ b/docs/docs/collections.md.template @@ -0,0 +1,143 @@ +## ARLAS Collections + +ARLAS collection are built on top on ES index. They describe yhe basic data structure for visualization and handle data access policy. + +`arlas_cli` provide tools to manage the ARLAS collections with the `collections` command. + +**List collections management commands** + + +```shell +> !!!execute arlas_cli collections --help +``` + +## create + +The ARLAS Collection is linked to an index or even an index pattern. + +The command line options let you specify how the index should be used by the collection. + + +```shell +> !!!execute arlas_cli collections --config local create --help +``` + +!!! note "Collection visibility" + + The visibility options (`--public`, `--owner` and `--orgs`) allow you to choose who can access the collection. + +!!! warning "--owner" + + It is mandatory to associate an organisation to a collection. + + Example: + + `--owner your.organisation.com` + + By default, the organisation referenced in your `config` is used as owner + +### Create an ARLAS collection + +The collection has to reference an ES index available with the [arlas_cli conf](confs.md#configurations). + +The index option set the targeted index:`--index index_name` + +!!! tip "Index pattern" + + The collection can target an index pattern. + Instead of an index name, the target is an expression using `*` to reference multiple ES indices. + + Example: + + `--index index_name_prefix*` + + All the indices have to share exactly the same data mapping. + + The data contained in all referenced indices are then explorable together in arlas. + + +The collection also describe a basic structure for spatio-temporel datasets: + +- **ID path**: A data field containing unique element identifier. `--id-path unique_id_field` +- **Centroid path**: A data field containing a point geometry used for aggregations `--centroid-path point_geom_field` +- **Geometry path**: A data field containing a geometry representing the element `--geometry-path element_geom_field` +- **Date path**: A data field containing the date associated to each element `--date-path date_field` + +!!! tip "Pretty name" + + A pretty name for the collection can be set at the creation with the `display-name` option + + Example: + + `--display-name "Pretty Collection Name"` + + +To create the collection, run the following command: + + +```shell +> arlas_cli collections \ + --config local \ + collection_name \ + --index index_name \ + --id-path unique_id_field \ + --centroid-path point_geom_field \ + --geometry-path element_geom_field \ + --date-path date_field \ + --display-name "Pretty Collection Name" +``` + +## name + + +```shell +> !!!execute arlas_cli collections --config local create --help +``` + +### Set a pretty name for the collection + +## set_alias + +### Set a pretty name for a data field + +## list + +### List available collections + +## describe + +This command line provides a description of the collection structure (fields) and of its metadata. + + +```shell +> !!!execute arlas_cli collections --config local describe --help +``` +### Describe a collection + +## count + +### Count the number of element within a collection + +## sample + +### Display a sample of the collection data + +## private + +### Set the collection as private + +## public + +### Set the collection as public + +### Define visibility policy for collections + +## share + +## unshare + +### Share collections between organisations + +## delete + +### Delete an organisation \ No newline at end of file diff --git a/docs/docs/collections_1.md b/docs/docs/collections_1.md deleted file mode 100644 index e65c136..0000000 --- a/docs/docs/collections_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## List collections management commands - - -```shell diff --git a/docs/docs/collections_2.md b/docs/docs/collections_2.md deleted file mode 100644 index 4a7803c..0000000 --- a/docs/docs/collections_2.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Create a collection - -You can create a collection from scratch or by providing a collection model (`--model`). The command line options let you specify how the index should be used by the collection. The visibility options (`--public`, `--owner` and `--orgs`) allow you to choose who can access the collection. - - -```shell diff --git a/docs/docs/collections_3.md b/docs/docs/collections_3.md deleted file mode 100644 index 1fad127..0000000 --- a/docs/docs/collections_3.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Describe a collection - -This command line provides a description of the collection structure (fields) and of its metadata. - - -```shell diff --git a/docs/docs/collections_4.md b/docs/docs/collections_4.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/collections_4.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/configuration.md.template b/docs/docs/configuration.md.template new file mode 100644 index 0000000..dff1e6e --- /dev/null +++ b/docs/docs/configuration.md.template @@ -0,0 +1,226 @@ +# Configuration + +`arlas_cli` uses a yaml file for storing various ARLAS and elasticsearch configurations. + +## Initial configuration + +At its first launch, `arlas_cli` will create a first [configuration file](https://raw.githubusercontent.com/gisaia/arlas-cli/master/configuration.yaml) for you. + + +```shell +> arlas_cli --version +X.X.X +Warning : no configuration file found, we created an empty one for +you (~/.arlas/cli/configuration.yaml). +``` + +By default, the file is located in `$HOME/.arlas/cli/configuration.yaml`. + +It contains two ARLAS configurations, one pointing at a local deployment, one on ARLAS demo (without elasticsearch). + +!!! Note + The configuration can also contain references to collection models for creating collections. A default one is provided for ARLAS EO. + + A reference can be an http url or a path to a local file. + It can also contain references to index mappings for creating indices. + +!!! warning + If you used the ARLAS Exploration Stack, it is possible that you already have a directory named `$HOME/.arlas`. This directory has been created by docker as root. + + The owner of the directory must be changed to the local user (`sudo chown ${USER}: $HOME/.arlas`). + + +## Configurations file + +The command line uses the `${HOME}/.arlas/cli/configuration.yaml` configuration file: + +```yaml +arlas: + local: + allow_delete: true + elastic: + headers: + Content-Type: application/json + location: http://localhost:9200 + server: + headers: + Content-Type: application/json + location: http://localhost:9999/arlas +mappings: + arlas_eo: + headers: null + location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/mapping.json +models: + arlas_eo: + headers: null + location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/collection.json +``` + +The [`arlas`](#arlas) section contains the different deployment configurations. The [mapping](#mappings) section lists the mapping template that you can use. + Finally, the [models](#models) are the templates for the collections. A [detailed description](model/README.md) of the configuration structure is provided. + +### arlas + +#### ARLAS Authentication + +- ARLAS server url +- organisation +- arlas-iam +- token-url +- auth-header +- login +- password +- client-id +- client-secret +- grant-type + +#### Elasticsearch + +- url +- login +- password +- header + +### mappings + +### models + +## Create, describe and delete a configuration for `arlas_cli` + +You can edit directly the `${HOME}/.arlas/cli/configuration.yaml` configuration file to update your configurations. You can also use the command line itself. + +To list the configurations: + +```shell +> arlas_cli confs list ++-----------+-----------------------------+ +| name | url | ++-----------+-----------------------------+ +| local | http://localhost:9999/arlas | +| test_conf | http://localhost:9999 | ++-----------+-----------------------------+ +``` + +To describe a configuration: + + +```shell +> arlas_cli confs describe local +allow_delete: true +authorization: null +elastic: + headers: + Content-Type: application/json + location: http://localhost:9200 +server: + headers: + Content-Type: application/json + location: http://localhost:9999/arlas +``` + +To create a simple configuration: + + +```shell +> arlas_cli confs create dev_conf \ + --server http://localhost:9999 \ + --headers "Content-Type:application/json" \ + --elastic http://localhost:9200 \ + --elastic-headers "Content-Type:application/json" \ + --no-allow-delete +``` + +For an arlas configuration with authentication: + + +```shell +> arlas_cli --config-file /tmp/configuration.yaml confs \ + create myarlas_as_user \ + --server http://myserver/arlas \ + --headers "arlas-org-filter:my_org_name" \ + --headers "Content-Type:application/json" \ + --no-allow-delete \ + --auth-token-url http://myserver/arlas_iam_server/session \ + --auth-login user \ + --auth-password my_password \ + --auth-headers "Content-Type:application/json;charset=utf-8"\ + --auth-arlas-iam +``` + +To delete the configuration: + + +```shell +> arlas_cli confs delete dev_conf +``` + +Also, it is possible to use a different configuration file than the one placed in your home directory (`$HOME/.arlas/cli/configuration.yaml`): + + +```shell +> arlas_cli --config-file /tmp/config.yaml +Warning : no configuration file found, we created an empty one for you (/tmp/config.yaml). +``` + +## Configure arlas_cli for ARLAS Cloud + +To configure `arlas_cli` for your cloud.arlas.io account: + +First, set the following environment variables by changing appropriately `SET_THIS_VALUE`: +```shell +export MY_ORGANIZATION=SET_THIS_VALUE +export ARLAS_USER=SET_THIS_VALUE +export ARLAS_PWD=SET_THIS_VALUE +export ELASTIC_ENDPOINT=SET_THIS_VALUE +export ELASTIC_USER=SET_THIS_VALUE +export ELASTIC_PWD=SET_THIS_VALUE +``` + +Then run the command `arlas_cli confs create cloud.arlas.io` with all the parameters: + +```shell +arlas_cli confs \ + create cloud.arlas.io \ + --server "https://cloud.arlas.io/arlas/server" \ + --headers "arlas-org-filter:${MY_ORGANIZATION}" \ + --headers "Content-Type:application/json" \ + --no-allow-delete \ + --auth-token-url https://cloud.arlas.io/arlas/iam/session \ + --auth-login "${ARLAS_USER}" \ + --auth-password "${ARLAS_PWD}" \ + --auth-headers "Content-Type:application/json;charset=utf-8" \ + --auth-org "${MY_ORGANIZATION}" \ + --elastic "${ELASTIC_ENDPOINT}" \ + --elastic-headers "Content-Type:application/json" \ + --elastic-login "${ELASTIC_USER}" \ + --elastic-password "${ELASTIC_PWD}" \ + --elastic-headers "Content-Type:application/json" \ + --auth-headers "Content-Type:application/json;charset=utf-8" \ + --persistence "https://cloud.arlas.io/arlas/persistence" \ + --persistence-headers "Content-Type:application/json" \ + --auth-arlas-iam +``` + +Check the configuration exists: + +```shell +arlas_cli confs list +``` + +You can now list the collections: + +```shell +arlas_cli collections --config cloud.arlas.io list +``` + +List the indices: + +```shell +arlas_cli indices --config cloud.arlas.io list +``` + +List the persisted elements: + +```shell +arlas_cli persist --config cloud.arlas.io groups config.json +``` diff --git a/docs/docs/confs.md b/docs/docs/confs.md index 66929ea..9d8726a 100644 --- a/docs/docs/confs.md +++ b/docs/docs/confs.md @@ -21,7 +21,7 @@ It is possible, with the `arlas_cli` command line, to manage the ARLAS configura ```shell > arlas_cli confs --help - Usage: python -m arlas.cli.cli confs [OPTIONS] COMMAND [ARGS]... + Usage: arlas_cli confs [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ @@ -42,7 +42,7 @@ It is possible, with the `arlas_cli` command line, to manage the ARLAS configura ```shell > arlas_cli confs create --help - Usage: python -m arlas.cli.cli confs create [OPTIONS] NAME + Usage: arlas_cli confs create [OPTIONS] NAME Add a configuration diff --git a/docs/docs/confs.md.template b/docs/docs/confs.md.template new file mode 100644 index 0000000..285c3f2 --- /dev/null +++ b/docs/docs/confs.md.template @@ -0,0 +1,51 @@ +# Configurations + + +## ARLAS configurations + +The configuration file (default is `$HOME/.arlas/cli/configuration.yaml`) contains 3 sections: + +- the list of ARLAS configurations +- the list of mappings +- the list of collection models + +An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and elasticsearch. + +It supports the addition of http headers. + +It is possible, with the `arlas_cli` command line, to manage the ARLAS configurations, but not the two last ones. + +**List configurations (confs) management commands** + + +```shell +> !!!execute arlas_cli confs --help +``` + + +## create + + +```shell +> !!!execute arlas_cli confs create --help +``` + +A configuration is meant to communicate with a deployed ARLAS. + + +For ARLAS, keycloak and ARLAS IAM authentications are supported. + +!!! note "--auth-arlas-iam " + Default is ARLAS IAM, use `--no-auth-arlas-iam` for keycloak. + + +## delete + +!!! danger + By default, it is not possible to run the `delete` command on an elasticsearch with `arlas_cli`. This is to prevent accidental data loss. In order to allow delete on a configuration, use the `--allow-delete` option. + + +## describe + + +## list \ No newline at end of file diff --git a/docs/docs/confs_1.md b/docs/docs/confs_1.md deleted file mode 100644 index b8ffbef..0000000 --- a/docs/docs/confs_1.md +++ /dev/null @@ -1,11 +0,0 @@ - -!!! tip - At its first launch, `arlas_cli` will create a first configuration file for you (`$HOME/.arlas/cli/configuration.yaml`), with two ARLAS configurations, one pointing at a local deployment, one on ARLAS demo (without elasticsearch). - -!!! bug - If you used the ARLAS Exploration Stack, it is possible that you already have a directory named `$HOME/.arlas`. This directory has been created by docker as root. The owner of the directory must be changed to the local user (`sudo chown ${USER}: $HOME/.arlas`). - -## List configuration management commands - - -```shell diff --git a/docs/docs/confs_2.md b/docs/docs/confs_2.md deleted file mode 100644 index b5c2ba0..0000000 --- a/docs/docs/confs_2.md +++ /dev/null @@ -1,19 +0,0 @@ -``` - -## Create a configuration - -The configuration file (default is `$HOME/.arlas/cli/configuration.yaml`) contains 3 sections: - -- the list of ARLAS configurations -- the list of mappings -- the list of collection models - -It is possible, with the `arlas_cli` command line, to manage the ARLAS configurations, but not the two last ones. - -An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and elasticsearch. It supports the addition of http headers. For ARLAS, keycloak and ARLAS IAM authentications are supported. Default is keycloak, use `--auth-arlas-iam` for ARLAS IAM. - -!!! danger - By default, it is not possible to run the `delete` command on an elasticsearch with `arlas_cli`. This is to prevent accidental data loass. In order to allow delete on a configuration, use the `--allow-delete` option. - - -```shell diff --git a/docs/docs/confs_3.md b/docs/docs/confs_3.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/confs_3.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/help.md b/docs/docs/help.md index 1354fc7..bd17f9b 100644 --- a/docs/docs/help.md +++ b/docs/docs/help.md @@ -29,22 +29,35 @@ or within a sub command: ```shell > arlas_cli indices --config demo mapping --help -Usage: arlas_cli indices mapping [OPTIONS] FILE + + Usage: arlas_cli indices mapping [OPTIONS] FILE + + Generate the mapping based on the data + +╭─ Arguments ────────────────────────────────────────────────────────╮ +│ * file TEXT Path to the file conaining the data. Format: │ +│ NDJSON │ +│ [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────╮ +│ --nb-lines INTEGER Number of line to consider for │ +│ generating the mapping. Avoid │ +│ going over 10. │ +│ [default: 2] │ +│ --field-mapping TEXT Overide the mapping with the │ +│ provided field path/type. Example: │ +│ fragment.location:geo_point. │ +│ Important: the full field path │ +│ must be provided. │ +│ --no-fulltext TEXT List of keyword or text fields │ +│ that should not be in the fulltext │ +│ search. Important: the field name │ +│ only must be provided. │ +│ --push-on TEXT Push the generated mapping for the │ +│ provided index name │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────╯ - Generate the mapping based on the data - -Arguments: - FILE Path to the file conaining the data. Format: NDJSON [required] - -Options: - --nb-lines INTEGER Number of line to consider for generating the mapping. - Avoid going over 10. [default: 2] - --field-mapping TEXT Overide the mapping with the provided field path/type. - Example: fragment.location:geo_point. Important: the - full field path must be provided. - --no-fulltext TEXT List of keyword or text fields that should not be in - the fulltext search. Important: the field name only - must be provided. - --push-on TEXT Push the generated mapping for the provided index name - --help Show this message and exit. ``` diff --git a/docs/docs/help.md.template b/docs/docs/help.md.template new file mode 100644 index 0000000..1d7f6cc --- /dev/null +++ b/docs/docs/help.md.template @@ -0,0 +1,32 @@ +## Getting help + +At any time, if you need some help on how to use the command line or one of its sub command, simply add `--help`. + +Whether you are at the top level: + +```shell +> arlas_cli --help +Usage: arlas_cli [OPTIONS] COMMAND [ARGS]... + +Options: + --config-file TEXT Path to the configuration file if you do not + want to use the default one: + .arlas/cli/configuration.yaml. + --print-curl / --no-print-curl Print curl command [default: no-print-curl] + --version Print command line version + --help Show this message and exit. + +Commands: + collections + confs + iam + indices + persist +``` + +or within a sub command: + + +```shell +> !!!execute arlas_cli indices --config demo mapping --help +``` diff --git a/docs/docs/iam.md b/docs/docs/iam.md index 675bf46..4bc73ff 100644 --- a/docs/docs/iam.md +++ b/docs/docs/iam.md @@ -6,7 +6,7 @@ ```shell > arlas_cli iam --config local orgs --help - Usage: python -m arlas.cli.cli iam orgs [OPTIONS] COMMAND [ARGS]... + Usage: arlas_cli iam orgs [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ @@ -50,7 +50,7 @@ ```shell > arlas_cli iam --config local users --help - Usage: python -m arlas.cli.cli iam users [OPTIONS] COMMAND [ARGS]... + Usage: arlas_cli iam users [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ diff --git a/docs/docs/iam.md.template b/docs/docs/iam.md.template new file mode 100644 index 0000000..a9c1a05 --- /dev/null +++ b/docs/docs/iam.md.template @@ -0,0 +1,15 @@ +# Identity & Access Management + +## Manage organisations, users, permissions and groups + + +```shell +> !!!execute arlas_cli iam --config local orgs --help +``` + +## Create, delete, activate / deactivate users + + +```shell +> !!!execute arlas_cli iam --config local users --help +``` diff --git a/docs/docs/iam_1.md b/docs/docs/iam_1.md deleted file mode 100644 index ac35aa2..0000000 --- a/docs/docs/iam_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## Manage organisations, users, permissions and groups - - -```shell diff --git a/docs/docs/iam_2.md b/docs/docs/iam_2.md deleted file mode 100644 index 7283b57..0000000 --- a/docs/docs/iam_2.md +++ /dev/null @@ -1,6 +0,0 @@ -``` - -## Create, delete, activate / deactivate users - - -```shell diff --git a/docs/docs/iam_3.md b/docs/docs/iam_3.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/iam_3.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/index.md.template b/docs/docs/index.md.template new file mode 100644 index 0000000..29acdb2 --- /dev/null +++ b/docs/docs/index.md.template @@ -0,0 +1,58 @@ +# About arlas_cli + +__ARLAS Exploration__ is an Open Source software for exploring and analysing Geo BigData. __ARLAS Command Line__ (`arlas_cli`) is for managing elasticsearch indices, ARLAS collections and elasticsearch/ARLAS endpoint configurations. + + +TODO: Definition of concepts and schema, at least: + +- ES index +- ES mapping +- ARLAS collection +- configuration +- persistence / entry +- organisation +- group +- user +- permission + + + +`arlas_cli` is a python command line for: + +- managing elasticsearch indices: + - generate an index mapping based on [NDJSON data](https://jsonlines.org/) + - create an index + - list indices + - describe an index + - delete an index +- managing [ARLAS collections](https://docs.arlas.io/arlas-api-collection/) + - create a collection + - list collections + - describe a collection + - delete a collection +- managing ARLAS persistence + - create a entry in a zone + - list entries within a zone + - describe an entry + - delete an entry + - get the groups accessing a zone +- managing ARLAS Identity and Access (ARLAS IAM) + - list organisations + - add an organisation + - within an organisation: + - list visible collections + - list groups + - list users + - list permissions + - add/delete a permission + - add/delete a group + - add/delete a permission to/from a group + - add/delete a user + - add/delete a user to/from a group + - create a user + - activate a user + - deactivate a user +- managing configurations + - register an ARLAS/elasticsearch configuration, with headers and authentication parameters + - delete a configuration + diff --git a/docs/docs/indices.md b/docs/docs/indices.md index 0238d81..15888dd 100644 --- a/docs/docs/indices.md +++ b/docs/docs/indices.md @@ -2,7 +2,7 @@ ## Elasticsearch index -ARLAS rely on [elasticsearch](https://www.elastic.co/fr/elasticsearch), an open-source search engine to perform quick aggregation requests over data. +ARLAS relies on [elasticsearch](https://www.elastic.co/fr/elasticsearch), an open-source search engine to perform quick aggregation requests over data. To be explored in ARLAS dashboards, the data has to be indexed in an ES index. An index contains the data and a mapping to describe how fields have to be interpreted (types). @@ -14,12 +14,12 @@ To be explored in ARLAS dashboards, the data has to be indexed in an ES index. A ```shell > arlas_cli indices --help - Usage: python -m arlas.cli.cli indices [OPTIONS] COMMAND [ARGS]... + Usage: arlas_cli indices [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ * --config TEXT Name of the ARLAS configuration to use │ │ from your configuration file │ -│ (~/.arlas/cli/configuration.yaml │ +│ (/Users/gaudan/.arlas/cli/configuration.… │ │ [default: None] │ │ [required] │ │ --help Show this message and exit. │ @@ -47,12 +47,12 @@ To be explored in ARLAS dashboards, the data has to be indexed in an ES index. A ```shell > arlas_cli indices --config local mapping --help - Usage: python -m arlas.cli.cli indices mapping [OPTIONS] FILE + Usage: arlas_cli indices mapping [OPTIONS] FILE Generate the mapping based on the data ╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * file TEXT Path to the file containing the data. Format: │ +│ * file TEXT Path to the file conaining the data. Format: │ │ NDJSON │ │ [default: None] │ │ [required] │ @@ -76,6 +76,7 @@ To be explored in ARLAS dashboards, the data has to be indexed in an ES index. A │ [default: None] │ │ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ + ``` ### Data file @@ -193,7 +194,7 @@ The `indices create` sub-function create the index from a mapping json file. ```shell > arlas_cli indices --config local create --help - Usage: python -m arlas.cli.cli indices create [OPTIONS] INDEX + Usage: arlas_cli indices create [OPTIONS] INDEX Create an index @@ -209,6 +210,7 @@ The `indices create` sub-function create the index from a mapping json file. │ [default: 1] │ │ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ + ``` ### Create an ES index with its mapping @@ -238,13 +240,13 @@ The `indices data` sub-function ingest the data in a given index. ```shell > arlas_cli indices --config local data --help - Usage: python -m arlas.cli.cli indices data [OPTIONS] INDEX FILES... + Usage: arlas_cli indices data [OPTIONS] INDEX FILES... Index data ╭─ Arguments ────────────────────────────────────────────────────────╮ │ * index TEXT index's name [default: None] [required] │ -│ * files FILES... List of paths to the file containing the │ +│ * files FILES... List of pathes to the file conaining the │ │ data. Format: NDJSON │ │ [default: None] │ │ [required] │ @@ -286,12 +288,12 @@ To list the available ES indices, simply use the `indices list` sub-function. No ```shell > arlas_cli indices --config local list --help - Usage: python -m arlas.cli.cli indices help [OPTIONS] + Usage: arlas_cli indices list [OPTIONS] - List indices + List indices ╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ +│ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ ``` @@ -320,14 +322,19 @@ Once the index is created, the description of the fields it contains (correspond ```shell > arlas_cli indices --config local data --help - Usage: python -m arlas.cli.cli indices describe [OPTIONS] INDEX + Usage: arlas_cli indices data [OPTIONS] INDEX FILES... - Describe an index + Index data ╭─ Arguments ────────────────────────────────────────────────────────╮ │ * index TEXT index's name [default: None] [required] │ +│ * files FILES... List of pathes to the file conaining the │ +│ data. Format: NDJSON │ +│ [default: None] │ +│ [required] │ ╰────────────────────────────────────────────────────────────────────╯ ╭─ Options ──────────────────────────────────────────────────────────╮ +│ --bulk INTEGER Bulk size for indexing data [default: 100] │ │ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ @@ -365,19 +372,18 @@ The first rows of the data contained in an index can be displayed with the `indi ```shell > arlas_cli indices --config local delete --help - Usage: python -m arlas.cli.cli indices sample [OPTIONS] INDEX + Usage: arlas_cli indices delete [OPTIONS] INDEX - Display a sample of an index + Delete an index ╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * index TEXT index's name [default: None] [required] │ +│ * index TEXT index's name [default: None] [required] │ ╰────────────────────────────────────────────────────────────────────╯ ╭─ Options ──────────────────────────────────────────────────────────╮ -│ --pretty / --no-pretty [default: pretty] │ -│ --size INTEGER [default: 100] │ -│ --help Show this message and exit. │ +│ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ + ``` ### Visualize few rows of your dataset @@ -412,15 +418,15 @@ The ES index can be deleted with `indices delete` sub command to free space on E ```shell > arlas_cli indices --config local delete --help - Usage: python -m arlas.cli.cli indices delete [OPTIONS] INDEX + Usage: arlas_cli indices delete [OPTIONS] INDEX - Delete an index + Delete an index ╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * index TEXT index's name [default: None] [required] │ +│ * index TEXT index's name [default: None] [required] │ ╰────────────────────────────────────────────────────────────────────╯ ╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ +│ --help Show this message and exit. │ ╰────────────────────────────────────────────────────────────────────╯ ``` diff --git a/docs/docs/indices.md.template b/docs/docs/indices.md.template new file mode 100644 index 0000000..6418f85 --- /dev/null +++ b/docs/docs/indices.md.template @@ -0,0 +1,325 @@ +# Indices + +## Elasticsearch index + +ARLAS relies on [elasticsearch](https://www.elastic.co/fr/elasticsearch), an open-source search engine to perform quick aggregation requests over data. + +To be explored in ARLAS dashboards, the data has to be indexed in an ES index. An index contains the data and a mapping to describe how fields have to be interpreted (types). + +`arlas_cli` provide tools to infer mapping from data and manage the ES indices with the `indices` command. + +**List index management commands** + + +```shell +> !!!execute arlas_cli indices --help +``` + +## mapping + +`arlas_cli` provide tools to infer the ES mapping directly from a data file. + + +```shell +> !!!execute arlas_cli indices --config local mapping --help +``` + +### Data file + +To generate a mapping, you need to provide a NDJSON `file` (New line delimiter JSON). + +The values of the first lines of the files are used for inferring the mapping for each field of the data. + +!!! note "--nb_lines" + By default, the `indices mapping` function uses the first 2 rows to infer mapping. + If a field is not present in the first rows, it will not appear in the mapping. + + Make sure to take enough rows into account to get all the fields with the option `--nb_lines` + +!!! tip + For NDJSON files generated with Apache Spark, files are often partitioned into several parts into a `data.json` folder: + ``` + part-00000-[...].json + part-00001-[...].json + ... + ``` + In practice, the `file` Argument can be filed with a pattern such as `path/to/data.json/part-00000-*.json` to avoid writing the generated hash + + +### Type identification + +The mapping associates to each field of the data a type (see [elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + +A **geometry** is identified as such if + +- it is a geojson +- it is a WKT string +- the field name contains `geohash` +- it is a string containing two float separated by a comma + +A **date** is identified as such if + +- its name is one of `timestamp`, `date`, `start` or `end` and that it can be parsed as a date +- its name contains `timestamp`, `date`, `start` or `end` and its values are number within [631152000, 4102444800] or [631152000000, 4102444800000] (year 1990 to 2100) + +!!! note "--field-mapping" + If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. + + It has the structure **field_name:field_type** (see [elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + + Examples: + + - `--field-mapping field_point:geo_point` + - `--field-mapping field_geometry:geo_shape` + - `--field-mapping field_short_text:keyword` + - `--field-mapping field_long_text:text` + - `--field-mapping field_float:double` + - `--field-mapping field_int:long` + + The date fields have a format that can be specified as **field_name:date-format** with all format accepted by [elasticsearch date type](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) + + Examples: + + - `--field-mapping field_time_epoch_second:date-epoch_second` + - `--field-mapping field_time_epoch_millisecond:date-epoch_millis` + - `--field-mapping field_time_pattern:date-"yyyy-MM-dd HH:mm:ss"` + +By default, the keywords and text fields are searchable as fulltext to be accessible in the searchbar. + +!!! note "--no-fulltext " + + If it is not required for a given field, it is more optimal to deactivate fulltext search. + + Example: + + - `--no-fulltext field_keyword` + +### Created mapping + +By default, the `arlas_cli indices mapping` directly returns the mapping in the command line. + +Once you're happy with the mapping, you can either store it in a file or directly push it on elasticsearch. + +!!! tip "Store mapping in a file" + To store the created mapping in a `mapping.json` file, simply use `>` as the end of your command. + + Example: + + ```shell + > arlas_cli indices \ + --config local \ + mapping path/to/data.json \ + --field-mapping timestamps.start:date-epoch_second \ + --field-mapping timestamps.end:date-epoch_second \ + > path/to/mapping.json + ``` + +!!! note "--push-on" + + To push the inferred mapping directly in an elasticsearch index, use the `--push-on` option with the target index name. + + Example: + + ```shell + > arlas_cli indices \ + --config local \ + mapping path/to/data.json \ + --push-on index_name + ``` + + The index is then created and the [index creation command](#create) can be skipped. + +## create + +Before putting the data in an elasticsearch index, the index has to be initialised with the [correct mapping](#mapping). + +The `indices create` sub-function create the index from a mapping json file. + + +```shell +> !!!execute arlas_cli indices --config local create --help +``` + +### Create an ES index with its mapping + +The index name and the path to the mapping json file have to be used to create the ES index. + +The `indices create` sub-function create the index from a mapping json file. + +Example: + +```shell +> arlas_cli indices \ + --config local \ + create index_name \ + --mapping path/to/mapping.json +``` + +Once the index is created, it is ready to [ingest data](#data). + +## data + +To explore data in ARLAS, it has to be indexed in the created ES index. + +The `indices data` sub-function ingest the data in a given index. + + +```shell +> !!!execute arlas_cli indices --config local data --help +``` +### Ingest data + +For indexing data, you'll need to provide one or several NDJSON (New line delimiter JSON) file(s). +Indexing uses bulks for optimal performances. + +Example: + +```shell +> arlas_cli indices \ + --config local \ + data index_name path/to/data.json +``` + +!!! warning + If the index already contains data, the data is added to the index. + + To reindex the same data, delete the index, and do not forget to recreate it with the correct mapping before ingesting the data + +!!! note "--bulk" + Indexing uses bulks for optimal performances. + + The default size of bulk is 5000 and can be changed with the `--bulk` option + +## list + +To list the available ES indices, simply use the `indices list` sub-function. No arguments are required. + + +```shell +> !!!execute arlas_cli indices --config local list --help +``` + +### List available ES indices + +It displays for each ES index its status, the number of elements it contains and the size of the index. + +Example: + +```shell +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| index_name | open | 100 | 1mb | ++--------------+--------+-------+--------+ +``` + +## describe + +Once the index is created, the description of the fields it contains (corresponding to the mapping) can be displayed with the `indices describe` sub function: + + +```shell +> !!!execute arlas_cli indices --config local data --help +``` + +### Describe the index mapping + +For a given index, the description of its fields and their type can be displayed. + +For example: + +```shell +> arlas_cli indices --config local describe index_name ++------------------+-----------+ +| field name | type | ++------------------+-----------+ +| field_keyword | keyword | +| field_point | geo_point | +| field_long | long | +| field_shape | geo_shape | +| field_double | double | +| field_date | date | +| field_text | text | +| field_object | object | +| field_boolean | boolean | ++------------------+-----------+ + +``` + +## sample + +The first rows of the data contained in an index can be displayed with the `indices sample` sub function. + + +```shell +> !!!execute arlas_cli indices --config local delete --help + +``` + +### Visualize few rows of your dataset + +For a given index `index_name`, the first rows of data can be displayed as a valid json dictionary. + +!!! note --size + The number of rows to display (default 100) can be changed + + Example: + + ```shell + > arlas_cli indices --config local sample index_name --size 10 + ``` + +By default, the json representation of the data is pretty printed (clear indentation and one line per field) + +!!! note --no-pretty + The pretty printing can be deactivated and data is displayed in a compact way + + Example: + + ```shell + > arlas_cli indices --config local sample index_name --no-pretty + ``` + +## delete + +The ES index can be deleted with `indices delete` sub command to free space on ES cluster. + + +```shell +> !!!execute arlas_cli indices --config local delete --help +``` + +### Delete an ES index + +To delete an ES index `index_name` on `local` configuration, run the following command: + + +```shell +> arlas_cli indices --config local delete index_name +``` +!!! warning + Delete an index is not reversible. Make sure you don't lost any data. + +!!! Note Delete is not always allowed + By default, it is not allowed to delete an index for a given configuration. + + To allow deleting, [edit the configuration file](configuration.md) and set `allow_delete` to `True`. + +!!! tip "Good practice: Set admin confs" + For a given ARLAS deployment, it is advised to set two configurations, with only the admin one with `allow_delete` at `True`. + + For example: + ``` + +----------------------+------------------------------------------+ + | name | url | + +----------------------+------------------------------------------+ + | cloud.arlas.io | https://cloud.arlas.io/arlas/server | + | cloud.arlas.io-admin | https://cloud.arlas.io/arlas/server | + | local | http://localhost/arlas | + +----------------------+------------------------------------------+ + ``` + + Here the configuration `--config cloud.arlas.io-admin` has to be used to delete any index. + \ No newline at end of file diff --git a/docs/docs/indices_1.md b/docs/docs/indices_1.md deleted file mode 100644 index 7ee130e..0000000 --- a/docs/docs/indices_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## List index management commands - - -```shell diff --git a/docs/docs/indices_2.md b/docs/docs/indices_2.md deleted file mode 100644 index f7c32bc..0000000 --- a/docs/docs/indices_2.md +++ /dev/null @@ -1,9 +0,0 @@ -``` - -## Generate mapping from a data file - - -To generate a mapping, you need to provide a NDJSON file (New line delimiter JSON). The first N lines are used for infering the mapping. If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. Once you're happy with the mapping, you can directly push it on elasticsearch (`--push-on`). - - -```shell diff --git a/docs/docs/indices_3.md b/docs/docs/indices_3.md deleted file mode 100644 index 9c8b195..0000000 --- a/docs/docs/indices_3.md +++ /dev/null @@ -1,6 +0,0 @@ -``` - -## Create an index with its mapping - - -```shell diff --git a/docs/docs/indices_4.md b/docs/docs/indices_4.md deleted file mode 100644 index 59a9079..0000000 --- a/docs/docs/indices_4.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Index data - -For indexing data, you'll need to provide one ore several NDJSON file(s). Indexing uses bulks for optimal performances. - - -```shell diff --git a/docs/docs/indices_5.md b/docs/docs/indices_5.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/indices_5.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/install.md.template b/docs/docs/install.md.template new file mode 100644 index 0000000..67fe9cd --- /dev/null +++ b/docs/docs/install.md.template @@ -0,0 +1,76 @@ +# Installation + +## Prerequisite + +- python 3.10 +- pip + +## Install + +### Python Virtual environment + +Particularly on Windows, it is advised to create a python virtual environment with [venv](https://docs.python.org/3.10/library/venv.html#creating-virtual-environments) before installing `arlas_cli` + +- Create the environment: +```shell +python -m venv env_arlas_cli +``` + +- Activate the environment: +=== "Linux/Mac" + ```bash + source env_arlas_cli/bin/activate (Can be added to .bashrc or .zshrc) + ``` +=== "Windows" + with cmd: + ```cmd + .\env_arlas_cli\Scripts\Activate.bat + ``` + with PowerShell: + ```PowerShell + .\env_arlas_cli\Scripts\Activate.ps1 + ``` + + !!! tip + On Microsoft Windows, it may be required to enable the Activate.ps1 script by setting the execution policy for the user. + + You can do this by issuing the following PowerShell command: + ```PowerShell + Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser + ``` + See [venv documentation](https://docs.python.org/3.10/library/venv.html#creating-virtual-environments) for more information. + +### Install arlas_cli +Install `arlas_cli` with pip: + + +```shell +> pip install arlas_cli +``` + +!!! success + + In a new terminal, you should be able to run it: + + + ```shell + > arlas_cli --version + X.Y.Z + ``` + + The version of `arlas_cli` is `X.Y.Z` where `X` is ARLAS Stack main version and `Y.Z` is the increment of arlas_cli version. + +## Run ARLAS Locally + +`arlas_cli` is meant to interact with a deployed ARLAS stack. + +To run the simplest ARLAS stack and elasticsearch on the local machine, clone the [ARLAS Stack Exploration](https://github.com/gisaia/ARLAS-Exploration-stack) project and run the stack: + + +```shell +> git clone https://github.com/gisaia/ARLAS-Exploration-stack.git +> cd ARLAS-Exploration-stack +> ./start.sh +``` + +More details about deployment can be found on [ARLAS Stack Exploration project](https://github.com/gisaia/ARLAS-Exploration-stack). diff --git a/docs/docs/persist.md b/docs/docs/persist.md index 90ccdc7..225b1d5 100644 --- a/docs/docs/persist.md +++ b/docs/docs/persist.md @@ -6,7 +6,7 @@ ARLAS Persistence allows you to place files within zones. A zone is visible by g ```shell > arlas_cli persist --help - Usage: python -m arlas.cli.cli persist [OPTIONS] COMMAND [ARGS]... + Usage: arlas_cli persist [OPTIONS] COMMAND [ARGS]... ╭─ Options ──────────────────────────────────────────────────────────╮ │ * --config TEXT Name of the ARLAS configuration to use │ diff --git a/docs/docs/persist.md.template b/docs/docs/persist.md.template new file mode 100644 index 0000000..8ce8d1d --- /dev/null +++ b/docs/docs/persist.md.template @@ -0,0 +1,88 @@ +## Manage persistence + +ARLAS Persistence allows you to place files within zones. A zone is visible by groups. It is also possible to set the access: writers, readers and whether it is public or not. + + +```shell +> !!!execute arlas_cli persist --help +``` + +## ARLAS Persistence + +### Add an entry + + +```shell +> arlas_cli persist \ + --config local \ + add path/to/config.json config.json \ + --name dashboard_name +32d2624b-d7cd-11ee-9a91-0242ac130004 +``` + +### Describe an entry + + +```shell +> arlas_cli persist --config local describe 32d2624b-d7cd-11ee-9a91-0242ac130004 ++------------------+--------------------------------------+ +| metadata | value | ++------------------+--------------------------------------+ +| ID | 6a415cec-d7cd-11ee-9a91-0242ac130004 | +| ispublic | None | +| last_update_date | 1709298774600 | +| name | courses_dashboard | +| organization | None | +| owner | anonymous | +| updatable | True | +| zone | config.json | ++------------------+--------------------------------------+ +``` + +### Get an entry value + + +```shell +> arlas_cli persist --config local get 32d2624b-d7cd-11ee-9a91-0242ac130004 +{ + "arlas": { + "web": { + "contributors": [ +... +} +``` + +### List entries within a zone + + +```shell +> arlas_cli persist --config local zone config.json ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| id | name | zone | last_update_date | owner | ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| 66984014-d0a1-11ee-b41c-0242ac190004 | Courses | config.json | 1708510231303 | anonymous | +... ++--------------------------------------+-------------------+-------------+------------------+-----------+ +``` + + +### List groups accessing a zone + + +```shell +> arlas_cli persist --config local groups config.json ++--------------+ +| group | ++--------------+ +| group/public | ++--------------+ +``` + + +### Delete an entry + + +```shell +> arlas_cli persist --config local delete 32d2624b-d7cd-11ee-9a91-0242ac130004 +Resource 32d2624b-d7cd-11ee-9a91-0242ac130004 deleted. +``` diff --git a/docs/docs/persist_1.md b/docs/docs/persist_1.md deleted file mode 100644 index 728cd7c..0000000 --- a/docs/docs/persist_1.md +++ /dev/null @@ -1,6 +0,0 @@ -## Manage persistence - -ARLAS Persistence allows you to place files within zones. A zone is visible by groups. It is also possible to set the access: writers, readers and whether it is public or not. - - -```shell diff --git a/docs/docs/persist_2.md b/docs/docs/persist_2.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/persist_2.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/started.md.template b/docs/docs/started.md.template new file mode 100644 index 0000000..662dacf --- /dev/null +++ b/docs/docs/started.md.template @@ -0,0 +1,380 @@ +# Getting started with ARLAS CLI + +## Install arlas_cli + +To install `arlas_cli`, run the following command from the command line: + +```shell +pip install arlas_cli +``` + +For more details, in particular for installation on Microsoft Windows, see the [Installation Guide](install.md#installation). + + +## Configuration + +!!! warning "Prerequisite" + For running the various examples bellow, ARLAS and elasticsearch must be running on the local machine: + + Clone the [ARLAS Stack Exploration](https://github.com/gisaia/ARLAS-Exploration-stack) project and run `./start.sh` at project root. + +By default, the local configuration is created during the installation. + +To list the available configurations, run the following command from the command line: + + +```shell +> arlas_cli confs list ++----------------------+------------------------------------------+ +| name | url | ++----------------------+------------------------------------------+ +| local | http://localhost/arlas | ++----------------------+------------------------------------------+ +``` + +!!! success + At least the `local` configuration has to be here + +It will be used in the tutorial as `--config local` + +For more details, see the [Configuration Guide](configuration.md#configuration). + +## Tutorial + +In the following tutorial, you will see how to: + +- Generate an elasticsearch mapping based on json objects +- Add the mapping in elasticsearch +- List the elasticsearch indices +- Add (index) data in the elasticsearch index +- Get the structure of the mapping +- Add a collection in ARLAS +- List the ARLAS collections +- Get the structure of a ARLAS collection +- Delete a collection from ARLAS +- Delete a mapping from elasticsearch +- Add, describe, get, list and delete an entry from ARLAS Persistence +- List, create, describe and delete a configuration for `arlas_cli` + +... with the `arlas_cli` command line only! + +!!! warning "Prerequisite" + First, let's get a sample data file: + + ```shell + > curl -X GET https://raw.githubusercontent.com/gisaia/arlas-cli/master/tests/sample.json -o sample.json + ``` + The downloaded `sample.json` contains a sample of processed AIS data. + + +### Generate the elasticsearch mapping + +Writing the elasticsearch mapping for an index can be laborious. `arlas_cli` does it for you. +`arlas_cli` can inspect a NDJSON file (one json object per line) and generate the corresponding elasticsearch mapping file. + +To generate the mapping file based on that sample, run the following command: + +```shell +> arlas_cli indices \ + --config local \ + mapping sample.json +``` + +!!! tip "--nb_lines" + By default, the `indices mapping` function uses the first two rows to infer mapping. + If a field is not present in the first rows, it will not appear in the mapping. + + Make sure to take enough rows into account to get all the fields with the option `--nb_lines` + + +By inspecting the mapping, we notice that the three timestamps are not identified as datetime by `arlas_cli`. + +!!! info "--field-mapping" + The `--field-mapping` option allows to overwrite the detected type. + +To generate the mapping with forced types, run the following command: + +```shell +> arlas_cli indices \ + --config local \ + mapping sample.json \ + --field-mapping track.timestamps.center:date-epoch_second \ + --field-mapping track.timestamps.start:date-epoch_second \ + --field-mapping track.timestamps.end:date-epoch_second \ + --nb-lines 20 +``` + +The three timestamps are now well identified as datetime. + +For more details, see the [Indices Mapping Guide](indices.md#generate-mapping-from-a-data-file). + + +### Create an empty index from inferred mapping + +Once the inferred mapping is fine, an elasticsearch index based on this mapping has to be created. + +!!! info "--push-on" + In the previous `arlas_cli indices mapping` command, the `--push-on` option registers the mapping in the specified index. + + An empty index is created with the inferred mapping + +To create the associated empty index `ais_courses` with the `--push-on` option, run the following command: + +```shell +> arlas_cli indices \ + --config local \ + mapping sample.json \ + --field-mapping track.timestamps.center:date-epoch_second \ + --field-mapping track.timestamps.start:date-epoch_second \ + --field-mapping track.timestamps.end:date-epoch_second \ + --push-on ais_courses +``` + +For more details, see the [Indices Creation Guide](indices.md#create-an-index-with-its-mapping). + +### Inspect the created indices + +To check that the index has been created, list the existing index: + + +```shell +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| ais_courses | open | 0 | 249b | ++--------------+--------+-------+--------+ +``` + +!!! success + The `ais_courses` index exists and contains 0 elements + +To describe the fields of the index, use the `arlas_cli indices describe` command: + + +```shell +> arlas_cli indices --config local describe ais_courses ++----------------------------------------------------+-----------+ +| field name | type | ++----------------------------------------------------+-----------+ +| arrival.address.country | keyword | +| arrival.address.port | keyword | +| arrival.location | geo_point | +... +| track.timestamps.start | date | +| track.trail | geo_shape | +| track.trail_geohashes_6 | geo_point | +| track.visibility.proportion | double | ++----------------------------------------------------+-----------+ +``` + +It corresponds to the inferred mapping. + +### Add data to index + +To add data to the created `ais_courses` index, use the `arlas_cli indices data` command with the data file `sample.json` + +```shell +> arlas_cli indices --config local data ais_courses sample.json +``` + +To check that data has been correctly indexed, inspect the indices with: + + +```shell +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| ais_courses | open | 100 | 1mb | ++--------------+--------+-------+--------+ +``` + +!!! success + The `ais_courses` index now contains 100 elements + +### Create a collection + +To explore the data in ARLAS, a collection has to be defined on top on an index. + +To create a `ais_courses` collection based on the `ais_courses` index, run the following command: + + +```shell +> arlas_cli collections \ + --config local \ + create ais_courses \ + --index ais_courses \ + --display-name "AIS Courses" \ + --id-path track.id \ + --centroid-path track.location \ + --geometry-path track.trail \ + --date-path track.timestamps.center +``` + +The `--index` option define the index to use and the `--display-name` define a pretty name used for collection in ARLAS. + +Several options define the data structure: + +- `--id-path`: The data field used as unique id of each element + +- `--centroid-path`: The data field containing a point geometry summarizing the location of each element (used for aggregation) + +- `--geometry-path`: The data field containing a geometry to represent the element (can be point, linestring, polygon) + +- `--date-path`: The data field containing the date associated to each element (used for timeline) + +For more details, see the [Collection Creation Guide](collections.md#create-a-collection). + +### Inspect the created collections + +To list the available collections, run the following command: + + +```shell +> arlas_cli collections --config local list ++-------------+-------------+ +| name | index | ++-------------+-------------+ +| ais_courses | ais_courses | ++-------------+-------------+ +``` + +!!! success + The `ais_courses` collection is now created + +To describe the fields of the collection, use the `arlas_cli collections describe` command: + + +```shell +> arlas_cli collections --config local describe ais_courses ++----------------------------------------------------+-----------+ +| field name | type | ++----------------------------------------------------+-----------+ +| arrival.address.country | KEYWORD | +| arrival.address.port | KEYWORD | +| arrival.location | GEO_POINT | +| arrival.stop_after.duration_s | LONG | +... +| track.timestamps.start | DATE | +| track.trail | GEO_SHAPE | +| track.trail_geohashes_6 | GEO_POINT | +| track.visibility.proportion | FLOAT | ++----------------------------------------------------+-----------+ +``` + +It corresponds to the mapping of the data within the collection + +### Delete an index + +To remove the indexed data from the local elasticsearch instance, remove the index with following command: + +```shell +> arlas_cli indices --config local delete ais_courses +``` + +Check that `ais_courses` index no longer exists: + +```shell +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | ++--------------+--------+-------+--------+ +``` + +!!! tip + Before reindexing data, do not forget to create [recreate the empty index from inferred mapping](#create-an-empty-index-from-inferred-mapping) + + Collection do not need to be declared again + +### Delete a collection + +To delete the `ais_courses` course collection + +```shell +> arlas_cli collections --config local delete courses +``` + +## ARLAS Persistence + +### Add an entry + + +```shell +> arlas_cli persist --config local add ../arlas-stacks4tests/conf/config.json config.json --name courses_dashboard +32d2624b-d7cd-11ee-9a91-0242ac130004 +``` + +### Describe an entry + + +```shell +> arlas_cli persist --config local describe 32d2624b-d7cd-11ee-9a91-0242ac130004 ++------------------+--------------------------------------+ +| metadata | value | ++------------------+--------------------------------------+ +| ID | 6a415cec-d7cd-11ee-9a91-0242ac130004 | +| ispublic | None | +| last_update_date | 1709298774600 | +| name | courses_dashboard | +| organization | None | +| owner | anonymous | +| updatable | True | +| zone | config.json | ++------------------+--------------------------------------+ +``` + +### Get an entry value + + +```shell +> arlas_cli persist --config local get 32d2624b-d7cd-11ee-9a91-0242ac130004 +{ + "arlas": { + "web": { + "contributors": [ +... +} +``` + +### List entries within a zone + + +```shell +> arlas_cli persist --config local zone config.json ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| id | name | zone | last_update_date | owner | ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| 66984014-d0a1-11ee-b41c-0242ac190004 | Courses | config.json | 1708510231303 | anonymous | +... ++--------------------------------------+-------------------+-------------+------------------+-----------+ +``` + + +### List groups accessing a zone + + +```shell +> arlas_cli persist --config local groups config.json ++--------------+ +| group | ++--------------+ +| group/public | ++--------------+ +``` + + +### Delete an entry + + +```shell +> arlas_cli persist --config local delete 32d2624b-d7cd-11ee-9a91-0242ac130004 +Resource 32d2624b-d7cd-11ee-9a91-0242ac130004 deleted. +``` + + + diff --git a/docs/docs/version.md.template b/docs/docs/version.md.template new file mode 100644 index 0000000..a4b7b34 --- /dev/null +++ b/docs/docs/version.md.template @@ -0,0 +1,2 @@ +ARLAS Version: +26.1.1 diff --git a/docs/inject_commands.py b/docs/inject_commands.py new file mode 100644 index 0000000..8357607 --- /dev/null +++ b/docs/inject_commands.py @@ -0,0 +1,16 @@ +import sys +from pathlib import Path +import os + +for source in sys.argv[1:]: + print(source) + target = source.removesuffix(".template") + with open(source) as f, open(target, 'w') as f2: + for line in f.readlines(): + if line.startswith("> !!!execute "): + cmd = line.split("> !!!execute ")[1] + result = os.popen("stty cols 70; " + cmd).read() + f2.write("> " + cmd) + f2.write(result) + else: + f2.write(line) diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 704d7fd..c31b69f 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -43,5 +43,4 @@ plugins: - termynal: prompt_literal_start: - ">" - - mkdocs-versioning: - version: 26.0.0 \ No newline at end of file + - search diff --git a/scripts/mkdocs.sh b/scripts/mkdocs.sh index a8e713d..431c213 100755 --- a/scripts/mkdocs.sh +++ b/scripts/mkdocs.sh @@ -1,84 +1,7 @@ #!/bin/bash set -e -# INDICES -export PYTHONPATH=`pwd`/target/src -stty cols 70 - -echo "ARLAS Version:" > docs/docs/version.md -cat version.txt >> docs/docs/version.md - -cat docs/docs/indices_1.md > docs/docs/indices.md -# general -echo "> arlas_cli indices --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --help >> docs/docs/indices.md -cat docs/docs/indices_2.md >> docs/docs/indices.md - -# mapping -echo "> arlas_cli indices --config local mapping --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local mapping --help >> docs/docs/indices.md -cat docs/docs/indices_3.md >> docs/docs/indices.md - -# create -echo "> arlas_cli indices --config local create --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local create --help >> docs/docs/indices.md -cat docs/docs/indices_4.md >> docs/docs/indices.md - -# data -echo "> arlas_cli indices --config local data --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local data --help >> docs/docs/indices.md -cat docs/docs/indices_5.md >> docs/docs/indices.md - - -# COLLECTIONS -cat docs/docs/collections_1.md > docs/docs/collections.md -# general -echo "> arlas_cli collections --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --help >> docs/docs/collections.md -cat docs/docs/collections_2.md >> docs/docs/collections.md - -# create -echo "> arlas_cli collections --config local create --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --config local create --help >> docs/docs/collections.md -cat docs/docs/collections_3.md >> docs/docs/collections.md - -# describe -echo "> arlas_cli collections --config local describe --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --config local describe --help >> docs/docs/collections.md -cat docs/docs/collections_4.md >> docs/docs/collections.md - - -# PERSISTENCE -cat docs/docs/persist_1.md > docs/docs/persist.md -# general -echo "> arlas_cli persist --help" >> docs/docs/persist.md -python3.10 -m arlas.cli.cli persist --help >> docs/docs/persist.md -cat docs/docs/persist_2.md >> docs/docs/persist.md - - -# IAM -cat docs/docs/iam_1.md > docs/docs/iam.md -echo "> arlas_cli iam --config local orgs --help" >> docs/docs/iam.md -python3.10 -m arlas.cli.cli iam --config local orgs --help >> docs/docs/iam.md -cat docs/docs/iam_2.md >> docs/docs/iam.md -echo "> arlas_cli iam --config local users --help" >> docs/docs/iam.md -python3.10 -m arlas.cli.cli iam --config local users --help >> docs/docs/iam.md -cat docs/docs/iam_3.md >> docs/docs/iam.md - - -# CONFIGURATIONS -cat docs/docs/confs_1.md > docs/docs/confs.md - -# general -echo "> arlas_cli confs --help" >> docs/docs/confs.md -python3.10 -m arlas.cli.cli confs --help >> docs/docs/confs.md -cat docs/docs/confs_2.md >> docs/docs/confs.md - -# create -echo "> arlas_cli confs create --help" >> docs/docs/confs.md -python3.10 -m arlas.cli.cli confs create --help >> docs/docs/confs.md -cat docs/docs/confs_3.md >> docs/docs/confs.md - +python3.10 docs/inject_commands.py docs/docs/*.template pip3.10 install mkdocs-material termynal mkdocs build -f docs/mkdocs.yml -d ../target/docs