diff --git a/README.md b/README.md index 1fc847e..366855b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # arlas_cli -ARLAS Command Line is for managing collections, indices and ARLAS configurations. +__ARLAS Command Line__ (`arlas_cli`) is for managing the ARLAS data and configurations. See the documentation at https://gisaia.github.io/arlas_cli/ + +## Developer section + +To run the commands locally from the project root, simply replace `arlas_cli` by `python3.10 -m arlas.cli.cli` \ No newline at end of file diff --git a/arlas/cli/configurations.py b/arlas/cli/configurations.py index ab5ab2c..64f3245 100644 --- a/arlas/cli/configurations.py +++ b/arlas/cli/configurations.py @@ -78,7 +78,7 @@ def create_configuration( @configurations.command(help="Add a configuration for ARLAS Cloud", name="login") def login( - auth_login: str = typer.Argument(help="login"), + auth_login: str = typer.Argument(help="ARLAS login"), elastic_login: str = typer.Argument(help="Elasticsearch login"), elastic: str = typer.Argument(help="Elasticsearch url"), auth_org: str = typer.Option(default=None, help="ARLAS IAM Organization, default is your email domain name"), diff --git a/docs/docs/arlas_cloud.md b/docs/docs/arlas_cloud.md deleted file mode 100644 index 2672444..0000000 --- a/docs/docs/arlas_cloud.md +++ /dev/null @@ -1,60 +0,0 @@ -To configure `arlas_cli` for your cloud.arlas.io account: - -First, set the following environment variables by changing appropriately `SET_THIS_VALUE`: -```shell -export MY_ORGANIZATION=SET_THIS_VALUE -export ARLAS_USER=SET_THIS_VALUE -export ARLAS_PWD=SET_THIS_VALUE -export ELASTIC_ENDPOINT=SET_THIS_VALUE -export ELASTIC_USER=SET_THIS_VALUE -export ELASTIC_PWD=SET_THIS_VALUE -``` - -Then run the command `arlas_cli confs create cloud.arlas.io` with all the parameters: - -```shell -arlas_cli confs \ - create cloud.arlas.io \ - --server "https://cloud.arlas.io/arlas/server" \ - --headers "arlas-org-filter:${MY_ORGANIZATION}" \ - --headers "Content-Type:application/json" \ - --no-allow-delete \ - --auth-token-url https://cloud.arlas.io/arlas/iam/session \ - --auth-login "${ARLAS_USER}" \ - --auth-password "${ARLAS_PWD}" \ - --auth-headers "Content-Type:application/json;charset=utf-8" \ - --auth-org "${MY_ORGANIZATION}" \ - --elastic "${ELASTIC_ENDPOINT}" \ - --elastic-headers "Content-Type:application/json" \ - --elastic-login "${ELASTIC_USER}" \ - --elastic-password "${ELASTIC_PWD}" \ - --elastic-headers "Content-Type:application/json" \ - --auth-headers "Content-Type:application/json;charset=utf-8" \ - --persistence "https://cloud.arlas.io/arlas/persistence" \ - --persistence-headers "Content-Type:application/json" \ - --auth-arlas-iam -``` - -Check the configuration exists: - -```shell -arlas_cli confs list -``` - -You can now list the collections: - -```shell -arlas_cli collections --config cloud.arlas.io list -``` - -List the indices: - -```shell -arlas_cli indices --config cloud.arlas.io list -``` - -List the persisted elements: - -```shell -arlas_cli persist --config cloud.arlas.io groups config.json -``` diff --git a/docs/docs/collections.md b/docs/docs/collections.md index f474d27..5c8f766 100644 --- a/docs/docs/collections.md +++ b/docs/docs/collections.md @@ -1,110 +1,451 @@ -## List collections management commands +## ARLAS Collections + +ARLAS collections are built on top of [elasticsearch (ES) index](concepts.md#es-index). They describe the basic data structure for its visualization and elements needed for the data access policy. + +`arlas_cli` provide tools to manage the ARLAS collections with the `collections` command. + +**List collections management commands** ```shell > arlas_cli collections --help - - Usage: python -m arlas.cli.cli collections [OPTIONS] COMMAND - [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --config TEXT Name of the ARLAS configuration to use from │ -│ your configuration file │ -│ (/Users/gaudan/.arlas/cli/configuration.yam… │ -│ [default: None] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ count Count the number of hits within a collection (or all │ -│ collection if not provided) │ -│ create Create a collection │ -│ delete Delete a collection │ -│ describe Describe a collection │ -│ list List collections │ -│ name Set the collection display name │ -│ private Set collection visibility to private │ -│ public Set collection visibility to public │ -│ sample Display a sample of a collection │ -│ set_alias Set the field display name │ -│ share Share the collection with the organisation │ -│ unshare Unshare the collection with the organisation │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli collections [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --config TEXT Name of the ARLAS configuration to use from your │ +│ configuration file │ +│ (/Users/gaudan/.arlas/cli/configuration.yaml). │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ count Count the number of hits within a collection (or all collection │ +│ if not provided) │ +│ create Create a collection │ +│ delete Delete a collection │ +│ describe Describe a collection │ +│ list List collections │ +│ name Set the collection display name │ +│ private Set collection visibility to private │ +│ public Set collection visibility to public │ +│ sample Display a sample of a collection │ +│ set_alias Set the field display name │ +│ share Share the collection with the organisation │ +│ unshare Unshare the collection with the organisation │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` -## Create a collection +## create + +An ARLAS Collection is linked to an index or even an index pattern. -You can create a collection from scratch or by providing a collection model (`--model`). The command line options let you specify how the index should be used by the collection. The visibility options (`--public`, `--owner` and `--orgs`) allow you to choose who can access the collection. +The command line options let you specify how the index should be used by the collection. ```shell > arlas_cli collections --config local create --help - - Usage: python -m arlas.cli.cli collections create [OPTIONS] - COLLECTION - - Create a collection - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * collection TEXT Collection's name [default: None] │ -│ [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --model TEXT Name of the model within │ -│ your configuration, or URL │ -│ or file path │ -│ [default: None] │ -│ --index TEXT Name of the index │ -│ referenced by the │ -│ collection │ -│ [default: None] │ -│ --display-name TEXT Display name of the │ -│ collection │ -│ [default: None] │ -│ --public --no-public Whether the collection is │ -│ public or not │ -│ [default: no-public] │ -│ --owner TEXT Organisation's owner │ -│ [default: None] │ -│ --orgs TEXT List of organisations │ -│ accessing the collection │ -│ --id-path TEXT Override the JSON path to │ -│ the id field. │ -│ [default: None] │ -│ --centroid-path TEXT Override the JSON path to │ -│ the centroid field. │ -│ [default: None] │ -│ --geometry-path TEXT Override the JSON path to │ -│ the geometry field. │ -│ [default: None] │ -│ --date-path TEXT Override the JSON path to │ -│ the date field. │ -│ [default: None] │ -│ --help Show this message and │ -│ exit. │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli collections create [OPTIONS] COLLECTION + + Create a collection + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --model TEXT Name of the model within your │ +│ configuration, or URL or file path │ +│ [default: None] │ +│ --index TEXT Name of the index referenced by │ +│ the collection │ +│ [default: None] │ +│ --display-name TEXT Display name of the collection │ +│ [default: None] │ +│ --public --no-public Whether the collection is public │ +│ or not │ +│ [default: no-public] │ +│ --owner TEXT Organisation's owner │ +│ [default: None] │ +│ --orgs TEXT List of organisations accessing │ +│ the collection │ +│ --id-path TEXT Override the JSON path to the id │ +│ field. │ +│ [default: None] │ +│ --centroid-path TEXT Override the JSON path to the │ +│ centroid field. │ +│ [default: None] │ +│ --geometry-path TEXT Override the JSON path to the │ +│ geometry field. │ +│ [default: None] │ +│ --date-path TEXT Override the JSON path to the date │ +│ field. │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +!!! note "Collection visibility" + + The visibility options (`--public`, `--private`, `--owner` and `--orgs`) allow you to choose who can access the collection. + +!!! note "--owner" + + If ARLAS is used with ARLAS IAM (for example with ARLAS Cloud), then a collection must be associated to an organisation. + + Example: + + `--owner your.organisation.com` + + By default, the organisation referenced in your `config` is used as owner. + + +### Create an ARLAS collection + +The collection has to reference an available ES index or index pattern with the [arlas_cli configuration](confs.md#configurations). + +The index option sets the targeted index:`--index index_name` + +!!! tip "Index pattern" + + The collection can target an index pattern. + Instead of an index name, the target is an expression using `*` to reference multiple ES indices. + + Example: + + `--index index_name_prefix*` + + All the indices have to share exactly the same data mapping. + + The data contained in all referenced indices are then explorable together in ARLAS. + + +The collection also describe a basic structure for spatio-temporel datasets: + +- **ID path**: A data field containing unique element identifier. `--id-path unique_id_field` +- **Centroid path**: A data field containing a point geometry used for aggregations `--centroid-path point_geom_field` +- **Geometry path**: A data field containing a geometry representing the element `--geometry-path element_geom_field` +- **Date path**: A data field containing the date associated to each element `--date-path date_field` + +!!! tip "Pretty name" + + A pretty name for the collection can be set at the creation with the `display-name` option + + Example: + + `--display-name "Pretty Collection Name"` + + +To create the collection, run the following command: + + +```shell +> arlas_cli collections \ + --config {local} \ + create {collection_name} \ + --index {index_name} \ + --id-path {unique_id_field} \ + --centroid-path {point_geom_field} \ + --geometry-path {element_geom_field} \ + --date-path {date_field} \ + --display-name {"Pretty Collection Name"} +``` + +## name + +The collection can be defined by a pretty name. It can be set with `name` subcommand: + +```shell +> arlas_cli collections --config local name --help + + Usage: arlas_cli collections name [OPTIONS] COLLECTION NAME + + Set the collection display name + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +│ * name TEXT The display name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Set a pretty name for the collection + +The pretty name can be used in ARLAS to display the collection. It can also be set by the `--display-name` option at the [collection creation](#create). + +To set it with the `name` subcommand: +```shell +> arlas_cli collections \ + --config {local} \ + name {collection_name} {"Pretty Collection Name"} ``` -## Describe a collection +## set_alias + +The data fields are sometimes not very suitable in ARLAS Exploration dashboards. You can set aliases to improve their display in the interface. + + +```shell +> arlas_cli collections --config local set_alias --help + + Usage: arlas_cli collections set_alias [OPTIONS] COLLECTION FIELD_PATH + [DISPLAY_NAME] + + Set the field display name + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] │ +│ [required] │ +│ * field_path TEXT The field path [default: None] │ +│ [required] │ +│ display_name [DISPLAY_NAME] The field's display name. If none │ +│ provided, then the alias is removed │ +│ if it existed │ +│ [default: None] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` +### Set a pretty name for a data field + +Each field of the data has a raw name. It can be replaced by a pretty name to display. For example: +```shell +> arlas_cli collections \ + --config {local} \ + set_alias {collection_name} \ + {raw_field_name} {"Pretty Field Name (unit)"} +``` + +## list + +### List available collections + +You can access the list of available collections with the `list` subcommand: + + +```shell +> arlas_cli collections --config local set_alias --help + +Usage: arlas_cli collections list [OPTIONS] + + List collections + +Options: + --help Show this message and exit. + +``` + + +## describe + +### Describe a collection -This command line provides a description of the collection structure (fields) and of its metadata. +The `describe` command line provides a description of the collection's structure (fields) and its metadata. ```shell > arlas_cli collections --config local describe --help - - Usage: python -m arlas.cli.cli collections describe [OPTIONS] - COLLECTION - - Describe a collection - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * collection TEXT Collection's name [default: None] │ -│ [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli collections describe [OPTIONS] COLLECTION + + Describe a collection + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` + +## count + +### Count the number of element within a collection + +The `count` command show the total number of elements (data rows) accessible in a collection. + + +```shell +> arlas_cli collections --config local count --help + + Usage: arlas_cli collections count [OPTIONS] [COLLECTION] + + Count the number of hits within a collection (or all collection if not + provided) + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ collection [COLLECTION] Collection's name [default: None] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## sample + +### Display a sample of the collection data + +The `sample` command show few data rows accessible in a collection. + + +```shell +> arlas_cli collections --config local sample --help + + Usage: arlas_cli collections sample [OPTIONS] COLLECTION + + Display a sample of a collection + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --pretty --no-pretty [default: pretty] │ +│ --size INTEGER [default: 10] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +!!! note + The number of rows to display can be set with `--size` option + +## private + +By default, a collection is private, it can only be seen by the members of the owner or shared organisation. + +In that case, an ARLAS user has to be logged and have the correct authorisations. + +### Set the collection as private + +To switch a collection from **public** to **private**, use the `private` command: + + +```shell +> arlas_cli collections --config local private --help + + Usage: arlas_cli collections private [OPTIONS] COLLECTION + + Set collection visibility to private + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## public + +A public collection can be accessed in ARLAS dashboards without being logged. It can be used to host demo dashboards for example. + +### Set the collection as public + +To switch a collection from **private** to **public**, use the `public` command: + + +```shell +> arlas_cli collections --config local public --help + + Usage: arlas_cli collections public [OPTIONS] COLLECTION + + Set collection visibility to public + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## share + +A collection can be shared between different organisations to make it available for its users. + +### Share collections between organisations + +A collection can be shared to other organisation with the `share` command: + + +```shell +> arlas_cli collections --config local share --help + + Usage: arlas_cli collections share [OPTIONS] COLLECTION ORGANISATION + + Share the collection with the organisation + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +│ * organisation TEXT Organisation's name [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## unshare + +The right to access a collection can be removed to the users of an organisation. + +### Remove collection access for an organisation + +The access to a collection can be removed with the `unshare` command: + + +```shell +> arlas_cli collections --config local unshare --help + + Usage: arlas_cli collections unshare [OPTIONS] COLLECTION ORGANISATION + + Unshare the collection with the organisation + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT Collection's name [default: None] [required] │ +│ * organisation TEXT Organisation's name [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## delete + +A collection can be deleted. It doesn't delete the data (ES index can still exist) but it will no longer be accessible in ARLAS. + +### Delete a collection + +The collection can be removed with the `delete` command: + + +```shell +> arlas_cli collections --config local delete --help + + Usage: arlas_cli collections delete [OPTIONS] COLLECTION + + Delete a collection + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * collection TEXT collection's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` \ No newline at end of file diff --git a/docs/docs/collections.md.template b/docs/docs/collections.md.template new file mode 100644 index 0000000..80c5101 --- /dev/null +++ b/docs/docs/collections.md.template @@ -0,0 +1,250 @@ +## ARLAS Collections + +ARLAS collections are built on top of [elasticsearch (ES) index](concepts.md#es-index). They describe the basic data structure for its visualization and elements needed for the data access policy. + +`arlas_cli` provide tools to manage the ARLAS collections with the `collections` command. + +**List collections management commands** + + +```shell +> !!!execute arlas_cli collections --help +``` + +## create + +An ARLAS Collection is linked to an index or even an index pattern. + +The command line options let you specify how the index should be used by the collection. + + +```shell +> !!!execute arlas_cli collections --config local create --help +``` + +!!! note "Collection visibility" + + The visibility options (`--public`, `--private`, `--owner` and `--orgs`) allow you to choose who can access the collection. + +!!! note "--owner" + + If ARLAS is used with ARLAS IAM (for example with ARLAS Cloud), then a collection must be associated to an organisation. + + Example: + + `--owner your.organisation.com` + + By default, the organisation referenced in your `config` is used as owner. + + +### Create an ARLAS collection + +The collection has to reference an available ES index or index pattern with the [arlas_cli configuration](confs.md#configurations). + +The index option sets the targeted index:`--index index_name` + +!!! tip "Index pattern" + + The collection can target an index pattern. + Instead of an index name, the target is an expression using `*` to reference multiple ES indices. + + Example: + + `--index index_name_prefix*` + + All the indices have to share exactly the same data mapping. + + The data contained in all referenced indices are then explorable together in ARLAS. + + +The collection also describe a basic structure for spatio-temporel datasets: + +- **ID path**: A data field containing unique element identifier. `--id-path unique_id_field` +- **Centroid path**: A data field containing a point geometry used for aggregations `--centroid-path point_geom_field` +- **Geometry path**: A data field containing a geometry representing the element `--geometry-path element_geom_field` +- **Date path**: A data field containing the date associated to each element `--date-path date_field` + +!!! tip "Pretty name" + + A pretty name for the collection can be set at the creation with the `display-name` option + + Example: + + `--display-name "Pretty Collection Name"` + + +To create the collection, run the following command: + + +```shell +> arlas_cli collections \ + --config {local} \ + create {collection_name} \ + --index {index_name} \ + --id-path {unique_id_field} \ + --centroid-path {point_geom_field} \ + --geometry-path {element_geom_field} \ + --date-path {date_field} \ + --display-name {"Pretty Collection Name"} +``` + +## name + +The collection can be defined by a pretty name. It can be set with `name` subcommand: + + +```shell +> !!!execute arlas_cli collections --config local name --help +``` + +### Set a pretty name for the collection + +The pretty name can be used in ARLAS to display the collection. It can also be set by the `--display-name` option at the [collection creation](#create). + +To set it with the `name` subcommand: +```shell +> arlas_cli collections \ + --config {local} \ + name {collection_name} {"Pretty Collection Name"} +``` + +## set_alias + +The data fields are sometimes not very suitable in ARLAS Exploration dashboards. You can set aliases to improve their display in the interface. + + +```shell +> !!!execute arlas_cli collections --config local set_alias --help +``` +### Set a pretty name for a data field + +Each field of the data has a raw name. It can be replaced by a pretty name to display. For example: +```shell +> arlas_cli collections \ + --config {local} \ + set_alias {collection_name} \ + {raw_field_name} {"Pretty Field Name (unit)"} +``` + +## list + +### List available collections + +You can access the list of available collections with the `list` subcommand: + + +```shell +> arlas_cli collections --config local set_alias --help + +Usage: arlas_cli collections list [OPTIONS] + + List collections + +Options: + --help Show this message and exit. + +``` + + +## describe + +### Describe a collection + +The `describe` command line provides a description of the collection's structure (fields) and its metadata. + + +```shell +> !!!execute arlas_cli collections --config local describe --help +``` + +## count + +### Count the number of element within a collection + +The `count` command show the total number of elements (data rows) accessible in a collection. + + +```shell +> !!!execute arlas_cli collections --config local count --help +``` + +## sample + +### Display a sample of the collection data + +The `sample` command show few data rows accessible in a collection. + + +```shell +> !!!execute arlas_cli collections --config local sample --help +``` + +!!! note + The number of rows to display can be set with `--size` option + +## private + +By default, a collection is private, it can only be seen by the members of the owner or shared organisation. + +In that case, an ARLAS user has to be logged and have the correct authorisations. + +### Set the collection as private + +To switch a collection from **public** to **private**, use the `private` command: + + +```shell +> !!!execute arlas_cli collections --config local private --help +``` + +## public + +A public collection can be accessed in ARLAS dashboards without being logged. It can be used to host demo dashboards for example. + +### Set the collection as public + +To switch a collection from **private** to **public**, use the `public` command: + + +```shell +> !!!execute arlas_cli collections --config local public --help +``` + +## share + +A collection can be shared between different organisations to make it available for its users. + +### Share collections between organisations + +A collection can be shared to other organisation with the `share` command: + + +```shell +> !!!execute arlas_cli collections --config local share --help +``` + +## unshare + +The right to access a collection can be removed to the users of an organisation. + +### Remove collection access for an organisation + +The access to a collection can be removed with the `unshare` command: + + +```shell +> !!!execute arlas_cli collections --config local unshare --help +``` + +## delete + +A collection can be deleted. It doesn't delete the data (ES index can still exist) but it will no longer be accessible in ARLAS. + +### Delete a collection + +The collection can be removed with the `delete` command: + + +```shell +> !!!execute arlas_cli collections --config local delete --help +``` \ No newline at end of file diff --git a/docs/docs/collections_1.md b/docs/docs/collections_1.md deleted file mode 100644 index e65c136..0000000 --- a/docs/docs/collections_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## List collections management commands - - -```shell diff --git a/docs/docs/collections_2.md b/docs/docs/collections_2.md deleted file mode 100644 index 4a7803c..0000000 --- a/docs/docs/collections_2.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Create a collection - -You can create a collection from scratch or by providing a collection model (`--model`). The command line options let you specify how the index should be used by the collection. The visibility options (`--public`, `--owner` and `--orgs`) allow you to choose who can access the collection. - - -```shell diff --git a/docs/docs/collections_3.md b/docs/docs/collections_3.md deleted file mode 100644 index 1fad127..0000000 --- a/docs/docs/collections_3.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Describe a collection - -This command line provides a description of the collection structure (fields) and of its metadata. - - -```shell diff --git a/docs/docs/collections_4.md b/docs/docs/collections_4.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/collections_4.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/concepts.md b/docs/docs/concepts.md new file mode 100644 index 0000000..2b82fa6 --- /dev/null +++ b/docs/docs/concepts.md @@ -0,0 +1,90 @@ +# Concepts + +## ARLAS Dashboards + +An ARLAS Dashboard is an interactive platform to explore datasets. + +![Example: ARLAS dashboard](./images/dashboard.png) +

+Example: ARLAS dashboard +

+ +## ARLAS Hub + +The Hub is the interface where you visualize all the available dashboards and their data collection. + +You can explore and manage the dashboards. + +![Example: ARLAS Hub to manage dashboards](./images/hub.png) +

+Example: ARLAS Hub to manage dashboards +

+ +## ARLAS Builder + +ARLAS Builder is the studio to edit ARLAS Dashboards. +It is a no code interactive interface to configure the geographic layers, the widgets and all elements of the dashboard. + +![Example: ARLAS Builder to configure widgets](./images/builder_widgets.png) +

+Example: ARLAS Builder to configure widgets +

+ +## ARLAS IAM + +ARLAS IAM can handle the authentication of ARLAS users. It has it's own no-code interface to manage who can access the ARLAS ressources. + +![Example: ARLAS IAM to manage users ](./images/iam.png) +

+Example: ARLAS IAM to manage users +

+ + +## Elasticsearch + +[Elasticsearch](https://www.elastic.co/fr/elasticsearch) is a distributed, open-source search and analytics engine built on Apache Lucene. +It allows users to store, search, and analyze large volumes of structured and unstructured data in near real-time. + +ARLAS rely on Elasticsearch, an open-source search engine to perform quick aggregation requests over data. + +## ES index + +An [Elasticsearch index](https://www.elastic.co/blog/what-is-an-elasticsearch-index/) is a logical container that stores and organizes documents, which are the fundamental data units in Elasticsearch. +Each index is made up of one or more shards and can be thought of as similar to a database table, containing related data with a defined structure. +It allows efficient storage, search, and retrieval of data by applying indexing and mapping techniques tailored to specific use cases. + +## ES mapping + +An [Elasticsearch mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) defines how documents and their fields are stored and indexed in an Elasticsearch index. +It specifies the data types (e.g., text, keyword, integer, date) and configurations for each field, influencing how data can be searched, analyzed, and retrieved. +Mappings also allow customization of field behaviors, such as enabling full-text search, controlling case sensitivity, or defining nested structures. +Proper mapping design is crucial for optimizing query performance and ensuring accurate search results. + +## ARLAS collection + +An [ARLAS collection](https://docs.arlas.io/arlas-api-collection/) is a structured data layer within the ARLAS platform, built on top of Elasticsearch indices, that organizes and exposes geospatial and temporal data for exploration, analysis, and visualization. +It serves as a virtual representation of an Elasticsearch index, allowing users to define metadata, filtering rules, and settings specific to their use cases. +Collections enable ARLAS to efficiently handle large-scale, multidimensional data while providing APIs for querying, aggregating, and displaying the information in user-friendly formats. + +## configuration + +An `arlas_cli` configuration is a set of values and credentials to target the ARLAS Stack. + +It is composed of several parts: + +- authorization: The authentication system configuration +- elastic: The link to elasticsearch cluster +- persistence: The link to ARLAS persistence system +- server: The link to ARLAS server + +More details in the [Configuration Guide](configuration.md). + +## persistence / entry + +ARLAS Persistence allows you to store files within "zones" of the ARLAS stack. A zone is visible by groups. + +It is also possible to set the access: writers, readers and whether it is public or not. + +The main persistence zone is the `config.json` which contains the dashboard configurations. + +An entry is an element stored in the persistence. \ No newline at end of file diff --git a/docs/docs/configuration.md b/docs/docs/configuration.md new file mode 100644 index 0000000..defd451 --- /dev/null +++ b/docs/docs/configuration.md @@ -0,0 +1,145 @@ +# Configuration + +`arlas_cli` uses a yaml file for storing various ARLAS and elasticsearch configurations. + +## Initial configuration + +At its first launch, `arlas_cli` will create a first [configuration file](https://raw.githubusercontent.com/gisaia/arlas-cli/master/configuration.yaml) for you. + + +```shell +> arlas_cli --version +X.X.X +Warning : no configuration file found, we created an empty one for +you (~/.arlas/cli/configuration.yaml). +``` + +By default, the file is located in `$HOME/.arlas/cli/configuration.yaml`. + +It contains one ARLAS configuration linked to a local deployment. + +!!! warning + If you used the ARLAS Exploration Stack, it is possible that you already have a directory named `$HOME/.arlas`. + This directory has been created by docker as root. + + The owner of the directory must be changed to the local user (`sudo chown ${USER}: $HOME/.arlas`). + + +## Configurations file + +By default, the command line uses the `${HOME}/.arlas/cli/configuration.yaml` configuration file: + +For example, the default configuration file with only the `local` configuration looks like: +```yaml +arlas: + local: + allow_delete: true + authorization: null + elastic: + location: http://localhost:9200 + ... + persistence: + location: http://localhost/persist + ... + server: + location: http://localhost/arlas + ... +``` + +The `arlas` section contains the different deployment configurations (here only `local`). + +Each deployment configuration is defined by: + +- authorization: The authentication system configuration +- elastic: The link to the elasticsearch cluster +- persistence: The link to ARLAS persistence +- server: The link to ARLAS server + +You can interact with this configuration file directly with the command line itself with the [`arlas_cli confs`](confs.md#configurations) commands: + +- [confs list](confs.md#list): List the available configurations +- [confs describe](confs.md#describe): Describe the content of a configuration +- [confs create](confs.md#create): Create a new configuration +- [confs delete](confs.md#delete): Delete a configuration + +## Custom configuration file path + +It is possible to use a different configuration file than the one placed in your home directory (`$HOME/.arlas/cli/configuration.yaml`): + + +```shell +> arlas_cli --config-file /tmp/config.yaml +Warning : no configuration file found, we created a default one with a 'local' confs accessing local ARLAS exploration stack (/tmp/config.yaml). +``` + +All the arlas_cli commands can then be run on this configuration file, for example to list the available confs: + + +```shell +> arlas_cli --config-file /tmp/config.yaml confs list ++-------+------------------------+ +| name | url | ++-------+------------------------+ +| local | http://localhost/arlas | ++-------+------------------------+ +``` + +## ARLAS Cloud configuration + +If you have an ARLAS cloud account, you can directly create the configurations to access your space with `arlas_cli login`. + +First, set the environment variables provided by Gisaïa and change appropriately `SET_THIS_VALUE` with your own ARLAS user login/password: + +=== "Linux/Mac" + ``` + export MY_ORGANIZATION= + export ARLAS_USER= + export ARLAS_PWD= + export ELASTIC_ENDPOINT= + export ELASTIC_USER= + export ELASTIC_PWD= + ``` + + Then run the command `arlas_cli confs create` with all the parameters to create the `cloud.arlas.io.{USER_NAME}` configuration: + + ```shell + arlas_cli confs \ + login ${ARLAS_USER} ${ELASTIC_USER} ${ELASTIC_ENDPOINT} \ + --auth-password "${ARLAS_PWD}" \ + --auth-org "${MY_ORGANIZATION}" \ + --elastic-password "${ELASTIC_PWD}" \ + --allow-delete + ``` + +=== "Windows PowerShell" + ``` + $env:MY_ORGANIZATION = "" + $env:ARLAS_USER = "" + $env:ARLAS_PWD = "" + $env:ELASTIC_ENDPOINT = "" + $env:ELASTIC_USER = "" + $env:ELASTIC_PWD = "" + ``` + + Then run the command `arlas_cli confs create` with all the parameters to create the `cloud.arlas.io.{USER_NAME}` configuration: + + ```shell + arlas_cli confs ` + login $env:ARLAS_USER $env:ELASTIC_USER $env:ELASTIC_ENDPOINT ` + --auth-password "$env:ARLAS_PWD" ` + --auth-org "$env:MY_ORGANIZATION" ` + --elastic-password "$env:ELASTIC_PWD" ` + --allow-delete + ``` + +Check that the configuration exist: + +```shell +arlas_cli confs list +``` + +You can now, for example, list the available collections: + +```shell +arlas_cli collections list +``` \ No newline at end of file diff --git a/docs/docs/confs.md b/docs/docs/confs.md index e3fcd5d..75189c1 100644 --- a/docs/docs/confs.md +++ b/docs/docs/confs.md @@ -1,110 +1,318 @@ +# Configurations -!!! tip - At its first launch, `arlas_cli` will create a first configuration file for you (`$HOME/.arlas/cli/configuration.yaml`), with two ARLAS configurations, one pointing at a local deployment, one on ARLAS demo (without elasticsearch). -!!! bug - If you used the ARLAS Exploration Stack, it is possible that you already have a directory named `$HOME/.arlas`. This directory has been created by docker as root. The owner of the directory must be changed to the local user (`sudo chown ${USER}: $HOME/.arlas`). +## ARLAS configurations -## List configuration management commands +An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and Elasticsearch. + +See [more information about configuration](configuration.md). + +It is possible, with the `arlas_cli confs` command lines, to manage the ARLAS configurations. + +**List configurations (confs) management commands** ```shell > arlas_cli confs --help - - Usage: python -m arlas.cli.cli confs [OPTIONS] COMMAND [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ create Add a configuration │ -│ delete Delete a configuration │ -│ describe Describe a configuration │ -│ list List configurations │ -│ login Add a configuration for ARLAS Cloud │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli confs [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ create Add a configuration │ +│ delete Delete a configuration │ +│ describe Describe a configuration │ +│ list List configurations │ +│ login Add a configuration for ARLAS Cloud │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` -## Create a configuration -The configuration file (default is `$HOME/.arlas/cli/configuration.yaml`) contains 3 sections: +## create + +`arlas_cli` is meant to communicate with a deployed ARLAS. This link is configured by creating a new configuration. + + +```shell +> arlas_cli confs create --help + + Usage: arlas_cli confs create [OPTIONS] NAME + + Add a configuration + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * name TEXT Name of the configuration [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ * --server TEXT ARLAS Server url │ +│ [default: None] │ +│ [required] │ +│ --headers TEXT header (name:value) │ +│ --persistence TEXT ARLAS Persistence │ +│ url │ +│ [default: None] │ +│ --persistence-head… TEXT header (name:value) │ +│ --elastic TEXT elasticsearch url │ +│ [default: None] │ +│ --elastic-login TEXT elasticsearch login │ +│ [default: None] │ +│ --elastic-password TEXT elasticsearch │ +│ password │ +│ [default: None] │ +│ --elastic-headers TEXT header (name:value) │ +│ --allow-delete --no-allow-delete Is delete command │ +│ allowed for this │ +│ configuration? │ +│ [default: │ +│ no-allow-delete] │ +│ --auth-token-url TEXT Token URL of the │ +│ authentication │ +│ service │ +│ [default: None] │ +│ --auth-headers TEXT header (name:value) │ +│ --auth-org TEXT ARLAS IAM │ +│ Organization │ +│ [default: None] │ +│ --auth-login TEXT login │ +│ [default: None] │ +│ --auth-password TEXT password │ +│ [default: None] │ +│ --auth-client-id TEXT Client ID │ +│ [default: None] │ +│ --auth-client-secr… TEXT Client secret │ +│ [default: None] │ +│ --auth-grant-type TEXT Grant type (e.g. │ +│ password) │ +│ [default: None] │ +│ --auth-arlas-iam --no-auth-arlas-iam Is it an ARLAS IAM │ +│ service? │ +│ [default: │ +│ auth-arlas-iam] │ +│ --help Show this message │ +│ and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + -- the list of ARLAS configurations -- the list of mappings -- the list of collection models +### Create a new configuration -It is possible, with the `arlas_cli` command line, to manage the ARLAS configurations, but not the two last ones. +The `arlas_cli confs create` command has to be filled with options specific to your ARLAS deployment. -An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and elasticsearch. It supports the addition of http headers. For ARLAS, keycloak and ARLAS IAM authentications are supported. Default is keycloak, use `--auth-arlas-iam` for ARLAS IAM. +Those options can be filled as described in the following sections. -!!! danger - By default, it is not possible to run the `delete` command on an elasticsearch with `arlas_cli`. This is to prevent accidental data loass. In order to allow delete on a configuration, use the `--allow-delete` option. +#### ARLAS Authentication + +For ARLAS, keycloak and ARLAS IAM authentications are supported. + +=== "ARLAS IAM" + + To create a configuration using ARLAS IAM, the following parameters have to be set with your values: + + - The IAM session url `IAM_URL` + - Your IAM user `ARLAS_USER` + - Your IAM password `ARLAS_PWD` + - Your ARLAS organization `ARLAS_ORGANIZATION` + + The following options are used by `confs create` sub-command to generate the conf: + ``` + --auth-arlas-iam + --auth-token-url {IAM_URL} + --auth-login {ARLAS_USER} + --auth-password {ARLAS_PWD} + --auth-headers "Content-Type:application/json;charset=utf-8" + --auth-org {ARLAS_ORGANIZATION} + ``` + +=== "Keycloak" + By default, the authentication is supposed to be with ARLAS IAM. For Keycloak, use `--no-auth-arlas-iam` + + Keycloak authentication details coming soon... + +#### ARLAS Server and Persistence + +The ARLAS server URL (`ARLAS_SERVER_URL`) and the ARLAS persistence server URL (`ARLAS_PERSISTENCE_URL`) have to be set in the configuration using the following options in the `confs create` sub-command: +``` +--server {ARLAS_SERVER_URL} +--headers "Content-Type:application/json" +--persistence {ARLAS_PERSISTENCE_URL} +--persistence-headers "Content-Type:application/json" +``` + + +#### Elasticsearch + +The used Elasticsearch instance and your credentials has to be set in the configuration: + +- `ELASTIC_ENDPOINT`: The used Elasticsearch endpoint (eg: http://localhost:9200) +- `ELASTIC_USER`: Your ES user name +- `ELASTIC_PWD`: Your ES user password + +The link to the ES instance is configured by using the following options in the `confs create` sub-command: +``` +--elastic {ELASTIC_ENDPOINT} +--elastic-headers "Content-Type:application/json" +--elastic-login {ELASTIC_USER} +--elastic-password {ELASTIC_PWD} +``` + +#### Allow data deletion + +By default, it is not possible to run the `indices delete` command on an elasticsearch with `arlas_cli`. +This is to prevent accidental data loss. + +!!! warning "--allow-delete" + In order to allow to delete on a configuration, use the `--allow-delete` option. + +## login + +### Create configuration for ARLAS Cloud + +The `confs longin` allows to create a configuration linked to an ARLAS Cloud account: ```shell -> arlas_cli confs create --help - - Usage: python -m arlas.cli.cli confs create [OPTIONS] NAME - - Add a configuration - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * name TEXT Name of the configuration [default: None] │ -│ [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ * --server TEXT ARLAS Server url │ -│ [default: None] │ -│ [required] │ -│ --headers TEXT header │ -│ (name:value) │ -│ --persistence TEXT ARLAS Persistence │ -│ url │ -│ [default: None] │ -│ --persistence-h… TEXT header │ -│ (name:value) │ -│ --elastic TEXT elasticsearch url │ -│ [default: None] │ -│ --elastic-login TEXT elasticsearch │ -│ login │ -│ [default: None] │ -│ --elastic-passw… TEXT elasticsearch │ -│ password │ -│ [default: None] │ -│ --elastic-heade… TEXT header │ -│ (name:value) │ -│ --allow-delete --no-allow-dele… Is delete command │ -│ allowed for this │ -│ configuration? │ -│ [default: │ -│ no-allow-delete] │ -│ --auth-token-url TEXT Token URL of the │ -│ authentication │ -│ service │ -│ [default: None] │ -│ --auth-headers TEXT header │ -│ (name:value) │ -│ --auth-org TEXT ARLAS IAM │ -│ Organization │ -│ [default: None] │ -│ --auth-login TEXT login │ -│ [default: None] │ -│ --auth-password TEXT password │ -│ [default: None] │ -│ --auth-client-id TEXT Client ID │ -│ [default: None] │ -│ --auth-client-s… TEXT Client secret │ -│ [default: None] │ -│ --auth-grant-ty… TEXT Grant type (e.g. │ -│ password) │ -│ [default: None] │ -│ --auth-arlas-iam --no-auth-arlas… Is it an ARLAS │ -│ IAM service? │ -│ [default: │ -│ auth-arlas-iam] │ -│ --help Show this message │ -│ and exit. │ -╰────────────────────────────────────────────────────────────────────╯ +> !!!execute arlas_cli confs login --help +``` + +Only your own ES and ARLAS credentials have to be set, the configuration is directly linked to the ARLAS Cloud instance. + +It creates a configuration based on your username (extracted from your ARLAS login) : `cloud.arlas.io.{USER_NAME}` + +!!! note + This created configuration is used as default. + + You no longer need to declare the `--config` in the arlas_cli commands + +See the [ARLAS Cloud configuration guide](configuration.md#arlas-cloud-configuration). +## delete + +### Delete an existing configuration + +An existing configuration can be deleted with the `confs delete` sub command: + + +```shell +> arlas_cli confs delete --help + + Usage: arlas_cli confs delete [OPTIONS] CONFIG + + Delete a configuration + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * config TEXT Name of the configuration [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +To remove an existing configuration from the default configuration file, simply run the following command: + +```shell +arlas_cli confs delete {conf_name} +``` + +The configuration will no longer appear in the configuration file. + +!!! warning + Once deleted, the configuration cannot be retrieved. + +## describe + +### Describe the content of a configuration + +The content of a configuration can be detailed with `confs describe` sub command: + + +```shell +> arlas_cli confs describe --help + + Usage: arlas_cli confs describe [OPTIONS] CONFIG + + Describe a configuration + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * config TEXT Name of the configuration [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +For example, the default local configuration looks like: + + +```shell +> arlas_cli confs describe local +allow_delete: true +authorization: null +elastic: + headers: + Content-Type: application/json + location: http://localhost:9200 + login: null + password: null +persistence: + headers: + Content-Type: application/json + location: http://localhost/persist + login: null + password: null +server: + headers: + Content-Type: application/json + location: http://localhost/arlas + login: null + password: null +``` + +We get the different elements of the configurations: + +- authorization: The authentication system configuration +- elastic: The link to Elasticsearch cluster +- persistence: The link to ARLAS persistence +- server: The link to ARLAS server + +See [more about the configuration](configuration.md). + +## list + +### List the available configurations + +The list of available configurations can be obtained with `confs list` sub command: + + +```shell +> arlas_cli confs list --help + + Usage: arlas_cli confs list [OPTIONS] + + List configurations + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +The `confs list` sub-command returns the list of available configuration names and their ARLAS server url. + +For example: + +```shell +> arlas_cli confs list ++-------+------------------------+ +| name | url | ++-------+------------------------+ +| local | http://localhost/arlas | ++-------+------------------------+ ``` diff --git a/docs/docs/confs.md.template b/docs/docs/confs.md.template new file mode 100644 index 0000000..1b0957f --- /dev/null +++ b/docs/docs/confs.md.template @@ -0,0 +1,218 @@ +# Configurations + + +## ARLAS configurations + +An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and Elasticsearch. + +See [more information about configuration](configuration.md). + +It is possible, with the `arlas_cli confs` command lines, to manage the ARLAS configurations. + +**List configurations (confs) management commands** + + +```shell +> !!!execute arlas_cli confs --help +``` + + +## create + +`arlas_cli` is meant to communicate with a deployed ARLAS. This link is configured by creating a new configuration. + + +```shell +> !!!execute arlas_cli confs create --help +``` + + +### Create a new configuration + +The `arlas_cli confs create` command has to be filled with options specific to your ARLAS deployment. + +Those options can be filled as described in the following sections. + +#### ARLAS Authentication + +For ARLAS, keycloak and ARLAS IAM authentications are supported. + +=== "ARLAS IAM" + + To create a configuration using ARLAS IAM, the following parameters have to be set with your values: + + - The IAM session url `IAM_URL` + - Your IAM user `ARLAS_USER` + - Your IAM password `ARLAS_PWD` + - Your ARLAS organization `ARLAS_ORGANIZATION` + + The following options are used by `confs create` sub-command to generate the conf: + ``` + --auth-arlas-iam + --auth-token-url {IAM_URL} + --auth-login {ARLAS_USER} + --auth-password {ARLAS_PWD} + --auth-headers "Content-Type:application/json;charset=utf-8" + --auth-org {ARLAS_ORGANIZATION} + ``` + +=== "Keycloak" + By default, the authentication is supposed to be with ARLAS IAM. For Keycloak, use `--no-auth-arlas-iam` + + Keycloak authentication details coming soon... + +#### ARLAS Server and Persistence + +The ARLAS server URL (`ARLAS_SERVER_URL`) and the ARLAS persistence server URL (`ARLAS_PERSISTENCE_URL`) have to be set in the configuration using the following options in the `confs create` sub-command: +``` +--server {ARLAS_SERVER_URL} +--headers "Content-Type:application/json" +--persistence {ARLAS_PERSISTENCE_URL} +--persistence-headers "Content-Type:application/json" +``` + + +#### Elasticsearch + +The used Elasticsearch instance and your credentials has to be set in the configuration: + +- `ELASTIC_ENDPOINT`: The used Elasticsearch endpoint (eg: http://localhost:9200) +- `ELASTIC_USER`: Your ES user name +- `ELASTIC_PWD`: Your ES user password + +The link to the ES instance is configured by using the following options in the `confs create` sub-command: +``` +--elastic {ELASTIC_ENDPOINT} +--elastic-headers "Content-Type:application/json" +--elastic-login {ELASTIC_USER} +--elastic-password {ELASTIC_PWD} +``` + +#### Allow data deletion + +By default, it is not possible to run the `indices delete` command on an elasticsearch with `arlas_cli`. +This is to prevent accidental data loss. + +!!! warning "--allow-delete" + In order to allow to delete on a configuration, use the `--allow-delete` option. + +## login + +### Create configuration for ARLAS Cloud + +The `confs longin` allows to create a configuration linked to an ARLAS Cloud account: + + +```shell +> !!!execute arlas_cli confs login --help +``` + +Only your own ES and ARLAS credentials have to be set, the configuration is directly linked to the ARLAS Cloud instance. + +It creates a configuration based on your username (extracted from your ARLAS login) : `cloud.arlas.io.{USER_NAME}` + +!!! note + This created configuration is used as default. + + You no longer need to declare the `--config` in the arlas_cli commands + +See the [ARLAS Cloud configuration guide](configuration.md#arlas-cloud-configuration). + +## delete + +### Delete an existing configuration + +An existing configuration can be deleted with the `confs delete` sub command: + + +```shell +> !!!execute arlas_cli confs delete --help +``` + +``` + +To remove an existing configuration from the default configuration file, simply run the following command: + +```shell +arlas_cli confs delete {conf_name} +``` + +The configuration will no longer appear in the configuration file. + +!!! warning + Once deleted, the configuration cannot be retrieved. + +## describe + +### Describe the content of a configuration + +The content of a configuration can be detailed with `confs describe` sub command: + + +```shell +> !!!execute arlas_cli confs describe --help +``` + +``` + +For example, the default local configuration looks like: + + +```shell +> arlas_cli confs describe local +allow_delete: true +authorization: null +elastic: + headers: + Content-Type: application/json + location: http://localhost:9200 + login: null + password: null +persistence: + headers: + Content-Type: application/json + location: http://localhost/persist + login: null + password: null +server: + headers: + Content-Type: application/json + location: http://localhost/arlas + login: null + password: null +``` + +We get the different elements of the configurations: + +- authorization: The authentication system configuration +- elastic: The link to Elasticsearch cluster +- persistence: The link to ARLAS persistence +- server: The link to ARLAS server + +See [more about the configuration](configuration.md). + +## list + +### List the available configurations + +The list of available configurations can be obtained with `confs list` sub command: + + +```shell +> !!!execute arlas_cli confs list --help +``` + +``` + +The `confs list` sub-command returns the list of available configuration names and their ARLAS server url. + +For example: + +```shell +> arlas_cli confs list ++-------+------------------------+ +| name | url | ++-------+------------------------+ +| local | http://localhost/arlas | ++-------+------------------------+ +``` diff --git a/docs/docs/confs_1.md b/docs/docs/confs_1.md deleted file mode 100644 index b8ffbef..0000000 --- a/docs/docs/confs_1.md +++ /dev/null @@ -1,11 +0,0 @@ - -!!! tip - At its first launch, `arlas_cli` will create a first configuration file for you (`$HOME/.arlas/cli/configuration.yaml`), with two ARLAS configurations, one pointing at a local deployment, one on ARLAS demo (without elasticsearch). - -!!! bug - If you used the ARLAS Exploration Stack, it is possible that you already have a directory named `$HOME/.arlas`. This directory has been created by docker as root. The owner of the directory must be changed to the local user (`sudo chown ${USER}: $HOME/.arlas`). - -## List configuration management commands - - -```shell diff --git a/docs/docs/confs_2.md b/docs/docs/confs_2.md deleted file mode 100644 index b5c2ba0..0000000 --- a/docs/docs/confs_2.md +++ /dev/null @@ -1,19 +0,0 @@ -``` - -## Create a configuration - -The configuration file (default is `$HOME/.arlas/cli/configuration.yaml`) contains 3 sections: - -- the list of ARLAS configurations -- the list of mappings -- the list of collection models - -It is possible, with the `arlas_cli` command line, to manage the ARLAS configurations, but not the two last ones. - -An ARLAS Configuration tells `arlas_cli` how to contact ARLAS Server and elasticsearch. It supports the addition of http headers. For ARLAS, keycloak and ARLAS IAM authentications are supported. Default is keycloak, use `--auth-arlas-iam` for ARLAS IAM. - -!!! danger - By default, it is not possible to run the `delete` command on an elasticsearch with `arlas_cli`. This is to prevent accidental data loass. In order to allow delete on a configuration, use the `--allow-delete` option. - - -```shell diff --git a/docs/docs/confs_3.md b/docs/docs/confs_3.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/confs_3.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/help.md b/docs/docs/help.md index 1354fc7..fc6fd27 100644 --- a/docs/docs/help.md +++ b/docs/docs/help.md @@ -6,45 +6,60 @@ Whether you are at the top level: ```shell > arlas_cli --help -Usage: arlas_cli [OPTIONS] COMMAND [ARGS]... + + Usage: arlas_cli [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --config-file TEXT Path to the configuration file │ +│ if you do not want to use the │ +│ default one: │ +│ .arlas/cli/configuration.yaml. │ +│ [default: None] │ +│ --print-curl --no-print-curl Print curl command │ +│ [default: no-print-curl] │ +│ --version Print command line version │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ collections │ +│ confs │ +│ iam │ +│ indices │ +│ persist │ +╰────────────────────────────────────────────────────────────────────────────╯ -Options: - --config-file TEXT Path to the configuration file if you do not - want to use the default one: - .arlas/cli/configuration.yaml. - --print-curl / --no-print-curl Print curl command [default: no-print-curl] - --version Print command line version - --help Show this message and exit. - -Commands: - collections - confs - iam - indices - persist ``` or within a sub command: ```shell -> arlas_cli indices --config demo mapping --help -Usage: arlas_cli indices mapping [OPTIONS] FILE - - Generate the mapping based on the data - -Arguments: - FILE Path to the file conaining the data. Format: NDJSON [required] +> arlas_cli indices --config local mapping --help + + Usage: arlas_cli indices mapping [OPTIONS] FILE + + Generate the mapping based on the data + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * file TEXT Path to the file containing the data. Format: NDJSON │ +│ [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --nb-lines INTEGER Number of line to consider for generating │ +│ the mapping. Avoid going over 10. │ +│ [default: 2] │ +│ --field-mapping TEXT Override the mapping with the provided │ +│ field path/type. Example: │ +│ fragment.location:geo_point. Important: │ +│ the full field path must be provided. │ +│ --no-fulltext TEXT List of keyword or text fields that should │ +│ not be in the fulltext search. Important: │ +│ the field name only must be provided. │ +│ --push-on TEXT Push the generated mapping for the │ +│ provided index name │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ -Options: - --nb-lines INTEGER Number of line to consider for generating the mapping. - Avoid going over 10. [default: 2] - --field-mapping TEXT Overide the mapping with the provided field path/type. - Example: fragment.location:geo_point. Important: the - full field path must be provided. - --no-fulltext TEXT List of keyword or text fields that should not be in - the fulltext search. Important: the field name only - must be provided. - --push-on TEXT Push the generated mapping for the provided index name - --help Show this message and exit. ``` diff --git a/docs/docs/help.md.template b/docs/docs/help.md.template new file mode 100644 index 0000000..462e6cf --- /dev/null +++ b/docs/docs/help.md.template @@ -0,0 +1,16 @@ +## Getting help + +At any time, if you need some help on how to use the command line or one of its sub command, simply add `--help`. + +Whether you are at the top level: + +```shell +> !!!execute arlas_cli --help +``` + +or within a sub command: + + +```shell +> !!!execute arlas_cli indices --config local mapping --help +``` diff --git a/docs/docs/iam.md b/docs/docs/iam.md index 9f97e9a..cd9176c 100644 --- a/docs/docs/iam.md +++ b/docs/docs/iam.md @@ -1,44 +1,40 @@ +# Identity & Access Management + ## Manage organisations, users, permissions and groups ```shell > arlas_cli iam --config local orgs --help - - Usage: python -m arlas.cli.cli iam orgs [OPTIONS] COMMAND [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ add Create organisation with the given │ -│ name │ -│ add_group Add a group to the organisation │ -│ add_permission Add a permission to the organisation │ -│ add_permission_to_group Add a permission to a group within │ -│ the organisation │ -│ add_user Add a user to the organisation, and │ -│ optionally within groups │ -│ add_user_to_group Add a user to a group within the │ -│ organisation │ -│ collections List the collections of the │ -│ organisation │ -│ delete Delete the organisation │ -│ delete_group Remove the group from the │ -│ organisation │ -│ delete_permission Remove the permission from the │ -│ organisation │ -│ delete_permission_from_group Remove a permission to a group │ -│ within the organisation │ -│ delete_user Remove the user from the │ -│ organisation │ -│ delete_user_from_group Remove a user from a group within │ -│ the organisation │ -│ groups List the groups of the organisation │ -│ list List organisations │ -│ permissions List the permissions of the │ -│ organisation │ -│ users List the users of the organisation │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli iam orgs [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ add Create organisation with the given name │ +│ add_group Add a group to the organisation │ +│ add_permission Add a permission to the organisation │ +│ add_permission_to_group Add a permission to a group within the │ +│ organisation │ +│ add_user Add a user to the organisation, and │ +│ optionally within groups │ +│ add_user_to_group Add a user to a group within the │ +│ organisation │ +│ collections List the collections of the organisation │ +│ delete Delete the organisation │ +│ delete_group Remove the group from the organisation │ +│ delete_permission Remove the permission from the organisation │ +│ delete_permission_from_group Remove a permission to a group within the │ +│ organisation │ +│ delete_user Remove the user from the organisation │ +│ delete_user_from_group Remove a user from a group within the │ +│ organisation │ +│ groups List the groups of the organisation │ +│ list List organisations │ +│ permissions List the permissions of the organisation │ +│ users List the users of the organisation │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` @@ -47,17 +43,17 @@ ```shell > arlas_cli iam --config local users --help - - Usage: python -m arlas.cli.cli iam users [OPTIONS] COMMAND [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ activate Activate user account │ -│ add Create user │ -│ deactivate Deactivate user account │ -│ delete Delete user │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli iam users [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ activate Activate user account │ +│ add Create user │ +│ deactivate Deactivate user account │ +│ delete Delete user │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` diff --git a/docs/docs/iam.md.template b/docs/docs/iam.md.template new file mode 100644 index 0000000..a9c1a05 --- /dev/null +++ b/docs/docs/iam.md.template @@ -0,0 +1,15 @@ +# Identity & Access Management + +## Manage organisations, users, permissions and groups + + +```shell +> !!!execute arlas_cli iam --config local orgs --help +``` + +## Create, delete, activate / deactivate users + + +```shell +> !!!execute arlas_cli iam --config local users --help +``` diff --git a/docs/docs/iam_1.md b/docs/docs/iam_1.md deleted file mode 100644 index ac35aa2..0000000 --- a/docs/docs/iam_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## Manage organisations, users, permissions and groups - - -```shell diff --git a/docs/docs/iam_2.md b/docs/docs/iam_2.md deleted file mode 100644 index 7283b57..0000000 --- a/docs/docs/iam_2.md +++ /dev/null @@ -1,6 +0,0 @@ -``` - -## Create, delete, activate / deactivate users - - -```shell diff --git a/docs/docs/iam_3.md b/docs/docs/iam_3.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/iam_3.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/images/builder_widgets.png b/docs/docs/images/builder_widgets.png new file mode 100644 index 0000000..ca203a9 Binary files /dev/null and b/docs/docs/images/builder_widgets.png differ diff --git a/docs/docs/images/dashboard.png b/docs/docs/images/dashboard.png new file mode 100644 index 0000000..1a5aa5d Binary files /dev/null and b/docs/docs/images/dashboard.png differ diff --git a/docs/docs/images/hub.png b/docs/docs/images/hub.png new file mode 100644 index 0000000..b66d7f8 Binary files /dev/null and b/docs/docs/images/hub.png differ diff --git a/docs/docs/images/iam.png b/docs/docs/images/iam.png new file mode 100644 index 0000000..2812ab8 Binary files /dev/null and b/docs/docs/images/iam.png differ diff --git a/docs/docs/index.md b/docs/docs/index.md index dcd184f..b7afe3f 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -1,43 +1,50 @@ # About arlas_cli -__ARLAS Exploration__ is an Open Source software for exploring and analysing Geo BigData. __ARLAS Command Line__ (`arlas_cli`) is for managing elasticsearch indices, ARLAS collections and elasticsearch/ARLAS endpoint configurations. +__ARLAS Exploration__ is an Open Source software for exploring and analysing Geo BigData. -`arlas_cli` is a python command line for: +__ARLAS Command Line__ (`arlas_cli`) is a tool to manage data and configurations in ARLAS. -- managing elasticsearch indices: - - generate an index mapping based on [NDJSON data](https://jsonlines.org/) - - create an index - - list indices - - describe an index - - delete an index -- managing [ARLAS collections](https://docs.arlas.io/arlas-api-collection/) - - create a collection - - list collections - - describe a collection - - delete a collection -- managing ARLAS persistence - - create a entry in a zone - - list entries within a zone - - describe an entry - - delete an entry - - get the groups accessing a zone -- managing ARLAS Identity and Access (ARLAS IAM) - - list organisations - - add an organisation - - within an oganisation: - - list visible collections - - list groups - - list users - - list permissions - - add/delete a permission - - add/delete a group - - add/delete a permission to/from a group - - add/delete a user - - add/delete a user to/from a group - - create a user - - activate a user - - deactivate a user -- managing configurations - - register an ARLAS/elasticsearch configuration, with headers and authentication parameters - - delete a configuration + +`arlas_cli` is a Python command line for: + +- Managing [Elasticsearch indices](concepts.md#es-index): + - Generate an index mapping based on [NDJSON data](https://jsonlines.org/) + - Create an index + - List indices + - Describe an index + - Clone an index + - Migrate an index + - Delete an index +- Managing [ARLAS collections](concepts.md#arlas-collection) + - Create a collection + - List collections + - Describe a collection + - Delete a collection +- Managing [arlas_cli configurations](concepts.md#configuration) + - Register an ARLAS/Elasticsearch configuration, with headers and authentication parameters + - Login to your ARLAS Cloud account + - List your configurations + - Delete a configuration +- Managing [ARLAS Dashboards](concepts.md#arlas-dashboards) persistence + - Create a dashboard from a configuration file + - List available dashboard + - Describe a dashboard + - Delete a dashboard + - Get the groups accessing dashboards +- Managing [ARLAS Identity and Access](concepts.md#arlas-iam) (ARLAS IAM) + - List organisations + - Add an organisation + - Within an organisation: + - List visible collections + - List groups + - List users + - List permissions + - Add/delete a permission + - Add/delete a group + - Add/delete a permission to/from a group + - Add/delete a user + - Add/delete a user to/from a group + - Create a user + - Activate a user + - Deactivate a user diff --git a/docs/docs/indices.md b/docs/docs/indices.md index e63fd0b..db3c8e1 100644 --- a/docs/docs/indices.md +++ b/docs/docs/indices.md @@ -1,121 +1,514 @@ -## List index management commands +# Indices + +## Elasticsearch index + +To be explored in ARLAS dashboards, the data has to be indexed in an [Elasticsearch](concepts.md#elasticsearch) (ES) [index](concepts.md#es-index). +An index contains the data and a [mapping](concepts.md#es-mapping) to describe how fields have to be interpreted (types). + +`arlas_cli` provide tools to infer mapping from data and manage the ES indices with the `indices` command. + +**List index management commands** ```shell > arlas_cli indices --help - - Usage: python -m arlas.cli.cli indices [OPTIONS] COMMAND [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --config TEXT Name of the ARLAS configuration to use from │ -│ your configuration file │ -│ (/Users/gaudan/.arlas/cli/configuration.yam… │ -│ [default: None] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ clone Clone an index and set its name │ -│ create Create an index │ -│ data Index data │ -│ delete Delete an index │ -│ describe Describe an index │ -│ list List indices │ -│ mapping Generate the mapping based on the data │ -│ migrate Migrate an index on another arlas configuration, and set │ -│ the target index name │ -│ sample Display a sample of an index │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli indices [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --config TEXT Name of the ARLAS configuration to use from your │ +│ configuration file │ +│ (/Users/gaudan/.arlas/cli/configuration.yaml). │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ clone Clone an index and set its name │ +│ create Create an index │ +│ data Index data │ +│ delete Delete an index │ +│ describe Describe an index │ +│ list List indices │ +│ mapping Generate the mapping based on the data │ +│ migrate Migrate an index on another arlas configuration, and set the │ +│ target index name │ +│ sample Display a sample of an index │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` -## Generate mapping from a data file - +## mapping -To generate a mapping, you need to provide a NDJSON file (New line delimiter JSON). The first N lines are used for infering the mapping. If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. Once you're happy with the mapping, you can directly push it on elasticsearch (`--push-on`). +`arlas_cli` provide tools to infer the ES mapping directly from a data file. ```shell > arlas_cli indices --config local mapping --help - - Usage: python -m arlas.cli.cli indices mapping [OPTIONS] FILE - - Generate the mapping based on the data - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * file TEXT Path to the file containing the data. Format: │ -│ NDJSON │ -│ [default: None] │ -│ [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --nb-lines INTEGER Number of line to consider for │ -│ generating the mapping. Avoid │ -│ going over 10. │ -│ [default: 2] │ -│ --field-mapping TEXT Override the mapping with the │ -│ provided field path/type. Example: │ -│ fragment.location:geo_point. │ -│ Important: the full field path │ -│ must be provided. │ -│ --no-fulltext TEXT List of keyword or text fields │ -│ that should not be in the fulltext │ -│ search. Important: the field name │ -│ only must be provided. │ -│ --push-on TEXT Push the generated mapping for the │ -│ provided index name │ -│ [default: None] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli indices mapping [OPTIONS] FILE + + Generate the mapping based on the data + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * file TEXT Path to the file containing the data. Format: NDJSON │ +│ [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --nb-lines INTEGER Number of line to consider for generating │ +│ the mapping. Avoid going over 10. │ +│ [default: 2] │ +│ --field-mapping TEXT Override the mapping with the provided │ +│ field path/type. Example: │ +│ fragment.location:geo_point. Important: │ +│ the full field path must be provided. │ +│ --no-fulltext TEXT List of keyword or text fields that should │ +│ not be in the fulltext search. Important: │ +│ the field name only must be provided. │ +│ --push-on TEXT Push the generated mapping for the │ +│ provided index name │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` -## Create an index with its mapping +### Data file + +To generate a mapping, you need to provide a NDJSON `file` (New line delimiter JSON). + +The values of the first lines of the files are used to infer the mapping for each field of the data. + +!!! note "--nb_lines" + The `indices mapping` function uses the first rows to infer mapping. + If a field is not present in the first rows, it will not appear in the mapping. + + Make sure to take enough rows to get all the fields with the option `--nb_lines` + +!!! tip + The data can be split in different NDJSON files in a folder: + ``` + part-00000-[...].json + part-00001-[...].json + ... + ``` + In practice, the `file` Argument can be filed with a **pattern** such as `path/to/data.json/part-0000*.json` to reference all the different files. + + +### Type identification + +The mapping associates to each field of the data a type (see [Elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + +A **geometry** is identified as such if + +- it is a geojson +- it is a WKT string +- the field name contains `geohash` +- it is a string containing two float separated by a comma + +A **date** is identified as such if + +- its name is one of `timestamp`, `date`, `start` or `end` and that it can be parsed as a date +- its name contains `timestamp`, `date`, `start` or `end` and its values are number within [631152000, 4102444800] or [631152000000, 4102444800000] (year 1990 to 2100) + +!!! note "--field-mapping" + If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. + + It has the structure **field_name:field_type** (see [Elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + + Examples: + + - `--field-mapping field_point:geo_point` + - `--field-mapping field_geometry:geo_shape` + - `--field-mapping field_short_text:keyword` + - `--field-mapping field_long_text:text` + - `--field-mapping field_float:double` + - `--field-mapping field_int:long` + + The date fields have a format that can be specified as **field_name:date-format** with all format accepted by [Elasticsearch date type](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) + + Examples: + + - `--field-mapping field_time_epoch_second:date-epoch_second` + - `--field-mapping field_time_epoch_millisecond:date-epoch_millis` + - `--field-mapping field_time_pattern:date-"yyyy-MM-dd HH:mm:ss"` + +By default, the keywords and text fields are searchable as fulltext to be accessible in the search bar. + +!!! note "--no-fulltext " + + If searching through a field value is not needed, it can be deactivated. + That would result in better performances for the fulltext search. + + Example: + + - `--no-fulltext field_keyword` + +### Created mapping + +By default, the `arlas_cli indices mapping` directly returns the mapping in the command line. + +Once you're happy with the mapping, you can either store it in a file or directly push it on elasticsearch. + +!!! tip "Store mapping in a file" + To store the created mapping in a `mapping.json` file, simply use `>` as the end of your command. + + Example: + + ```shell + > arlas_cli indices \ + --config {local} \ + mapping {path/to/data.json} \ + --field-mapping {timestamps.start:date-epoch_second} \ + --field-mapping {timestamps.end:date-epoch_second} \ + > {path/to/mapping.json} + ``` + +!!! note "--push-on" + + To push the inferred mapping directly in an Elasticsearch index, use the `--push-on` option with the target index name. + + Example: + + ```shell + > arlas_cli indices \ + --config {local} \ + mapping {path/to/data.json} \ + --push-on {index_name} + ``` + + The index is then created and the [index creation command](#create) can be skipped. + +## create + +Before putting the data in an elasticsearch index, the index has to be initialised with the [correct mapping](#mapping). + +The `indices create` sub-function create the index from a mapping json file. ```shell > arlas_cli indices --config local create --help - - Usage: python -m arlas.cli.cli indices create [OPTIONS] INDEX - - Create an index - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * index TEXT index's name [default: None] [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ * --mapping TEXT Name of the mapping within your │ -│ configuration, or URL or file path │ -│ [default: None] │ -│ [required] │ -│ --shards INTEGER Number of shards for the index │ -│ [default: 1] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli indices create [OPTIONS] INDEX + + Create an index + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * index TEXT index's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ * --mapping TEXT Name of the mapping within your │ +│ configuration, or URL or file path │ +│ [default: None] │ +│ [required] │ +│ --shards INTEGER Number of shards for the index [default: 1] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` -## Index data +### Create an ES index with its mapping -For indexing data, you'll need to provide one ore several NDJSON file(s). Indexing uses bulks for optimal performances. +The index name and the path to the mapping json file have to be used to create the ES index. + +The `indices create` sub-function create the index from a mapping json file. + +Example: + +```shell +> arlas_cli indices \ + --config local \ + create {index_name} \ + --mapping {path/to/mapping.json} +``` + +Once the index is created, Elasticsearch can index data to fill that index. + +## data + +To explore data in ARLAS, it has to be indexed in the created ES index. + +The `indices data` sub-function ingest the data in a given index. ```shell > arlas_cli indices --config local data --help - - Usage: python -m arlas.cli.cli indices data [OPTIONS] INDEX FILES... - - Index data - -╭─ Arguments ────────────────────────────────────────────────────────╮ -│ * index TEXT index's name [default: None] [required] │ -│ * files FILES... List of paths to the file(s) containing │ -│ the data. Format: NDJSON │ -│ [default: None] │ -│ [required] │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --bulk INTEGER Bulk size for indexing data [default: 5000] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli indices data [OPTIONS] INDEX FILES... + + Index data + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * index TEXT index's name [default: None] [required] │ +│ * files FILES... List of paths to the file(s) containing the │ +│ data. Format: NDJSON │ +│ [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --bulk INTEGER Bulk size for indexing data [default: 5000] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` +### Ingest data + +To index data, you'll need to provide one or several NDJSON (New line delimiter JSON) file(s). +Indexing uses bulks for optimal performances. + +Example: + +```shell +> arlas_cli indices \ + --config {local} \ + data {index_name} {path/to/data.json} +``` + +!!! warning + If the index already contains data, the data is added to the index. + + To reindex the same data, delete the index, and do not forget to recreate it with the correct mapping before ingesting the data. + +!!! note "--bulk" + Indexing uses bulks for optimal performances. + + The size of bulk can be changed with the `--bulk` option + +## list + +To list the available ES indices, simply use the `indices list` sub-function. No arguments are required. + + +```shell +> arlas_cli indices --config local list --help + + Usage: arlas_cli indices list [OPTIONS] + + List indices + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### List available ES indices + +It displays for each ES index its status, the number of elements it contains and the size of the index. + +Example: + +```shell +> arlas_cli indices --config {local} list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| index_name | open | 100 | 1mb | ++--------------+--------+-------+--------+ +``` + +## describe + +Once the index is created, the description of the fields it contains (corresponding to the mapping) can be displayed with the `indices describe` sub function: + + +```shell +> arlas_cli indices --config local data --help + + Usage: arlas_cli indices data [OPTIONS] INDEX FILES... + + Index data + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * index TEXT index's name [default: None] [required] │ +│ * files FILES... List of paths to the file(s) containing the │ +│ data. Format: NDJSON │ +│ [default: None] │ +│ [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --bulk INTEGER Bulk size for indexing data [default: 5000] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Describe the index mapping + +For a given index, the description of its fields and their type can be displayed. + +For example: + +```shell +> arlas_cli indices --config {local} describe {index_name} ++------------------+-----------+ +| field name | type | ++------------------+-----------+ +| field_keyword | keyword | +| field_point | geo_point | +| field_long | long | +| field_shape | geo_shape | +| field_double | double | +| field_date | date | +| field_text | text | +| field_object | object | +| field_boolean | boolean | ++------------------+-----------+ +``` + +## sample + +The first rows of the data contained in an index can be displayed with the `indices sample` sub function. + + +```shell +> arlas_cli indices --config local delete --help + + Usage: arlas_cli indices delete [OPTIONS] INDEX + + Delete an index + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * index TEXT index's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Visualize few rows of your dataset + +For a given index `index_name`, the first rows of data can be displayed as a valid json dictionary. + +!!! note "--size" + The number of rows to display (default 100) can be changed + + Example: + + ```shell + > arlas_cli indices --config {local} sample {index_name} --size {10} + ``` + +By default, the json representation of the data is pretty printed (clear indentation and one line per field) + +!!! note "--no-pretty" + The pretty printing can be deactivated and data is displayed in a compact way + + Example: + + ```shell + > arlas_cli indices --config {local} sample {index_name} --no-pretty + ``` + +## clone + +### Duplicate an index with a new index name + +An ES index can be cloned on the same ES deployment with the `indices clone` sub-command: + + +```shell +> arlas_cli indices --config local clone --help + + Usage: arlas_cli indices clone [OPTIONS] SOURCE TARGET + + Clone an index and set its name + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * source TEXT Source index name [default: None] [required] │ +│ * target TEXT Target cloned index name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +Both indices co-exist with exactly the same mapping and data content. + +## migrate + +### Copy an index in another arlas configuration + +An index can be copied from an ES instance to another. + +!!! note + The two instances have to be accessible by `arlas_cli` with two configurations (see [Configuration guide](configuration.md)). + +The target configuration and the name of the new created index are given to the `indices migrate` sub-command. + + +```shell +> arlas_cli indices --config local clone --help + + Usage: arlas_cli indices clone [OPTIONS] SOURCE TARGET + + Clone an index and set its name + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * source TEXT Source index name [default: None] [required] │ +│ * target TEXT Target cloned index name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +Both indices co-exist with exactly the same mapping and data content. + +## delete + +The ES index can be deleted with `indices delete` sub command to free space on the ES cluster. + + +```shell +> arlas_cli indices --config local delete --help + + Usage: arlas_cli indices delete [OPTIONS] INDEX + + Delete an index + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * index TEXT index's name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Delete an ES index + +To delete an ES index `index_name` on `local` configuration, run the following command: + + +```shell +> arlas_cli indices --config {local} delete {index_name} +``` +!!! warning + A deleted index cannot be restored. + +!!! Note Delete is not always allowed + By default, it is not allowed to delete an index for a given configuration. + + To allow deleting, [edit the configuration file](configuration.md) and set `allow_delete` to `True`. + +!!! tip "Good practice: Set admin confs" + For a given ARLAS deployment, it is advised to set two configurations, with only the admin one that can delete an index. + + For example: + ``` + +----------------------+------------------------------------------+ + | name | url | + +----------------------+------------------------------------------+ + | cloud.arlas.io | https://cloud.arlas.io/arlas/server | + | cloud.arlas.io-admin | https://cloud.arlas.io/arlas/server | + | local | http://localhost/arlas | + +----------------------+------------------------------------------+ + ``` + + Here the configuration `--config cloud.arlas.io-admin` has to be used to delete any index. + \ No newline at end of file diff --git a/docs/docs/indices.md.template b/docs/docs/indices.md.template new file mode 100644 index 0000000..0ddbf95 --- /dev/null +++ b/docs/docs/indices.md.template @@ -0,0 +1,354 @@ +# Indices + +## Elasticsearch index + +To be explored in ARLAS dashboards, the data has to be indexed in an [Elasticsearch](concepts.md#elasticsearch) (ES) [index](concepts.md#es-index). +An index contains the data and a [mapping](concepts.md#es-mapping) to describe how fields have to be interpreted (types). + +`arlas_cli` provide tools to infer mapping from data and manage the ES indices with the `indices` command. + +**List index management commands** + + +```shell +> !!!execute arlas_cli indices --help +``` + +## mapping + +`arlas_cli` provide tools to infer the ES mapping directly from a data file. + + +```shell +> !!!execute arlas_cli indices --config local mapping --help +``` + +### Data file + +To generate a mapping, you need to provide a NDJSON `file` (New line delimiter JSON). + +The values of the first lines of the files are used to infer the mapping for each field of the data. + +!!! note "--nb_lines" + The `indices mapping` function uses the first rows to infer mapping. + If a field is not present in the first rows, it will not appear in the mapping. + + Make sure to take enough rows to get all the fields with the option `--nb_lines` + +!!! tip + The data can be split in different NDJSON files in a folder: + ``` + part-00000-[...].json + part-00001-[...].json + ... + ``` + In practice, the `file` Argument can be filed with a **pattern** such as `path/to/data.json/part-0000*.json` to reference all the different files. + + +### Type identification + +The mapping associates to each field of the data a type (see [Elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + +A **geometry** is identified as such if + +- it is a geojson +- it is a WKT string +- the field name contains `geohash` +- it is a string containing two float separated by a comma + +A **date** is identified as such if + +- its name is one of `timestamp`, `date`, `start` or `end` and that it can be parsed as a date +- its name contains `timestamp`, `date`, `start` or `end` and its values are number within [631152000, 4102444800] or [631152000000, 4102444800000] (year 1990 to 2100) + +!!! note "--field-mapping" + If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. + + It has the structure **field_name:field_type** (see [Elasticsearch type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html)) + + Examples: + + - `--field-mapping field_point:geo_point` + - `--field-mapping field_geometry:geo_shape` + - `--field-mapping field_short_text:keyword` + - `--field-mapping field_long_text:text` + - `--field-mapping field_float:double` + - `--field-mapping field_int:long` + + The date fields have a format that can be specified as **field_name:date-format** with all format accepted by [Elasticsearch date type](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) + + Examples: + + - `--field-mapping field_time_epoch_second:date-epoch_second` + - `--field-mapping field_time_epoch_millisecond:date-epoch_millis` + - `--field-mapping field_time_pattern:date-"yyyy-MM-dd HH:mm:ss"` + +By default, the keywords and text fields are searchable as fulltext to be accessible in the search bar. + +!!! note "--no-fulltext " + + If searching through a field value is not needed, it can be deactivated. + That would result in better performances for the fulltext search. + + Example: + + - `--no-fulltext field_keyword` + +### Created mapping + +By default, the `arlas_cli indices mapping` directly returns the mapping in the command line. + +Once you're happy with the mapping, you can either store it in a file or directly push it on elasticsearch. + +!!! tip "Store mapping in a file" + To store the created mapping in a `mapping.json` file, simply use `>` as the end of your command. + + Example: + + ```shell + > arlas_cli indices \ + --config {local} \ + mapping {path/to/data.json} \ + --field-mapping {timestamps.start:date-epoch_second} \ + --field-mapping {timestamps.end:date-epoch_second} \ + > {path/to/mapping.json} + ``` + +!!! note "--push-on" + + To push the inferred mapping directly in an Elasticsearch index, use the `--push-on` option with the target index name. + + Example: + + ```shell + > arlas_cli indices \ + --config {local} \ + mapping {path/to/data.json} \ + --push-on {index_name} + ``` + + The index is then created and the [index creation command](#create) can be skipped. + +## create + +Before putting the data in an elasticsearch index, the index has to be initialised with the [correct mapping](#mapping). + +The `indices create` sub-function create the index from a mapping json file. + + +```shell +> !!!execute arlas_cli indices --config local create --help +``` + +### Create an ES index with its mapping + +The index name and the path to the mapping json file have to be used to create the ES index. + +The `indices create` sub-function create the index from a mapping json file. + +Example: + +```shell +> arlas_cli indices \ + --config local \ + create {index_name} \ + --mapping {path/to/mapping.json} +``` + +Once the index is created, Elasticsearch can index data to fill that index. + +## data + +To explore data in ARLAS, it has to be indexed in the created ES index. + +The `indices data` sub-function ingest the data in a given index. + + +```shell +> !!!execute arlas_cli indices --config local data --help +``` +### Ingest data + +To index data, you'll need to provide one or several NDJSON (New line delimiter JSON) file(s). +Indexing uses bulks for optimal performances. + +Example: + +```shell +> arlas_cli indices \ + --config {local} \ + data {index_name} {path/to/data.json} +``` + +!!! warning + If the index already contains data, the data is added to the index. + + To reindex the same data, delete the index, and do not forget to recreate it with the correct mapping before ingesting the data. + +!!! note "--bulk" + Indexing uses bulks for optimal performances. + + The size of bulk can be changed with the `--bulk` option + +## list + +To list the available ES indices, simply use the `indices list` sub-function. No arguments are required. + + +```shell +> !!!execute arlas_cli indices --config local list --help +``` + +### List available ES indices + +It displays for each ES index its status, the number of elements it contains and the size of the index. + +Example: + +```shell +> arlas_cli indices --config {local} list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| index_name | open | 100 | 1mb | ++--------------+--------+-------+--------+ +``` + +## describe + +Once the index is created, the description of the fields it contains (corresponding to the mapping) can be displayed with the `indices describe` sub function: + + +```shell +> !!!execute arlas_cli indices --config local data --help +``` + +### Describe the index mapping + +For a given index, the description of its fields and their type can be displayed. + +For example: + +```shell +> arlas_cli indices --config {local} describe {index_name} ++------------------+-----------+ +| field name | type | ++------------------+-----------+ +| field_keyword | keyword | +| field_point | geo_point | +| field_long | long | +| field_shape | geo_shape | +| field_double | double | +| field_date | date | +| field_text | text | +| field_object | object | +| field_boolean | boolean | ++------------------+-----------+ +``` + +## sample + +The first rows of the data contained in an index can be displayed with the `indices sample` sub function. + + +```shell +> !!!execute arlas_cli indices --config local delete --help +``` + +### Visualize few rows of your dataset + +For a given index `index_name`, the first rows of data can be displayed as a valid json dictionary. + +!!! note "--size" + The number of rows to display (default 100) can be changed + + Example: + + ```shell + > arlas_cli indices --config {local} sample {index_name} --size {10} + ``` + +By default, the json representation of the data is pretty printed (clear indentation and one line per field) + +!!! note "--no-pretty" + The pretty printing can be deactivated and data is displayed in a compact way + + Example: + + ```shell + > arlas_cli indices --config {local} sample {index_name} --no-pretty + ``` + +## clone + +### Duplicate an index with a new index name + +An ES index can be cloned on the same ES deployment with the `indices clone` sub-command: + + +```shell +> !!!execute arlas_cli indices --config local clone --help +``` + +Both indices co-exist with exactly the same mapping and data content. + +## migrate + +### Copy an index in another arlas configuration + +An index can be copied from an ES instance to another. + +!!! note + The two instances have to be accessible by `arlas_cli` with two configurations (see [Configuration guide](configuration.md)). + +The target configuration and the name of the new created index are given to the `indices migrate` sub-command. + + +```shell +> !!!execute arlas_cli indices --config local clone --help +``` + +Both indices co-exist with exactly the same mapping and data content. + +## delete + +The ES index can be deleted with `indices delete` sub command to free space on the ES cluster. + + +```shell +> !!!execute arlas_cli indices --config local delete --help +``` + +### Delete an ES index + +To delete an ES index `index_name` on `local` configuration, run the following command: + + +```shell +> arlas_cli indices --config {local} delete {index_name} +``` +!!! warning + A deleted index cannot be restored. + +!!! Note Delete is not always allowed + By default, it is not allowed to delete an index for a given configuration. + + To allow deleting, [edit the configuration file](configuration.md) and set `allow_delete` to `True`. + +!!! tip "Good practice: Set admin confs" + For a given ARLAS deployment, it is advised to set two configurations, with only the admin one that can delete an index. + + For example: + ``` + +----------------------+------------------------------------------+ + | name | url | + +----------------------+------------------------------------------+ + | cloud.arlas.io | https://cloud.arlas.io/arlas/server | + | cloud.arlas.io-admin | https://cloud.arlas.io/arlas/server | + | local | http://localhost/arlas | + +----------------------+------------------------------------------+ + ``` + + Here the configuration `--config cloud.arlas.io-admin` has to be used to delete any index. + \ No newline at end of file diff --git a/docs/docs/indices_1.md b/docs/docs/indices_1.md deleted file mode 100644 index 7ee130e..0000000 --- a/docs/docs/indices_1.md +++ /dev/null @@ -1,4 +0,0 @@ -## List index management commands - - -```shell diff --git a/docs/docs/indices_2.md b/docs/docs/indices_2.md deleted file mode 100644 index f7c32bc..0000000 --- a/docs/docs/indices_2.md +++ /dev/null @@ -1,9 +0,0 @@ -``` - -## Generate mapping from a data file - - -To generate a mapping, you need to provide a NDJSON file (New line delimiter JSON). The first N lines are used for infering the mapping. If the mapping is wrong, you can overwrite the typing with the `--field-mapping` option. Once you're happy with the mapping, you can directly push it on elasticsearch (`--push-on`). - - -```shell diff --git a/docs/docs/indices_3.md b/docs/docs/indices_3.md deleted file mode 100644 index 9c8b195..0000000 --- a/docs/docs/indices_3.md +++ /dev/null @@ -1,6 +0,0 @@ -``` - -## Create an index with its mapping - - -```shell diff --git a/docs/docs/indices_4.md b/docs/docs/indices_4.md deleted file mode 100644 index 59a9079..0000000 --- a/docs/docs/indices_4.md +++ /dev/null @@ -1,8 +0,0 @@ -``` - -## Index data - -For indexing data, you'll need to provide one ore several NDJSON file(s). Indexing uses bulks for optimal performances. - - -```shell diff --git a/docs/docs/indices_5.md b/docs/docs/indices_5.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/indices_5.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/install.md b/docs/docs/install.md index 8c1065d..23703e2 100644 --- a/docs/docs/install.md +++ b/docs/docs/install.md @@ -1,24 +1,87 @@ +# Installation -## Prerequisite +## Prerequisites - python 3.10 -- pip - -If you manage your own ARLAS stack, you will also need: -- [ARLAS](https://github.com/gisaia/ARLAS-Exploration-stack) +- pip ## Install +### Install arlas_cli +Simply run in a terminal: + ```shell > pip install arlas_cli ``` -In a new terminal, you should be able to run it: +!!! warning "Microsoft Windows" + For Windows users, the installation has to be made in a python virtual environment, follow the [venv guide](#python-virtual-environment) before installing. + +!!! success + + In a new Terminal/Powershell, you should be able to run it: + + + ```shell + > arlas_cli --version + X.Y.Z + ``` + + The version of `arlas_cli` is `X.Y.Z` where `X` is ARLAS Stack main version and `Y.Z` is the increment of arlas_cli version. + +### Python Virtual environment + +Particularly on Windows, it is advised to create a python virtual environment with [venv](https://docs.python.org/3.10/library/venv.html#creating-virtual-environments) before installing `arlas_cli` + +- Create the environment: +```shell +python -m venv env_arlas_cli +``` + +- Activate the environment: + +=== "Windows" + with cmd: + ```cmd + .\env_arlas_cli\Scripts\Activate.bat + ``` + with PowerShell: + ```PowerShell + .\env_arlas_cli\Scripts\Activate.ps1 + ``` + + !!! tip + On Microsoft Windows, it may be required to enable the Activate.ps1 script by setting the execution policy for the user. + + You can do this by issuing the following PowerShell command: + ```PowerShell + Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser + ``` + See [venv documentation](https://docs.python.org/3.10/library/venv.html#creating-virtual-environments) for more information. + +=== "Linux/Mac" + ```bash + source env_arlas_cli/bin/activate (Can be added to .bashrc or .zshrc) + ``` + +# ARLAS + +`arlas_cli` is meant to interact with a deployed ARLAS stack. + +### ARLAS Cloud + +If you want to connect arlas_cli to an existing ARLAS Cloud account, simply follow the [configuration guide](configuration.md#arlas-cloud-configuration). + +### Run ARLAS Locally + +To run the simplest ARLAS stack and Elasticsearch on the local machine, clone the [ARLAS Stack Exploration](https://github.com/gisaia/ARLAS-Exploration-stack) project and run the stack: ```shell -> arlas_cli --version -X.X.X +> git clone https://github.com/gisaia/ARLAS-Exploration-stack.git +> cd ARLAS-Exploration-stack +> ./start.sh ``` +More details about deployment can be found on [ARLAS Stack Exploration project](https://github.com/gisaia/ARLAS-Exploration-stack). diff --git a/docs/docs/persist.md b/docs/docs/persist.md index 6ab8596..a606e12 100644 --- a/docs/docs/persist.md +++ b/docs/docs/persist.md @@ -1,27 +1,271 @@ -## Manage persistence +## ARLAS Persistence -ARLAS Persistence allows you to place files within zones. A zone is visible by groups. It is also possible to set the access: writers, readers and whether it is public or not. +ARLAS Persistence allows you to place files within **zones**. + +A **zone** is a group of files of the same nature (configurations, dashboard previews, bookmarks, ...) + +A **zone** is visible by **groups**. + +It is also possible to set who can access the file. + +The main persistence zone is the `config.json` which contains the dashboard configurations. + +An entry is an element stored in the persistence. ```shell > arlas_cli persist --help - - Usage: python -m arlas.cli.cli persist [OPTIONS] COMMAND [ARGS]... - -╭─ Options ──────────────────────────────────────────────────────────╮ -│ --config TEXT Name of the ARLAS configuration to use from │ -│ your configuration file │ -│ (/Users/gaudan/.arlas/cli/configuration.yam… │ -│ [default: None] │ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────╯ -╭─ Commands ─────────────────────────────────────────────────────────╮ -│ add Add an entry, returns its ID │ -│ delete Delete an entry │ -│ describe Describe an entry │ -│ get Retrieve an entry │ -│ groups List groups allowed to access a zone │ -│ zone List entries within a zone │ -╰────────────────────────────────────────────────────────────────────╯ + + Usage: arlas_cli persist [OPTIONS] COMMAND [ARGS]... + +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --config TEXT Name of the ARLAS configuration to use from your │ +│ configuration file │ +│ (/Users/gaudan/.arlas/cli/configuration.yaml). │ +│ [default: None] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Commands ─────────────────────────────────────────────────────────────────╮ +│ add Add an entry, returns its ID │ +│ delete Delete an entry │ +│ describe Describe an entry │ +│ get Retrieve an entry │ +│ groups List groups allowed to access a zone │ +│ zone List entries within a zone │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +## add + +The `persist add` sub-command allows to create an entry from a file. + + +```shell +> arlas_cli persist --config local add --help + + Usage: arlas_cli persist add [OPTIONS] FILE ZONE + + Add an entry, returns its ID + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * file TEXT File path [default: None] [required] │ +│ * zone TEXT zone [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --name TEXT name [default: none] │ +│ --reader TEXT Readers │ +│ --writer TEXT writers │ +│ --encode --no-encode Encode in BASE64 [default: no-encode] │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Add an entry from a file + +For example to create an ARLAS dashboard directly from a json file, run the following command to add the entry to the `config.json` zone: + + +```shell +> arlas_cli persist \ + --config {local} \ + add {path/to/config.dashboard.json} config.json \ + --name {dashboard_name} +32d2624b-d7cd-11ee-9a91-0242ac130004 +``` + +The returned identifier is the entry unique identifier. + +!!! Note + The `config.dashboard.json` file has to contain all the dashboard configurations merged (`config.json` and `config.map.json`). + +## zone + +The available entries in a zone can be listed with the `persist zone` sub-command: + + +```shell +> arlas_cli persist --config local zone --help + + Usage: arlas_cli persist zone [OPTIONS] ZONE + + List entries within a zone + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * zone TEXT Zone name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### List entries within a zone + +For example, to list all the available ARLAS dashboards (`config.json` zone), run the following command: + + +```shell +> arlas_cli persist --config {local} zone config.json ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| id | name | zone | last_update_date | owner | ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| 66984014-d0a1-11ee-b41c-0242ac190004 | Courses | config.json | 1708510231303 | anonymous | +... ++--------------------------------------+-------------------+-------------+------------------+-----------+ +``` + +The `id` is the unique identifier of the entry. + +## describe + +An entry (defined by its unique identifier) can be described with the `persist describe` sub-command: + + +```shell +> arlas_cli persist --config local describe --help + + Usage: arlas_cli persist describe [OPTIONS] ID + + Describe an entry + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * id TEXT entry identifier [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Describe an entry + +For example, a dashboard configuration can be described by its metadata with the following command: + + +```shell +> arlas_cli persist --config {local} describe {99acb6b7-3cfd-11ef-aee2-2e6497b109c4} ++------------------+--------------------------------------------+ +| metadata | value | ++------------------+--------------------------------------------+ +| ID | 99acb6b7-3cfd-11ef-aee2-2e6497b109c4 | +| name | AIS Course | +| zone | config.json | +| last_update_date | 2024-10-21T18:02:59.456000 | +| owner | 6ce7f1f2-2ee3-4532-987d-26079c1b3043 | +| organization | gisaia.com | +| ispublic | False | +| updatable | True | +| readers | group/config.json/gisaia.com, group/public | +| writers | group/config.json/gisaia.com | ++------------------+--------------------------------------------+ +``` + +In the metadata, we can find which organisation own the dashboard and which groups can access the dashboard as readers and writers. + +## get + +The content of an entry can be accessed with the `persist get` sub-command: + + +```shell +> arlas_cli persist --config local get --help + + Usage: arlas_cli persist get [OPTIONS] ID + + Retrieve an entry + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * id TEXT entry identifier [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### Get an entry value + +The content of the configuration file of an ARLAS dashboard can be read directly from its entry unique identifier: + + +```shell +> arlas_cli persist --config {local} get {32d2624b-d7cd-11ee-9a91-0242ac130004} +{ + "arlas": { + "web": { + "contributors": [ +... +} +``` + +## groups + +The groups accessing a zone can be listed with the `persist groups` sub-command: + + +```shell +> arlas_cli persist --config local groups --help + + Usage: arlas_cli persist groups [OPTIONS] ZONE + + List groups allowed to access a zone + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * zone TEXT Zone name [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ + +``` + +### List groups accessing a zone + +For example, the groups accessing the dashboards `config.json` zone can be listed with the following command: + + +```shell +> arlas_cli persist --config {local} groups config.json ++-------------------------------+ +| group | ++-------------------------------+ +| group/config.json/gisaia.com | +| group/public | ++-------------------------------+ +``` + +## delete + +An entry defined by its unique identifier can be deleted with the `persist delete` sub-command: + + +```shell +> arlas_cli persist --config local delete --help + + Usage: arlas_cli persist delete [OPTIONS] ID + + Delete an entry + +╭─ Arguments ────────────────────────────────────────────────────────────────╮ +│ * id TEXT entry identifier [default: None] [required] │ +╰────────────────────────────────────────────────────────────────────────────╯ +╭─ Options ──────────────────────────────────────────────────────────────────╮ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────╯ ``` + +### Delete an entry + +For example, to delete a dashboard from its identifier, run the following command: + + +```shell +> arlas_cli persist --config {local} delete {32d2624b-d7cd-11ee-9a91-0242ac130004} +Resource 32d2624b-d7cd-11ee-9a91-0242ac130004 deleted. +``` + +!!! warning + A deleted entry cannot be restored. diff --git a/docs/docs/persist.md.template b/docs/docs/persist.md.template new file mode 100644 index 0000000..f588b25 --- /dev/null +++ b/docs/docs/persist.md.template @@ -0,0 +1,175 @@ +## ARLAS Persistence + +ARLAS Persistence allows you to place files within **zones**. + +A **zone** is a group of files of the same nature (configurations, dashboard previews, bookmarks, ...) + +A **zone** is visible by **groups**. + +It is also possible to set who can access the file. + +The main persistence zone is the `config.json` which contains the dashboard configurations. + +An entry is an element stored in the persistence. + + +```shell +> !!!execute arlas_cli persist --help +``` + +## add + +The `persist add` sub-command allows to create an entry from a file. + + +```shell +> !!!execute arlas_cli persist --config local add --help +``` + +### Add an entry from a file + +For example to create an ARLAS dashboard directly from a json file, run the following command to add the entry to the `config.json` zone: + + +```shell +> arlas_cli persist \ + --config {local} \ + add {path/to/config.dashboard.json} config.json \ + --name {dashboard_name} +32d2624b-d7cd-11ee-9a91-0242ac130004 +``` + +The returned identifier is the entry unique identifier. + +!!! Note + The `config.dashboard.json` file has to contain all the dashboard configurations merged (`config.json` and `config.map.json`). + +## zone + +The available entries in a zone can be listed with the `persist zone` sub-command: + + +```shell +> !!!execute arlas_cli persist --config local zone --help +``` + +### List entries within a zone + +For example, to list all the available ARLAS dashboards (`config.json` zone), run the following command: + + +```shell +> arlas_cli persist --config {local} zone config.json ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| id | name | zone | last_update_date | owner | ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| 66984014-d0a1-11ee-b41c-0242ac190004 | Courses | config.json | 1708510231303 | anonymous | +... ++--------------------------------------+-------------------+-------------+------------------+-----------+ +``` + +The `id` is the unique identifier of the entry. + +## describe + +An entry (defined by its unique identifier) can be described with the `persist describe` sub-command: + + +```shell +> !!!execute arlas_cli persist --config local describe --help +``` + +### Describe an entry + +For example, a dashboard configuration can be described by its metadata with the following command: + + +```shell +> arlas_cli persist --config {local} describe {99acb6b7-3cfd-11ef-aee2-2e6497b109c4} ++------------------+--------------------------------------------+ +| metadata | value | ++------------------+--------------------------------------------+ +| ID | 99acb6b7-3cfd-11ef-aee2-2e6497b109c4 | +| name | AIS Course | +| zone | config.json | +| last_update_date | 2024-10-21T18:02:59.456000 | +| owner | 6ce7f1f2-2ee3-4532-987d-26079c1b3043 | +| organization | gisaia.com | +| ispublic | False | +| updatable | True | +| readers | group/config.json/gisaia.com, group/public | +| writers | group/config.json/gisaia.com | ++------------------+--------------------------------------------+ +``` + +In the metadata, we can find which organisation own the dashboard and which groups can access the dashboard as readers and writers. + +## get + +The content of an entry can be accessed with the `persist get` sub-command: + + +```shell +> !!!execute arlas_cli persist --config local get --help +``` + +### Get an entry value + +The content of the configuration file of an ARLAS dashboard can be read directly from its entry unique identifier: + + +```shell +> arlas_cli persist --config {local} get {32d2624b-d7cd-11ee-9a91-0242ac130004} +{ + "arlas": { + "web": { + "contributors": [ +... +} +``` + +## groups + +The groups accessing a zone can be listed with the `persist groups` sub-command: + + +```shell +> !!!execute arlas_cli persist --config local groups --help +``` + +### List groups accessing a zone + +For example, the groups accessing the dashboards `config.json` zone can be listed with the following command: + + +```shell +> arlas_cli persist --config {local} groups config.json ++-------------------------------+ +| group | ++-------------------------------+ +| group/config.json/gisaia.com | +| group/public | ++-------------------------------+ +``` + +## delete + +An entry defined by its unique identifier can be deleted with the `persist delete` sub-command: + + +```shell +> !!!execute arlas_cli persist --config local delete --help +``` + +### Delete an entry + +For example, to delete a dashboard from its identifier, run the following command: + + +```shell +> arlas_cli persist --config {local} delete {32d2624b-d7cd-11ee-9a91-0242ac130004} +Resource 32d2624b-d7cd-11ee-9a91-0242ac130004 deleted. +``` + +!!! warning + A deleted entry cannot be restored. diff --git a/docs/docs/persist_1.md b/docs/docs/persist_1.md deleted file mode 100644 index 728cd7c..0000000 --- a/docs/docs/persist_1.md +++ /dev/null @@ -1,6 +0,0 @@ -## Manage persistence - -ARLAS Persistence allows you to place files within zones. A zone is visible by groups. It is also possible to set the access: writers, readers and whether it is public or not. - - -```shell diff --git a/docs/docs/persist_2.md b/docs/docs/persist_2.md deleted file mode 100644 index 8c5b07d..0000000 --- a/docs/docs/persist_2.md +++ /dev/null @@ -1 +0,0 @@ -``` diff --git a/docs/docs/started.md b/docs/docs/started.md index b29bd21..d34c1e6 100644 --- a/docs/docs/started.md +++ b/docs/docs/started.md @@ -1,62 +1,94 @@ -Install `arlas_cli` ([prerequisite](install.md#Prerequisite)) - +# Getting started with ARLAS CLI + +## Install arlas_cli + +To install `arlas_cli`, run the following command from the command line: + ```shell -> pip install arlas_cli +pip install arlas_cli ``` -The version of `arlas_cli` is `xxx.yyy` where `xxx` is ARLAS Stack main version and `yyy` is the increment of arlas_cli version. +For more details, in particular for installation on Microsoft Windows, see the [Installation Guide](install.md#installation). + + +## Configuration !!! warning "Prerequisite" - For running the various examples bellow, ARLAS and elasticsearch must be running on the local machine: clone the [ARLAS Stack Exploration](https://github.com/gisaia/ARLAS-Exploration-stack) project and run `./start.sh` . + For running the various examples below, ARLAS and Elasticsearch must be running on the local machine: + + Clone the [ARLAS Stack Exploration](https://github.com/gisaia/ARLAS-Exploration-stack) project and run `./start.sh` at project root. -## Initial configuration -`arlas_cli` uses a yaml file for storing various ARLAS and elasticsearch configurations. By default, the file is located in `~/.arlas/cli/configuration.yaml`. [One is automatically created for your convenience at the first launch](https://raw.githubusercontent.com/gisaia/arlas-cli/master/configuration.yaml). It contains the ARLAS demo endpoint and the local ARLAS and elasticsearch endpoints. +!!! tip + If you want to connect `arlas_cli` to an existing ARLAS Cloud account, simply follow the [configuration guide](configuration.md#arlas-cloud-configuration). -The configuration can also contain references to collection models for creating collections. A default one is provided for ARLAS EO. A reference can be an http url or a path to a local file. -It can also contain references to index mappings for creating indices. A default one is provided for ARLAS EO. +By default, the local configuration is created during the installation. -## Running +To list the available configurations, run the following command from the command line: ```shell -> arlas_cli --version -X.X.X -Warning : no configuration file found, we created an empty one for -you (~/.arlas/cli/configuration.yaml). +> arlas_cli confs list ++----------------------+------------------------------------------+ +| name | url | ++----------------------+------------------------------------------+ +| local | http://localhost/arlas | ++----------------------+------------------------------------------+ ``` -## Examples +!!! success + At least the `local` configuration has to be here + +It will be used in the tutorial as `--config local` + +For more details, see the [Configuration Guide](configuration.md#configuration). -In the following examples, you will see how to: +## Tutorial -- generate an elasticsearch mapping based on json objects -- add the mapping in elasticsearch -- list the elasticsearch indices -- add (index) data in the elasticsearch index -- get the structure of the mapping -- add a collection in ARLAS -- list the ARLAS collections -- get the structure of a ARLAS collection -- delete a collection from ARLAS -- delete a mapping from elasticsearch -- add, describe, get, list and delete an entry from ARLAS Persistence -- list, create, describe and delete a configuration for `arlas_cli` +In the following tutorial, you will see how to: + +- Generate an Elasticsearch mapping based on json objects +- Add the mapping in Elasticsearch +- List the Elasticsearch indices +- Add (index) data in the Elasticsearch index +- Get the structure of the mapping +- Add a collection in ARLAS +- List the ARLAS collections +- Get the structure of a ARLAS collection +- Delete a collection from ARLAS +- Delete a mapping from Elasticsearch +- Add, describe, get, list and delete an entry from ARLAS Persistence +- List, create, describe and delete a configuration for `arlas_cli` ... with the `arlas_cli` command line only! -We suppose you have an elasticsearch and arlas server running. +!!! warning "Prerequisite" + First, let's get a sample data file: + + ```shell + > curl -X GET https://raw.githubusercontent.com/gisaia/arlas-cli/master/tests/sample.json -o sample.json + ``` + The downloaded `sample.json` contains a sample of processed AIS data. + -### Generate the elasticsearch mapping +### Generate the Elasticsearch mapping -Writing the elasticsearch mapping for an index can be laborious. `arlas_cli` does it for you. `arlas_cli` can inspect a NDJSON file (one json object per line) and generate the corresponding elasticsearch mapping file. +Writing the elasticsearch mapping for an index can be laborious. `arlas_cli` does it for you. +`arlas_cli` can inspect a NDJSON file (one json object per line) and generate the corresponding elasticsearch mapping file. -First, let's get a sample data file: +To generate the mapping file based on that sample, run the following command: ```shell -> curl -X GET https://raw.githubusercontent.com/gisaia/arlas-cli/master/tests/sample.json -o sample.json +> arlas_cli indices \ + --config local \ + mapping sample.json ``` -Now we can generate the mapping file based on that sample: +By inspecting the mapping, we notice that the three timestamps are not identified as datetime by `arlas_cli`. + +!!! info "--field-mapping" + The `--field-mapping` option allows to overwrite the detected type. + +To generate the mapping with forced types, run the following command: ```shell > arlas_cli indices \ @@ -65,63 +97,60 @@ Now we can generate the mapping file based on that sample: --field-mapping track.timestamps.center:date-epoch_second \ --field-mapping track.timestamps.start:date-epoch_second \ --field-mapping track.timestamps.end:date-epoch_second \ - --push-on courses + --nb-lines 20 ``` -The `--push-on` option registers the mapping in the specified index. +The three timestamps are now well identified as datetime. -Note that the three timestamps are not identified as datetimes by `arlas_cli`. The `--field-mapping` allows to overwrite the detected type. +For more details, see the [Indices Mapping Guide](indices.md#mapping). -#### Type identification -A geometry is identified as such if +### Create an empty index from inferred mapping -- it is a geojson -- it is a WKT string -- the field name contains `geohash` -- it is a string containing two float seperated by a comma +Once the inferred mapping is fine, an elasticsearch index based on this mapping has to be created. -A date is identified as such if +!!! info "--push-on" + In the previous `arlas_cli indices mapping` command, the `--push-on` option registers the mapping in the specified index. -- its name is one of `timestamp`, `date`, `start` or `end` and that it can be parsed as a date -- its name contains `timestamp`, `date`, `start` or `end` and its values are number within [631152000, 4102444800] or [631152000000, 4102444800000] (year 1990 to 2100) - - -### Generate the elasticsearch mapping - -To add a specific mapping, it is possible to use the `create`` command: + An empty index is created with the inferred mapping +To create the associated empty index `ais_courses` with the `--push-on` option, run the following command: ```shell -> arlas_cli indices \ +> arlas_cli indices \ --config local \ - create courses \ - --mapping mapping.json + mapping sample.json \ + --field-mapping track.timestamps.center:date-epoch_second \ + --field-mapping track.timestamps.start:date-epoch_second \ + --field-mapping track.timestamps.end:date-epoch_second \ + --push-on ais_courses ``` -### List indices +For more details, see the [Indices Mapping Guide](indices.md#mapping). + +### Inspect the created indices + +To check that the index has been created, list the existing index: + ```shell > arlas_cli indices --config local list -+----------+--------+-------+--------+ -| name | status | count | size | -+----------+--------+-------+--------+ -| .arlas | open | 4 | 11.9kb | -| courses | open | 0 | 249b | -+----------+--------+-------+--------+ ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| ais_courses | open | 0 | 249b | ++--------------+--------+-------+--------+ ``` -### Add data - -```shell -> arlas_cli indices --config local data courses sample.json -``` +!!! success + The `ais_courses` index exists and contains 0 elements -### Describe an index +To describe the fields of the index, use the `arlas_cli indices describe` command: ```shell -> arlas_cli indices --config local describe courses +> arlas_cli indices --config local describe ais_courses +----------------------------------------------------+-----------+ | field name | type | +----------------------------------------------------+-----------+ @@ -136,248 +165,172 @@ To add a specific mapping, it is possible to use the `create`` command: +----------------------------------------------------+-----------+ ``` -### Add a collection - -```shell -> arlas_cli collections \ - --config local \ - create courses \ - --index courses --display-name courses \ - --id-path track.id \ - --centroid-path track.location \ - --geometry-path track.trail \ - --date-path track.timestamps.center -``` +It corresponds to the inferred mapping. +### Add data to index -### List collections +To add data to the created `ais_courses` index, use the `arlas_cli indices data` command with the data file `sample.json` ```shell -> arlas_cli collections --config local list -+---------+---------+ -| name | index | -+---------+---------+ -| courses | courses | -+---------+---------+ +> arlas_cli indices --config local data ais_courses sample.json ``` -### Describe a collection +To check that data has been correctly indexed, inspect the indices with: ```shell -> arlas_cli collections --config local describe courses -+----------------------------------------------------+-----------+ -| field name | type | -+----------------------------------------------------+-----------+ -| arrival.address.country | KEYWORD | -| arrival.address.port | KEYWORD | -| arrival.location | GEO_POINT | -| arrival.stop_after.duration_s | LONG | -... -| track.timestamps.start | DATE | -| track.trail | GEO_SHAPE | -| track.trail_geohashes_6 | GEO_POINT | -| track.visibility.proportion | FLOAT | -+----------------------------------------------------+-----------+ +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | +| ais_courses | open | 100 | 1mb | ++--------------+--------+-------+--------+ ``` -### Delete a collection - -```shell -> arlas_cli collections --config local delete courses -``` +!!! success + The `ais_courses` index now contains 100 elements. + +### Create a collection + +To explore the data in ARLAS, a collection has to be defined on top of an index. + +To create an `ais_courses` collection based on the `ais_courses` index, run the following command: -### Delete an index ```shell -> arlas_cli indices --config local delete courses +> arlas_cli collections \ + --config local \ + create ais_courses \ + --index ais_courses \ + --display-name "AIS Courses" \ + --id-path track.id \ + --centroid-path track.location \ + --geometry-path track.trail \ + --date-path track.timestamps.center ``` -!!! Note Delete is not always allowed - By default, it is not allowed to delete an index for a given configuration. To allow deleting, edit the configuration file and set `allow_delete` to `True`. +The `--index` option define the index to use and the `--display-name` define a pretty name used for collection in ARLAS. +Several elements define the data structure: -## ARLAS Persistence +- `--id-path`: The data field used as unique id of each element -### Add an entry +- `--centroid-path`: The data field containing a point geometry summarizing the location of each element (used for aggregation) - -```shell -> arlas_cli persist --config local add ../arlas-stacks4tests/conf/config.json config.json --name courses_dashboard -32d2624b-d7cd-11ee-9a91-0242ac130004 -``` +- `--geometry-path`: The data field containing a geometry to represent the element (can be point, linestring, polygon) -### Describe an entry +- `--date-path`: The data field containing the date associated to each element (used for timeline) - -```shell -> arlas_cli persist --config local describe 32d2624b-d7cd-11ee-9a91-0242ac130004 -+------------------+--------------------------------------+ -| metadata | value | -+------------------+--------------------------------------+ -| ID | 6a415cec-d7cd-11ee-9a91-0242ac130004 | -| ispublic | None | -| last_update_date | 1709298774600 | -| name | courses_dashboard | -| organization | None | -| owner | anonymous | -| updatable | True | -| zone | config.json | -+------------------+--------------------------------------+ -``` +For more details, see the [Collection Creation Guide](collections.md#create-an-arlas-collection). + +### Inspect the created collections -### Get an entry value +To list the available collections, run the following command: ```shell -> arlas_cli persist --config local get 32d2624b-d7cd-11ee-9a91-0242ac130004 -{ - "arlas": { - "web": { - "contributors": [ -... -} +> arlas_cli collections --config local list ++-------------+-------------+ +| name | index | ++-------------+-------------+ +| ais_courses | ais_courses | ++-------------+-------------+ ``` -### List entries within a zone +!!! success + The `ais_courses` collection is now created + +To describe the fields of the collection, use the `arlas_cli collections describe` command: ```shell -> arlas_cli persist --config local zone config.json -+--------------------------------------+-------------------+-------------+------------------+-----------+ -| id | name | zone | last_update_date | owner | -+--------------------------------------+-------------------+-------------+------------------+-----------+ -| 66984014-d0a1-11ee-b41c-0242ac190004 | Courses | config.json | 1708510231303 | anonymous | +> arlas_cli collections --config local describe ais_courses ++----------------------------------------------------+-----------+ +| field name | type | ++----------------------------------------------------+-----------+ +| arrival.address.country | KEYWORD | +| arrival.address.port | KEYWORD | +| arrival.location | GEO_POINT | +| arrival.stop_after.duration_s | LONG | ... -+--------------------------------------+-------------------+-------------+------------------+-----------+ +| track.timestamps.start | DATE | +| track.trail | GEO_SHAPE | +| track.trail_geohashes_6 | GEO_POINT | +| track.visibility.proportion | FLOAT | ++----------------------------------------------------+-----------+ ``` +It corresponds to the mapping of the data within the collection -### List groups accessing a zone +### Delete an index +To remove the indexed data from the local Elasticsearch instance, remove the index with the following command: ```shell -> arlas_cli persist --config local groups config.json -+--------------+ -| group | -+--------------+ -| group/public | -+--------------+ +> arlas_cli indices --config local delete ais_courses ``` - -### Delete an entry - +Check that `ais_courses` index no longer exists: ```shell -> arlas_cli persist --config local delete 32d2624b-d7cd-11ee-9a91-0242ac130004 -Resource 32d2624b-d7cd-11ee-9a91-0242ac130004 deleted. +> arlas_cli indices --config local list ++--------------+--------+-------+--------+ +| name | status | count | size | ++--------------+--------+-------+--------+ +| .arlas | open | 4 | 11.9kb | ++--------------+--------+-------+--------+ ``` +!!! tip + Before reindexing data, do not forget to [recreate the empty index from inferred mapping](#create-an-empty-index-from-inferred-mapping) + Collection does not need to be declared again. -## Configurations - -The command line uses the `${HOME}/.arlas/cli/configuration.yaml` configuration file: - -```yaml -arlas: - local: - allow_delete: true - elastic: - headers: - Content-Type: application/json - location: http://localhost:9200 - server: - headers: - Content-Type: application/json - location: http://localhost:9999/arlas -mappings: - arlas_eo: - headers: null - location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/mapping.json -models: - arlas_eo: - headers: null - location: https://raw.githubusercontent.com/gisaia/ARLAS-EO/master/collection.json -``` - -The `arlas` section contains the different deployment configurations. The mapping section lists the mapping template that you can use. - Finally, the models are the templates for the collections. A [detailed description](model/README.md) of the configuration structure is provided. - -### Create, describe and delete a configuration for `arlas_cli` - -You can edit directly the `${HOME}/.arlas/cli/configuration.yaml` configuration file to update your configurations. You can also use the command line itself. +### Delete a collection -To list the configurations: +To delete the `ais_courses` course collection ```shell -> arlas_cli confs list -+-----------+-----------------------------+ -| name | url | -+-----------+-----------------------------+ -| local | http://localhost:9999/arlas | -| test_conf | http://localhost:9999 | -+-----------+-----------------------------+ +> arlas_cli collections --config local delete courses ``` -To describe a configuration: +### Add an ARLAS Dashboard from configuration file + +The configuration of an ARLAS Dashboard describes all its elements (widgets, map layers...). Such a file can be directly imported to create a dashboard. +First, let's get an example of a dashboard configuration file: ```shell -arlas_cli confs describe local -allow_delete: true -authorization: null -elastic: - headers: - Content-Type: application/json - location: http://localhost:9200 -server: - headers: - Content-Type: application/json - location: http://localhost:9999/arlas +> curl -X GET https://raw.githubusercontent.com/gisaia/arlas-cli/master/tests/dashboard.json -o dashboard.json ``` - -To create a simple configuration: +The downloaded `dashboard.json` contains the configuration of a dashboard to explore AIS data. ```shell -> arlas_cli confs create dev_conf \ - --server http://localhost:9999 \ - --headers "Content-Type:application/json" \ - --elastic http://localhost:9200 \ - --elastic-headers "Content-Type:application/json" \ - --no-allow-delete +> arlas_cli persist --config local add ./dashboard.json config.json --name courses_dashboard +32d2624b-d7cd-11ee-9a91-0242ac130004 ``` -For an arlas configuration with authentication: +!!! note + The dashboard generated unique identifier is returned - -```shell -> arlas_cli --config-file /tmp/configuration.yaml confs \ - create myarlas_as_user \ - --server http://myserver/arlas \ - --headers "arlas-org-filter:my_org_name" \ - --headers "Content-Type:application/json" \ - --no-allow-delete \ - --auth-token-url http://myserver/arlas_iam_server/session \ - --auth-login user \ - --auth-password my_password \ - --auth-headers "Content-Type:application/json;charset=utf-8"\ - --auth-arlas-iam -``` +### List available dashboards -To delete the configuration: +The available dashboards can be list with the following command: ```shell -> arlas_cli confs delete dev_conf +> arlas_cli persist --config local zone config.json ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| id | name | zone | last_update_date | owner | ++--------------------------------------+-------------------+-------------+------------------+-----------+ +| 32d2624b-d7cd-11ee-9a91-0242ac130004 | courses_dashboard | config.json | 1708510231303 | anonymous | +... ++--------------------------------------+-------------------+-------------+------------------+-----------+ ``` -Also, it is possible to use a different configuration file than the one placed in your home directory (`$HOME/.arlas/cli/configuration.yaml`): +!!! note + The created dashboards can be accessed and managed in [ARLAS Hub](concepts.md#arlas-hub) and edited with [ARLAS Builder](concepts.md#arlas-builder). - -```shell -> arlas_cli --config-file /tmp/config.yaml -Warning : no configuration file found, we created an empty one for you (/tmp/config.yaml). -``` + The `arlas_cli` commands to manage dashboards are detailed in [Persistence Documentation](persist.md). diff --git a/docs/inject_commands.py b/docs/inject_commands.py new file mode 100644 index 0000000..915e12d --- /dev/null +++ b/docs/inject_commands.py @@ -0,0 +1,16 @@ +import sys +from pathlib import Path +import os + +for source in sys.argv[1:]: + print(source) + target = source.removesuffix(".template") + with open(source) as f, open(target, 'w') as f2: + for line in f.readlines(): + if line.startswith("> !!!execute "): + cmd = line.split("> !!!execute ")[1] + result = os.popen("stty cols 78; " + cmd).read() + f2.write("> " + cmd) + f2.write(result) + else: + f2.write(line) diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index c02f5d2..17990c5 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -27,21 +27,20 @@ theme: accent: 'light-blue' nav: -- arlas_cli version: version.md - About arlas_cli: index.md -- Installation: install.md - Getting started: started.md +- Installation: install.md +- Configuration: configuration.md - Commands: - Getting help: help.md - - Indices: indices.md - - Collections: collections.md - - Persistence: persist.md - - Identity & Access: iam.md - - Configurations: confs.md -- Configuration data model: model/model.md -- Configuring ARLAS Cloud: arlas_cloud.md + - confs: confs.md + - indices: indices.md + - collections: collections.md + - persist: persist.md + - iam: iam.md +- Concepts: concepts.md plugins: - termynal: prompt_literal_start: - - ">" + - ">" \ No newline at end of file diff --git a/scripts/mkdocs.sh b/scripts/mkdocs.sh index a8e713d..431c213 100755 --- a/scripts/mkdocs.sh +++ b/scripts/mkdocs.sh @@ -1,84 +1,7 @@ #!/bin/bash set -e -# INDICES -export PYTHONPATH=`pwd`/target/src -stty cols 70 - -echo "ARLAS Version:" > docs/docs/version.md -cat version.txt >> docs/docs/version.md - -cat docs/docs/indices_1.md > docs/docs/indices.md -# general -echo "> arlas_cli indices --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --help >> docs/docs/indices.md -cat docs/docs/indices_2.md >> docs/docs/indices.md - -# mapping -echo "> arlas_cli indices --config local mapping --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local mapping --help >> docs/docs/indices.md -cat docs/docs/indices_3.md >> docs/docs/indices.md - -# create -echo "> arlas_cli indices --config local create --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local create --help >> docs/docs/indices.md -cat docs/docs/indices_4.md >> docs/docs/indices.md - -# data -echo "> arlas_cli indices --config local data --help" >> docs/docs/indices.md -python3.10 -m arlas.cli.cli indices --config local data --help >> docs/docs/indices.md -cat docs/docs/indices_5.md >> docs/docs/indices.md - - -# COLLECTIONS -cat docs/docs/collections_1.md > docs/docs/collections.md -# general -echo "> arlas_cli collections --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --help >> docs/docs/collections.md -cat docs/docs/collections_2.md >> docs/docs/collections.md - -# create -echo "> arlas_cli collections --config local create --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --config local create --help >> docs/docs/collections.md -cat docs/docs/collections_3.md >> docs/docs/collections.md - -# describe -echo "> arlas_cli collections --config local describe --help" >> docs/docs/collections.md -python3.10 -m arlas.cli.cli collections --config local describe --help >> docs/docs/collections.md -cat docs/docs/collections_4.md >> docs/docs/collections.md - - -# PERSISTENCE -cat docs/docs/persist_1.md > docs/docs/persist.md -# general -echo "> arlas_cli persist --help" >> docs/docs/persist.md -python3.10 -m arlas.cli.cli persist --help >> docs/docs/persist.md -cat docs/docs/persist_2.md >> docs/docs/persist.md - - -# IAM -cat docs/docs/iam_1.md > docs/docs/iam.md -echo "> arlas_cli iam --config local orgs --help" >> docs/docs/iam.md -python3.10 -m arlas.cli.cli iam --config local orgs --help >> docs/docs/iam.md -cat docs/docs/iam_2.md >> docs/docs/iam.md -echo "> arlas_cli iam --config local users --help" >> docs/docs/iam.md -python3.10 -m arlas.cli.cli iam --config local users --help >> docs/docs/iam.md -cat docs/docs/iam_3.md >> docs/docs/iam.md - - -# CONFIGURATIONS -cat docs/docs/confs_1.md > docs/docs/confs.md - -# general -echo "> arlas_cli confs --help" >> docs/docs/confs.md -python3.10 -m arlas.cli.cli confs --help >> docs/docs/confs.md -cat docs/docs/confs_2.md >> docs/docs/confs.md - -# create -echo "> arlas_cli confs create --help" >> docs/docs/confs.md -python3.10 -m arlas.cli.cli confs create --help >> docs/docs/confs.md -cat docs/docs/confs_3.md >> docs/docs/confs.md - +python3.10 docs/inject_commands.py docs/docs/*.template pip3.10 install mkdocs-material termynal mkdocs build -f docs/mkdocs.yml -d ../target/docs