From 4296b571f30787310000861c4b3c96b2c4a590fe Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Wed, 24 Jun 2020 20:57:17 +0000 Subject: [PATCH] Update documentation --- docs/hive.md | 50 +++++++++++----------- docs/misc.md | 118 +++++++++++++++++++++++++-------------------------- 2 files changed, 85 insertions(+), 83 deletions(-) diff --git a/docs/hive.md b/docs/hive.md index d103ae1..3dedb9d 100644 --- a/docs/hive.md +++ b/docs/hive.md @@ -1,3 +1,27 @@ +# hive/glue-sync.sh + +Provides function to install Hive Glue Catalog Sync Agent + +* [install_glue_sync()](#installgluesync) + + +## install_glue_sync() + +Installs Hive Glue Catalog Sync Agent + +Requires Hive 2.x +Currently supported only on AWS + +### Example + +```bash +install_glue_sync us-east-1 +``` + +### Arguments + +* **$1** (string): Region for AWS Athena. Defaults to `us-east-1` + # hive/thrift-metastore.sh Provides functions to start/stop/restart thrift metastore server @@ -68,30 +92,8 @@ install_ranger -h example.host -p 6080 -r examplerepo * -h string Hostname of Ranger admin. Defaults to `localhost` * -p int Port where Ranger admin is running. Defaults to `6080` * -r string Name of Ranger repository. Defaults to `hivedev` - -# hive/glue-sync.sh - -Provides function to install Hive Glue Catalog Sync Agent - -* [install_glue_sync()](#installgluesync) - - -## install_glue_sync() - -Installs Hive Glue Catalog Sync Agent - -Requires Hive 2.x -Currently supported only on AWS - -### Example - -```bash -install_glue_sync us-east-1 -``` - -### Arguments - -* **$1** (string): Region for AWS Athena. Defaults to `us-east-1` +* -S string Hostname of Solr admin. Defaults to `""` +* -P int Port where Solr admin is running. Defaults to `6083` # hive/hiveserver2.sh diff --git a/docs/misc.md b/docs/misc.md index e4e1401..ca0ab3e 100644 --- a/docs/misc.md +++ b/docs/misc.md @@ -1,35 +1,45 @@ -# misc/mount_nfs.sh +# misc/util.sh -Provides function to mount a NFS volume +Provides miscellaneous utility functions -* [mount_nfs_volume()](#mountnfsvolume) +* [set_timezone()](#settimezone) +* [add_to_authorized_keys()](#addtoauthorizedkeys) -## mount_nfs_volume() +## set_timezone() -Mounts an NFS volume on master and worker nodes +Set the timezone -Instructions for AWS EFS mount: -1. After creating the EFS file system, create a security group -2. Create an inbound traffic rule for this security group that allows traffic on -port 2049 (NFS) from this security group as described here: -https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html -3. Add this security group as a persistent security group for the cluster from which -you want to mount the EFS store, as described here: -http://docs.qubole.com/en/latest/admin-guide/how-to-topics/persistent-security-group.html +This function sets the timezone on the cluster node. +The timezone to set is a mandatory parameter and must be present in /usr/share/zoneinfo +Eg: "US/Mountain", "America/Los_Angeles" etc. -TODO: add instructions for Azure file share +After setting the timezone, it is advised to restart engine daemons on the master and worker nodes ### Example ```bash -mount_nfs_volume "example.nfs.share:/" /mnt/efs +set_timezone "America/Los_Angeles" ``` ### Arguments -* **$1** (string): Path to NFS share -* **$2** (string): Mount point to use +* **$1** (string): Timezone to set + +## add_to_authorized_keys() + +Add a public key to authorized_keys + +### Example + +```bash +add_to_authorized_keys "ssh-rsa xyzxyzxyzxyz...xyzxyz user@example.com" ec2-user +``` + +### Arguments + +* **$1** (string): Public key to add to authorized_keys file +* **$2** (string): User for which the public key is added. Defaults to `ec2-user` # misc/python_venv.sh @@ -67,77 +77,67 @@ install_python_env 3.6 /path/to/virtualenv/py36 * **0**: Python virtualenv was created and activated * **1**: Python executable for virtualenv couldn't be found or installed -# misc/awscli.sh +# misc/mount_nfs.sh -Provides function to configure AWS CLI +Provides function to mount a NFS volume -* [configure_awscli()](#configureawscli) +* [mount_nfs_volume()](#mountnfsvolume) -## configure_awscli() +## mount_nfs_volume() -Configure AWS CLI +Mounts an NFS volume on master and worker nodes -A credentials file containing the AWS Access Key and the AWS Secret Key -separated by a space, comma, tab or newline must be provided +Instructions for AWS EFS mount: +1. After creating the EFS file system, create a security group +2. Create an inbound traffic rule for this security group that allows traffic on +port 2049 (NFS) from this security group as described here: +https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html +3. Add this security group as a persistent security group for the cluster from which +you want to mount the EFS store, as described here: +http://docs.qubole.com/en/latest/admin-guide/how-to-topics/persistent-security-group.html + +TODO: add instructions for Azure file share ### Example ```bash -configure_awscli -p exampleprofile -r us-east-1 -c /path/to/credentials/file +mount_nfs_volume "example.nfs.share:/" /mnt/efs ``` ### Arguments -* -p string Name of the profile. Defaults to `default` -* -r string AWS region. Defaults to `us-east-1` -* -c string Path to credentials file - -### Exit codes - -* **0**: AWS CLI is configured -* **1**: AWS CLI or credentials file not found - -# misc/util.sh +* **$1** (string): Path to NFS share +* **$2** (string): Mount point to use -Provides miscellaneous utility functions +# misc/awscli.sh -* [set_timezone()](#settimezone) -* [add_to_authorized_keys()](#addtoauthorizedkeys) +Provides function to configure AWS CLI +* [configure_awscli()](#configureawscli) -## set_timezone() -Set the timezone +## configure_awscli() -This function sets the timezone on the cluster node. -The timezone to set is a mandatory parameter and must be present in /usr/share/zoneinfo -Eg: "US/Mountain", "America/Los_Angeles" etc. +Configure AWS CLI -After setting the timezone, it is advised to restart engine daemons on the master and worker nodes +A credentials file containing the AWS Access Key and the AWS Secret Key +separated by a space, comma, tab or newline must be provided ### Example ```bash -set_timezone "America/Los_Angeles" +configure_awscli -p exampleprofile -r us-east-1 -c /path/to/credentials/file ``` ### Arguments -* **$1** (string): Timezone to set - -## add_to_authorized_keys() - -Add a public key to authorized_keys - -### Example - -```bash -add_to_authorized_keys "ssh-rsa xyzxyzxyzxyz...xyzxyz user@example.com" ec2-user -``` +* -p string Name of the profile. Defaults to `default` +* -r string AWS region. Defaults to `us-east-1` +* -c string Path to credentials file -### Arguments +### Exit codes -* **$1** (string): Public key to add to authorized_keys file -* **$2** (string): User for which the public key is added. Defaults to `ec2-user` +* **0**: AWS CLI is configured +* **1**: AWS CLI or credentials file not found