Skip to content

Commit

Permalink
fix link issues and typo
Browse files Browse the repository at this point in the history
  • Loading branch information
jiwenyu0531 committed Jun 29, 2020
1 parent 4769cbf commit 7cc8d9d
Show file tree
Hide file tree
Showing 14 changed files with 112 additions and 121 deletions.
9 changes: 0 additions & 9 deletions CONTRIBUTING.md

This file was deleted.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ A new connector is directly connected to another openLooKeng cluster to implemen
1. Parallel data access: Data sources are concurrently accessed to improve access efficiency. Clients can concurrently obtain data from the server to accelerate data obtaining.
2. Data compression: Data is compressed using GZIP compression algorithm before being serialized during data transmission, reducing the amount of data transmitted over the network.
3. Cross-DC dynamic filtering: Filters data to reduce the amount of data to be pulled from the remote end, ensuring network stability and improving query efficiency.
4. High availability: In OpenLooKeng, Coordinator AA is supported. Therefore, you can use a proxy (for example, Nginx) to implement load balancing among Coordinators to achieve high availability.If a coordinator is faulty, the availability of the entire cluster is not affected.
4. High availability: In openLooKeng, Coordinator AA is supported. Therefore, you can use a proxy (for example, Nginx) to implement load balancing among Coordinators to achieve high availability.If a coordinator is faulty, the availability of the entire cluster is not affected.

### Dynamic Filtering
In the multi-table join scenario with low correlation, most probe side rows are filtered out because they do not match the join conditions after being read. As a result, unnecessary join calculation, I/O read, and network transmission are caused. Dynamic filtering dynamically generates filtering conditions based on join conditions and data read from the build-side table during query running, and applies the filtering conditions to the table scan phase of the probe-side table. This reduces the data volume of the probe table that participates in the join operation, effectively reducing network transmission and improving performance by 30%.
Expand Down
4 changes: 2 additions & 2 deletions docker/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Hetu Docker Image
# openLooKeng Docker Image

## About the Container
This Docker image is designed to provide the following
* An out-of-the-box single node cluster with the JMX, memory, TPC-DS, and TPC-H
catalogs
* An image that can be deployed as a full cluster by mounting in configuration
* An image to be used as the basis for the Kubernetes Hetu operator
* An image to be used as the basis for the Kubernetes openLooKeng operator

## Configuration

Expand Down
66 changes: 33 additions & 33 deletions hetu-docs/en/_index.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,34 @@
+++

weight = 1
bookToc = false
title = "Release Notes"
+++

# Release Notes

| Feature | Description |
| -------------------------------------------- | ------------------------------------------------------------ |
| Adaptive Dynamic Filter | The dynamic feature is enhanced so that in addition to bloom filter, hashsets can be used to store the build side values to filter out the probe side. The filters are stored in a distributed memory store so that they can be reused by subsequent queries without having to be rebuilt. |
| Dynamically Add Catalog | Add catalog REST api allows an adminstrator to add new catalogs of any connector during run time. Catalog property files are written to a shared storage where they are discovered by hetu cluster nodes and are registered and loaded. |
| Cross Region Dynamic Filter | In a cross data center scenario when the probe side is in the remote data center and the build side is in the local data center, the adaptive filter will create a bloom filter. The data center connector is enhanced to send the bloom filter over the network to the remote data center, where it can be used to filter out the probe side. This reduces the amount of data that needs to be transferred over the network from the remote data center to the local data center. |
| Horizontal Scaling Support | Introduced ISOLATING and ISOLATED node states so that a node can be quiesced during scale-in. ISOLATED nodes do not accept any new tasks. |
| VDM + General Metastore | Virtual Data Mart, allows an administrator to create virtual catalogs, and virtual schems. Within a virtual schema, the administrator can create views over tables spanning multiple data sources. Virtual data marts simplifies the access to tables across data sources, and across regions. |
| IUD support for ORC | Support transactional Insert, Update, Delete on Hive ORC tables |
| Compaction for ORC | Support compaction of hive ORC transactional tables so that the number of files to be read is reduced there by increasing the data fetch per reader and hence helps in improving the query performance and also improves concurrency |
| Access Control for Update For Hive Connector | Support for validation of user rights and Grants for the Hive tables before accessing them.These rights and Grants are set in Hive Metastore through the Hive supported APIs/functions external to Hetu |
| IUD support for CarbonData | Support Insert, Update, Delete Operations on CarbonData tables |
| Insert Overwrite | Support for Insert overwrite syntax.This is easy method for Truncating and loading into the existing table |
| Sql Migration Tool | Supplementary tool to assist in migrating the Hive SQL to Hetu Compatible SQLs |
| ODBC connector | ODBC Driver and gateway for the 3rd party BI tools like PowerBI, Tableau, YongHong Desktop to connect to Hetu |
| Dynamic Hive UDF | Dynamically load custom hive UDFs into Hetu |
| HBase Connector | HBase Connector |
| CarbonData Connector | CarbonData Connector with read support |
| CREATE TABLE AS WITH LOCATION | Allow users to specify the external location of managed hive tables when running CREATE TABLE AS command in Hetu |
| CREATE TRANSACTIONAL Table | Allow users to create Hive transactional table in Hetu |
| RawSlice Optimization | This optimization reduces the memory footprint of OneQuery by reusing the RawSlice object instead of constructing new Slice objects. |
| Metadata Cache | A generic metadata cache SPI is introduced to provide a transparent caching layer that can be leveraged by any connector. The metadata cache delegates to the connector specific metadata if the cache does not exist. Currently being used by JDBC connectors, and DC Connector |
| Cross DC Connector | A new connector is introduced to support responsive queries across a WAN allowing a client to query a data source that is sitting in another data center that is physically far away. |
| IUD for ORC | Support Insert, Update, & Delete for ORC. Works with Hive 2 & Hive 3 transactional tables. Limitation: Only backward compatible with Hive 3 transactional tables, that is, if a Hive 3 transactional table is updated or deleted via OneQuery, then that table is still queryable in Hive. This is not the case for Hive 2 transactional tables. |
| HA AA | Supports HA AA mode by storing runtime state information into a distributed cache like Hazelcast. Hazelcast cluster formation is done using a seed file. Discovery Service, OOM, & CPU Usage uses a distributed lock to ensure only one coordinator starts these services. |
+++

weight = 1
bookToc = false
title = "Release Notes"
+++

# Release Notes

| Feature | Description |
| -------------------------------------------- | ------------------------------------------------------------ |
| Adaptive Dynamic Filter | The dynamic feature is enhanced so that in addition to bloom filter, hashsets can be used to store the build side values to filter out the probe side. The filters are stored in a distributed memory store so that they can be reused by subsequent queries without having to be rebuilt. |
| Dynamically Add Catalog | Add catalog REST api allows an adminstrator to add new catalogs of any connector during run time. Catalog property files are written to a shared storage where they are discovered by openLooKeng cluster nodes and are registered and loaded. |
| Cross Region Dynamic Filter | In a cross data center scenario when the probe side is in the remote data center and the build side is in the local data center, the adaptive filter will create a bloom filter. The data center connector is enhanced to send the bloom filter over the network to the remote data center, where it can be used to filter out the probe side. This reduces the amount of data that needs to be transferred over the network from the remote data center to the local data center. |
| Horizontal Scaling Support | Introduced ISOLATING and ISOLATED node states so that a node can be quiesced during scale-in. ISOLATED nodes do not accept any new tasks. |
| VDM + General Metastore | Virtual Data Mart, allows an administrator to create virtual catalogs, and virtual schems. Within a virtual schema, the administrator can create views over tables spanning multiple data sources. Virtual data marts simplifies the access to tables across data sources, and across regions. |
| IUD support for ORC | Support transactional Insert, Update, Delete on Hive ORC tables |
| Compaction for ORC | Support compaction of hive ORC transactional tables so that the number of files to be read is reduced there by increasing the data fetch per reader and hence helps in improving the query performance and also improves concurrency |
| Access Control for Update For Hive Connector | Support for validation of user rights and Grants for the Hive tables before accessing them.These rights and Grants are set in Hive Metastore through the Hive supported APIs/functions external to openLooKeng |
| IUD support for CarbonData | Support Insert, Update, Delete Operations on CarbonData tables |
| Insert Overwrite | Support for Insert overwrite syntax.This is easy method for Truncating and loading into the existing table |
| Sql Migration Tool | Supplementary tool to assist in migrating the Hive SQL to openLooKeng Compatible SQLs |
| ODBC connector | ODBC Driver and gateway for the 3rd party BI tools like PowerBI, Tableau, YongHong Desktop to connect to openLooKeng |
| Dynamic Hive UDF | Dynamically load custom hive UDFs into openLooKeng |
| HBase Connector | HBase Connector |
| CarbonData Connector | CarbonData Connector with read support |
| CREATE TABLE AS WITH LOCATION | Allow users to specify the external location of managed hive tables when running CREATE TABLE AS command in openLooKeng |
| CREATE TRANSACTIONAL Table | Allow users to create Hive transactional table in openLooKeng |
| RawSlice Optimization | This optimization reduces the memory footprint of OneQuery by reusing the RawSlice object instead of constructing new Slice objects. |
| Metadata Cache | A generic metadata cache SPI is introduced to provide a transparent caching layer that can be leveraged by any connector. The metadata cache delegates to the connector specific metadata if the cache does not exist. Currently being used by JDBC connectors, and DC Connector |
| Cross DC Connector | A new connector is introduced to support responsive queries across a WAN allowing a client to query a data source that is sitting in another data center that is physically far away. |
| IUD for ORC | Support Insert, Update, & Delete for ORC. Works with Hive 2 & Hive 3 transactional tables. Limitation: Only backward compatible with Hive 3 transactional tables, that is, if a Hive 3 transactional table is updated or deleted via OneQuery, then that table is still queryable in Hive. This is not the case for Hive 2 transactional tables. |
| HA AA | Supports HA AA mode by storing runtime state information into a distributed cache like Hazelcast. Hazelcast cluster formation is done using a seed file. Discovery Service, OOM, & CPU Usage uses a distributed lock to ensure only one coordinator starts these services. |
| Implicit Conversion | This feature supports data type implicit conversion. For example: If the query type does not match the table type in the Insert statement, it can convert the query type to the table type implicitly. |
22 changes: 11 additions & 11 deletions hetu-docs/en/installation/deployment-auto.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ In addition to the manual deployment of openLooKeng Sever, you can follow below
Execute below command can help you download the necessary packages and deploy openLooKeng server in one-click:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh)
bash <(wget -qO- https://download.openlookeng.io/install.sh)
```

or:

```shell
wget -O - http://openlookeng.io/install.sh|bash
wget -O - https://download.openlookeng.io/install.sh|bash
```

Normally, you don\'t need to do any thing, except for the installation to complete. It will automatically start the service.
Expand All @@ -47,13 +47,13 @@ Execute below command to start openLooKeng Command Line client:
Execute below command to install openLooKeng cluster:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh) -m
bash <(wget -qO- https://download.openlookeng.io/install.sh) -m
```

or:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh) --multi-node
bash <(wget -qO- https://download.openlookeng.io/install.sh) --multi-node
```

First of all, this command will download scripts and packages required by openLooKeng service. After the download is completed, it will check whether the dependent packages `expect` and `sshpass` are installed. If not, those dependencies will be installed automatically.
Expand Down Expand Up @@ -104,16 +104,16 @@ Execute below command to start openLooKeng Command Line client.:

If you are going to deploy openLooKeng on a big cluster with lots of nodes, instead of inputting the nodes' IP address one by one. It is better to prepare a file containing all nodes' IP address then pass this file as parameter to the installation script. Here is the command:

`bash <(wget -qO- http://openlookeng.io/install.sh) -f <cluster_node_info_path>`
`bash <(wget -qO- https://download.openlookeng.io/install.sh) -f <cluster_node_info_path>`
or:
`bash <(wget -qO- http://openlookeng.io/install.sh) --file <cluster_node_info_path>`
`bash <(wget -qO- https://download.openlookeng.io/install.sh) --file <cluster_node_info_path>`



For more help message,execute below command to deploy single node cluster:
`bash <(wget -qO- http://openlookeng.io/install.sh) -h`
`bash <(wget -qO- https://download.openlookeng.io/install.sh) -h`
or:
`bash <(wget -qO- http://openlookeng.io/install.sh) --help`
`bash <(wget -qO- https://download.openlookeng.io/install.sh) --help`



Expand All @@ -122,21 +122,21 @@ or:
Execute below command to Upgrade openLooKeng Service:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh) -u <version>
bash <(wget -qO- https://download.openlookeng.io/install.sh) -u <version>
```

This command will upgrade the current openLooKeng Service to target version,
preserving all the existing configurations on current cluster. Execute
below command to list all available versions:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh) -l
bash <(wget -qO- https://download.openlookeng.io/install.sh) -l
```

or:

```shell
bash <(wget -qO- http://openlookeng.io/install.sh) --list
bash <(wget -qO- https://download.openlookeng.io/install.sh) --list
```

## Deploying Configuration to openLooKeng Cluster
Expand Down
4 changes: 2 additions & 2 deletions hetu-docs/en/installation/odbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,9 @@ Before an application uses the openLooKeng ODBC driver, the data source DSN must

1. In the ODBC Data Source Administrator (64-bit), click the **User DSN** tab and click **Add**.

2. In the displayed **Create New Data Source** dialog box, click **Hetu ODBC Driver** and click **Finish**.
2. In the displayed **Create New Data Source** dialog box, click **openLooKeng ODBC Driver** and click **Finish**.

3. The Hetu ODBC Driver configuration page is displayed. On the welcome page, enter the name of the DSN to be created in the **Name** text box, enter the additional description of the DSN in the **Description** text box, and click **Next**.
3. The openLooKeng ODBC Driver configuration page is displayed. On the welcome page, enter the name of the DSN to be created in the **Name** text box, enter the additional description of the DSN in the **Description** text box, and click **Next**.

4. The second page contains the following six text boxes. The functions and usage are as follows:

Expand Down
4 changes: 2 additions & 2 deletions hetu-docs/en/start/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ title = "Get Started"

# Get Started

This document provides guidance for users to quickly deploy and start the openLooKeng service on the local host. For details about the installation requirements and deployment mode, see [Installation Guide] (https://www.openlookeng.io/docs/docs/installation.html).
This document provides guidance for users to quickly deploy and start the openLooKeng service on the local host. For details about the installation requirements and deployment mode, see [Installation Guide](https://www.openlookeng.io/docs/docs/installation.html).

## Preparing the Environment

Expand Down Expand Up @@ -48,7 +48,7 @@ After successful deployment, you can learn the following information to better u

- One-click deployment also provides scripts for common management commands. You can find `/opt/openlookeng/bin` here.

- OpenLooKeng run logs are stored in `/home/openlookeng/.openlkadmin/`.
- openLooKeng run logs are stored in `/home/openlookeng/.openlkadmin/`.

- One-click deployment also provides [CLI](https://www.openlookeng.io/docs/docs/installation/cli.html) for connecting to the openLooKeng service.

Expand Down
Loading

0 comments on commit 7cc8d9d

Please sign in to comment.