Skip to content

Commit

Permalink
[Refactor](Tvf) delete some unused code of tvf and add doc for `queri…
Browse files Browse the repository at this point in the history
…es` tvf (apache#26460)

1. delete some unused code of tvf
2. add doc for `queries` tvf: apache#25051
  • Loading branch information
BePPPower authored Nov 9, 2023
1 parent 66e591f commit 7df60a4
Show file tree
Hide file tree
Showing 8 changed files with 168 additions and 50 deletions.
79 changes: 79 additions & 0 deletions docs/en/docs/sql-manual/sql-functions/table-functions/queries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
{
"title": "QUERIES",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

## `queries`

### Name

<version since="dev">

queries

</version>

### description

Table-Value-Function, generate a temporary table named queries. This tvf is used to view the information of running queries and history queries in doris cluster.

This function is used in FROM clauses.

#### syntax
`queries()`

queries() table schema:
```
mysql> desc function queries();
+------------------+--------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------+------+-------+---------+-------+
| QueryId | TEXT | No | false | NULL | NONE |
| StartTime | BIGINT | No | false | NULL | NONE |
| EndTime | BIGINT | Yes | false | NULL | NONE |
| EventTime | BIGINT | Yes | false | NULL | NONE |
| Latency | BIGINT | No | false | NULL | NONE |
| State | TEXT | No | false | NULL | NONE |
| Database | TEXT | Yes | false | NULL | NONE |
| Sql | TEXT | No | false | NULL | NONE |
| FrontendInstance | TEXT | No | false | NULL | NONE |
+------------------+--------+------+-------+---------+-------+
9 rows in set (0.00 sec)
```

### example
```
mysql> select* from queries();
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
| QueryId | StartTime | EndTime | EventTime | Latency | State | Database | Sql | FrontendInstance |
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
| e1293f2ed2a5427a-982301c462586043 | 1699255138730 | 1699255139823 | 1699255139823 | 1093 | FINISHED | demo | select* from queries() | localhost |
| 46fa3ad0e7814ebd-b1cd34940a29b1e9 | 1699255143588 | -1 | 1699255143588 | 20 | RUNNING | demo | select* from queries() | localhost |
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
2 rows in set (0.04 sec)
```

### keywords

queries
5 changes: 3 additions & 2 deletions docs/sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -749,7 +749,8 @@
"sql-manual/sql-functions/table-functions/frontends",
"sql-manual/sql-functions/table-functions/workload-group",
"sql-manual/sql-functions/table-functions/catalogs",
"sql-manual/sql-functions/table-functions/frontends_disks"
"sql-manual/sql-functions/table-functions/frontends_disks",
"sql-manual/sql-functions/table-functions/queries"
]
},
{
Expand Down Expand Up @@ -1342,4 +1343,4 @@
]
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
{
"title": "QUERIES",
"language": "zh-CN"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

## `queries`

### Name

<version since="dev">

queries

</version>

### description

表函数,生成queries临时表,可以查看当前doris集群中正在运行的以及历史的 query 信息。

该函数用于from子句中。

#### syntax
`queries()`

queries()表结构:
```
mysql> desc function queries();
+------------------+--------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------+------+-------+---------+-------+
| QueryId | TEXT | No | false | NULL | NONE |
| StartTime | BIGINT | No | false | NULL | NONE |
| EndTime | BIGINT | Yes | false | NULL | NONE |
| EventTime | BIGINT | Yes | false | NULL | NONE |
| Latency | BIGINT | No | false | NULL | NONE |
| State | TEXT | No | false | NULL | NONE |
| Database | TEXT | Yes | false | NULL | NONE |
| Sql | TEXT | No | false | NULL | NONE |
| FrontendInstance | TEXT | No | false | NULL | NONE |
+------------------+--------+------+-------+---------+-------+
9 rows in set (0.00 sec)
```

### example
```
mysql> select* from queries();
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
| QueryId | StartTime | EndTime | EventTime | Latency | State | Database | Sql | FrontendInstance |
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
| e1293f2ed2a5427a-982301c462586043 | 1699255138730 | 1699255139823 | 1699255139823 | 1093 | FINISHED | demo | select* from queries() | localhost |
| 46fa3ad0e7814ebd-b1cd34940a29b1e9 | 1699255143588 | -1 | 1699255143588 | 20 | RUNNING | demo | select* from queries() | localhost |
+-----------------------------------+---------------+---------------+---------------+---------+----------+----------+------------------------+------------------+
2 rows in set (0.04 sec)
```

### keywords

queries
13 changes: 0 additions & 13 deletions fe/fe-core/src/main/java/org/apache/doris/load/ExportFailMsg.java
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,6 @@ public enum CancelType {
@SerializedName("msg")
private String msg;

public ExportFailMsg() {
this.cancelType = CancelType.UNKNOWN;
this.msg = "";
}

public ExportFailMsg(CancelType cancelType, String msg) {
this.cancelType = cancelType;
this.msg = msg;
Expand All @@ -54,18 +49,10 @@ public CancelType getCancelType() {
return cancelType;
}

public void setCancelType(CancelType cancelType) {
this.cancelType = cancelType;
}

public String getMsg() {
return msg;
}

public void setMsg(String msg) {
this.msg = msg;
}

@Override
public String toString() {
return "ExportFailMsg [cancelType=" + cancelType + ", msg=" + msg + "]";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,6 @@ public ExportJob(long jobId) {
public void generateOutfileStatement() throws UserException {
exportTable.readLock();
try {
// generateQueryStmtOld
generateQueryStmt();
} finally {
exportTable.readUnlock();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.BrokerUtil;
Expand Down Expand Up @@ -311,13 +310,13 @@ public List<Column> getTableColumns() throws AnalysisException {
if (!csvSchema.isEmpty()) {
return csvSchema;
}
if (FeConstants.runningUnitTest) {
Object mockedUtObj = FeConstants.unitTestConstant;
if (mockedUtObj instanceof List) {
return ((List<Column>) mockedUtObj);
}
return new ArrayList<>();
}
// if (FeConstants.runningUnitTest) {
// Object mockedUtObj = FeConstants.unitTestConstant;
// if (mockedUtObj instanceof List) {
// return ((List<Column>) mockedUtObj);
// }
// return new ArrayList<>();
// }
if (this.columns != null) {
return columns;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ private void init(Map<String, String> properties) throws AnalysisException {
// 3. analyze other properties
for (String key : otherProps.keySet()) {
if (HdfsResource.HADOOP_FS_NAME.equalsIgnoreCase(key)) {
// because HADOOP_FS_NAME contains upper and lower case
locationProperties.put(HdfsResource.HADOOP_FS_NAME, otherProps.get(key));
} else {
locationProperties.put(key, otherProps.get(key));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,12 @@
import org.apache.doris.catalog.Env;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.ClientPool;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.common.proc.FrontendsProcNode;
import org.apache.doris.common.util.NetUtils;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.HMSExternalCatalog;
import org.apache.doris.datasource.property.constants.HMSProperties;
import org.apache.doris.planner.external.iceberg.IcebergMetadataCache;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.QueryDetail;
Expand All @@ -55,10 +52,7 @@
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.gson.Gson;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.iceberg.Snapshot;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.thrift.TException;
Expand All @@ -67,13 +61,10 @@
import java.time.Instant;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;



public class MetadataGenerator {
private static final Logger LOG = LogManager.getLogger(MetadataGenerator.class);

Expand Down Expand Up @@ -480,22 +471,6 @@ private static void filterColumns(TFetchSchemaTableDataResult result,
result.setDataBatch(filterColumnsRows);
}

private static org.apache.iceberg.Table getIcebergTable(HMSExternalCatalog catalog, String db, String tbl)
throws MetaNotFoundException {
org.apache.iceberg.hive.HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
Configuration conf = new HdfsConfiguration();
Map<String, String> properties = catalog.getCatalogProperty().getHadoopProperties();
for (Map.Entry<String, String> entry : properties.entrySet()) {
conf.set(entry.getKey(), entry.getValue());
}
hiveCatalog.setConf(conf);
Map<String, String> catalogProperties = new HashMap<>();
catalogProperties.put(HMSProperties.HIVE_METASTORE_URIS, catalog.getHiveMetastoreUris());
catalogProperties.put("uri", catalog.getHiveMetastoreUris());
hiveCatalog.initialize("hive", catalogProperties);
return hiveCatalog.loadTable(TableIdentifier.of(db, tbl));
}

private static long convertToDateTimeV2(
int year, int month, int day, int hour, int minute, int second, int microsecond) {
return (long) microsecond | (long) second << 20 | (long) minute << 26 | (long) hour << 32
Expand Down

0 comments on commit 7df60a4

Please sign in to comment.