Merge branch '2.1.0_dev' of github.com:Teingi/obdiag into 2.1.0_dev

README-CN.md

@@ -1,18 +1,18 @@
 <p align="center">
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool/blob/master/LICENSE">
+    <a href="https://github.com/oceanbase/obdiag/blob/master/LICENSE">
         <img alt="license" src="https://img.shields.io/badge/license-MulanPubL--2.0-blue" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool/releases/latest">
-        <img alt="license" src="https://img.shields.io/badge/dynamic/json?color=blue&label=release&query=tag_name&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool%2Freleases%2Flatest" />
+    <a href="https://github.com/oceanbase/obdiag/releases/latest">
+        <img alt="license" src="https://img.shields.io/badge/dynamic/json?color=blue&label=release&query=tag_name&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag%2Freleases%2Flatest" />
     </a>
     <a href="https://img.shields.io/badge/python%20-3.8.0%2B-blue.svg">
         <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.8.0%2B-blue.svg" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool">
-        <img alt="stars" src="https://img.shields.io/badge/dynamic/json?color=blue&label=stars&query=stargazers_count&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool" />
+    <a href="https://github.com/oceanbase/obdiag">
+        <img alt="stars" src="https://img.shields.io/badge/dynamic/json?color=blue&label=stars&query=stargazers_count&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool">
-        <img alt="forks" src="https://img.shields.io/badge/dynamic/json?color=blue&label=forks&query=forks&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool" />
+    <a href="https://github.com/oceanbase/obdiag">
+        <img alt="forks" src="https://img.shields.io/badge/dynamic/json?color=blue&label=forks&query=forks&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag" />
     </a>
     <a href="https://www.oceanbase.com/docs/obdiag-cn">
         <img alt="Chinese doc" src="https://img.shields.io/badge/文档-简体中文-blue" />
@@ -41,12 +41,6 @@ sh /usr/local/oceanbase-diagnostic-tool/init.sh
 ```
 
 ## 方式二：源码安装
-源码编译环境确保有如下依赖
-- gcc
-- wget
-- python-devel
-- mysql-devel
-
 源码安装需要在python >= 3.8的环境下进行
 
 ```shell
@@ -62,49 +56,45 @@ obdiag config -h <db_host> -u <sys_user> [-p password] [-P port]
 ```
 
 # obdiag 功能
+- 一键集群巡检
+- 一键诊断分析
+- 一键根因分析
+- 一键信息采集
 
-## obdiag 巡检功能
-- [一键巡检](./docs/check.md)
 
-## obdiag 一键场景化信息采集功能
-- [一键场景化信息采集](./docs/gather_scene.md)
+# 参与贡献
 
-## obdiag 一键信息采集功能
+obdiag 期望构建一个开放的社区，我们欢迎任何形式的贡献，您可以：
+- 通过 [Issues](https://github.com/oceanbase/obdiag/issues) 提交 bug。
+- 通过 [Discussion](https://github.com/oceanbase/obdiag/discussions) 参与或发起讨论。
+- 通过 [Pull requests](https://github.com/oceanbase/obdiag/pulls) 提交问题修复或者功能特性。
 
-- [一键收集OB日志](./docs/gather_ob_log.md)
-- [一键收集AWR报告](./docs/gather_awr.md)
-- [一键收集主机信息](./docs/gather_sysstat.md)
-- [一键收集slog/clog日志](./docs/gather_admin.md)
-- [一键收集火焰图信息](./docs/gather_perf.md)
-- [一键收集OB堆栈信息](./docs/gather_ob_stack.md)
-- [一键收集并行SQL的执行详情信息](./docs/gather_sql_plan_monitor.md)
-- [一键收集OBPROXY日志](./docs/gather_obproxy_log.md)
-- [一键收集AWR报告](./docs/gather_awr.md)
-- [一键收集全部诊断信息](./docs/gather_all.md)
+# Roadmap Ahead
 
-## obdiag 一键分析功能说明
-- [一键分析OB日志](./docs/analyze_ob_log.md)
-- [一键全链路诊断](./docs/analyze_flt_trace.md)
+| 版本 | 迭代周期 | 功能点 | 
+|---------|---------------|---------|
+|1.6.0| 2024.01| <ul><li> 场景化信息采集 </li><li> 场景化根因分析 </li></ul> |
+|2.0.0|2024.03| <ul><li> context改造，场景化扩展能力增强 </li><li> 支持在线更新巡检、采集的task </li><li> 根因分析二期 </li></ul>|
+|2.1.0|2024.04| <ul><li> 根因分析场景扩展 </li><li> 新增 ash 报告 采集 </li></ul>|
+|2.2.0|2024.05| <ul><li> 根因分析场景扩展 </li></ul>|
+|2.3.0|2024.06| <ul><li> 根因分析场景扩展 </li><li> 支持 SQL 诊断 </li></ul>|
+|2.4.0|2024.07| <ul><li> 根因分析场景扩展 </li><li> 适配两款内核的诊断工具 </li></ul>|
+|2.5.0|2024.08| <ul><li> 根因分析场景扩展 </li><li> 支持 OMS 诊断 </li></ul>|
+|3.0.0|2024.09| <ul><li> 根因分析场景扩展 </li><li> 服务化改造 </li></ul>|
+|3.1.0|2024.10| <ul><li> 根因分析场景扩展 </li><li> 支持巡检报告比对 </li></ul>|
+|3.2.0|2024.11| <ul><li> 根因分析场景扩展 </li><li> SQL 诊断二期，支持SQL问题的根因分析 </li></ul>|
+|3.3.0|2024.12| <ul><li> AI 化探索 </li></ul>|
 
-## obdiag 一键场景化根因分析功能
-- [一键场景化根因分析](./docs/rca.md)
 
 # 许可证
 
 OceanBase Diagnostic Tool 使用 [MulanPSL - 2.0](http://license.coscl.org.cn/MulanPSL2) 许可证。
 您可以免费复制及使用源代码。当您修改或分发源代码时，请遵守木兰协议。
 
-
-## 贡献
-
-我们热烈欢迎并高度赞赏您的贡献。您可以通过以下几种方式做出贡献：
-
-- 向我们提出一个[issue](https://github.com/oceanbase/oceanbase-diagnostic-tool/issues)。
-- 提交请求。
-
 ## 支持
 
 如果您在使用 OceanBase LogProxy 时遇到任何问题，欢迎联系我们寻求帮助：
 
-- [GitHub Issue](https://github.com/oceanbase/oceanbase-diagnostic-tool/issues)
+- [GitHub Issue](https://github.com/oceanbase/obdiag/issues)
 - [官方网站](https://www.oceanbase.com/docs/obdiag-cn)
+- obdiag SIG 微信号: obdiagsig

README.md

@@ -1,20 +1,20 @@
 English | [中文版](README-CN.md)
 
 <p align="center">
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool/blob/master/LICENSE">
+    <a href="https://github.com/oceanbase/obdiag/blob/master/LICENSE">
         <img alt="license" src="https://img.shields.io/badge/license-MulanPubL--2.0-blue" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool/releases/latest">
-        <img alt="license" src="https://img.shields.io/badge/dynamic/json?color=blue&label=release&query=tag_name&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool%2Freleases%2Flatest" />
+    <a href="https://github.com/oceanbase/obdiag/releases/latest">
+        <img alt="license" src="https://img.shields.io/badge/dynamic/json?color=blue&label=release&query=tag_name&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag%2Freleases%2Flatest" />
     </a>
     <a href="https://img.shields.io/badge/python%20-3.8.0%2B-blue.svg">
         <img alt="pyversions" src="https://img.shields.io/badge/python%20-3.8.0%2B-blue.svg" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool">
-        <img alt="stars" src="https://img.shields.io/badge/dynamic/json?color=blue&label=stars&query=stargazers_count&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool" />
+    <a href="https://github.com/oceanbase/obdiag">
+        <img alt="stars" src="https://img.shields.io/badge/dynamic/json?color=blue&label=stars&query=stargazers_count&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag" />
     </a>
-    <a href="https://github.com/oceanbase/oceanbase-diagnostic-tool">
-        <img alt="forks" src="https://img.shields.io/badge/dynamic/json?color=blue&label=forks&query=forks&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Foceanbase-diagnostic-tool" />
+    <a href="https://github.com/oceanbase/obdiag">
+        <img alt="forks" src="https://img.shields.io/badge/dynamic/json?color=blue&label=forks&query=forks&url=https%3A%2F%2Fapi.github.com%2Frepos%2Foceanbase%2Fobdiag" />
     </a>
     <a href="https://www.oceanbase.com/docs/obdiag-cn">
         <img alt="Chinese doc" src="https://img.shields.io/badge/文档-简体中文-blue" />
@@ -41,15 +41,7 @@ sh /usr/local/oceanbase-diagnostic-tool/init.sh
 ```
 
 ## Method 2: Install obdiag by using the source code
-
-Before you install obdiag by using the source code, make sure that you have installed these dependencies:
-
-- gcc
-- wget
-- python-devel
-- mysql-devel
-
-To install obdiag on Python3.8, run these commands:
+To install obdiag on Python >= 3.8, run these commands:
 
 ```shell
 pip3 install -r requirements3.txt
@@ -64,48 +56,48 @@ obdiag config -h <db_host> -u <sys_user> [-p password] [-P port]
 ```
 
 # obdiag Fuctions
+- One-click cluster inspection
+- One-click diagnostic analyze
+- One-click root cause analysis
+- One-click information collection
+
+For more details, please refer to [Official docs](https://www.oceanbase.com/docs/obdiag-cn)
+
+# Join the Contributing Community
 
-## obdiag check Fuctions
-- [check](./docs/check.md)
+obdiag envisions an open community. We welcome your contributions in any form:
 
-## obdiag gather scene Fuctions
-- [gather scene](./docs/gather_scene.md)
+- Report bugs through [Issues](https://github.com/oceanbase/obdiag/issues).
+- Participate in or initiate discussions via [Discussion](https://github.com/oceanbase/obdiag/discussions).
+- Contribute bug fixes or new features through [Pull requests](https://github.com/oceanbase/obdiag/pulls).
 
-## obdiag gather Fuctions
 
-- [gather log](./docs/gather_ob_log.md)
-- [gather sysstat](./docs/gather_sysstat.md)
-- [gather slog/clog](./docs/gather_admin.md)
-- [gather perf](./docs/gather_perf.md)
-- [gather obstack](./docs/gather_ob_stack.md)
-- [gather sql plan monitor](./docs/gather_sql_plan_monitor.md)
-- [gather obproxy_log](./docs/gather_obproxy_log.md)
-- [gather awr](./docs/gather_awr.md)
-- [gather all](./docs/gather_all.md)
 
-## obdiag analyze Fuctions
-- [analyze log](./docs/analyze_ob_log.md)
-- [analyze flt trace log](./docs/analyze_flt_trace.md)
+# Roadmap Ahead
 
-## obdiag rca Fuctions
-- [rca](./docs/rca.md)
+| Version | Iteration Period | Function | 
+|---------|---------------|---------|
+|1.6.0| 2024.01| <ul><li> Scenario based fault information collection </li><li> Scenario based root cause analysis </li></ul> |
+|2.0.0|2024.03| <ul><li> Context Transformation, Enhanced Scene Expansion Capabilities</li><li> Support online updating of inspection and gather tasks </li><li>Root Cause Analysis Phase II Transformation </li></ul>|
+|2.1.0|2024.04| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Gather ash report </li></ul>|
+|2.2.0|2024.05| <ul><li> Root Cause Analysis Scenario Expansion </li></ul>|
+|2.3.0|2024.06| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Support SQL Diagnosis </li></ul>|
+|2.4.0|2024.07| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Adapting Two Additional Kernel Diagnostic Tools </li></ul>|
+|2.5.0|2024.08| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Support OMS Diagnosis </li></ul>|
+|3.0.0|2024.09| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Service-ification Transformation </li></ul>|
+|3.1.0|2024.10| <ul><li> Root Cause Analysis Scenario Expansion </li><li> Supporting Comparative Functionality for Patrol Inspection Reports </li></ul>|
+|3.2.0|2024.11| <ul><li> Root Cause Analysis Scenario Expansion </li><li> SQL Diagnosis Phase II, Supporting Root Cause Analysis for SQL problems </li></ul>|
+|3.3.0|2024.12| <ul><li> AI for obdiag </li></ul>|
 
 # Licencing
 OceanBase Database is under MulanPubL - 2.0 license. You can freely copy and use the source code. When you modify or
 distribute the source code, please obey the MulanPubL - 2.0 license.
 
-
-# Contributing
-
-Contributions are warmly welcomed and greatly appreciated. Here are a few ways you can contribute:
-
-- Raise us an [issue](https://github.com/oceanbase/oceanbase-diagnostic-tool/issues).
-- Submit Pull Requests.
-
 # Support
 
 In case you have any problems when using obdiag, welcome reach out for help:
 
-- [GitHub Issue](https://github.com/oceanbase/oceanbase-diagnostic-tool/issues)
+- [GitHub Issue](https://github.com/oceanbase/obdiag/issues)
 - [Official Website](https://www.oceanbase.com/docs/obdiag-cn)
+- obdiag SIG WeChat: obdiagsig
 

cmd.py

@@ -322,6 +322,8 @@ def do_command(self):
             return False
         cmd = '%s %s' % (self.prev_cmd, base)
         ROOT_IO.track_limit += 1
+        if "main.py" in cmd:
+            telemetry.work_tag=False
         telemetry.push_cmd_info("cmd: {0}. args:{1}".format(cmd,args))
         return self.commands[base].init(cmd, args).do_command()
 
@@ -560,19 +562,50 @@ def _do_command(self, obdiag):
         return obdiag.gather_function('gather_scenes_run', self.opts)
 
 
+class ObdiagGatherAshReportCommand(ObdiagOriginCommand):
+
+    def __init__(self):
+        super(ObdiagGatherAshReportCommand, self).__init__('ash', 'Gather ash report')
+        self.parser.add_option('--trace_id', type='string',
+                               help="The TRACE.ID of the SQL to be sampled, if left blank or filled with NULL, indicates that TRACE.ID is not restricted.")
+        self.parser.add_option('--sql_id', type='string',
+                               help="The SQL.ID, if left blank or filled with NULL, indicates that SQL.ID is not restricted.")
+        # WAIT_CLASS
+        self.parser.add_option('--wait_class', type='string',
+                               help='Event types to be sampled.')
+        self.parser.add_option('--report_type', type='string',
+                               help='Report type, currently only supports text type.', default='TEXT')
+        self.parser.add_option('--from', type='string',
+                               help="specify the start of the time range. format: 'yyyy-mm-dd hh:mm:ss'")
+        self.parser.add_option('--to', type='string',
+                               help="specify the end of the time range. format: 'yyyy-mm-dd hh:mm:ss'")
+        self.parser.add_option('--store_dir', type='string',
+                               help='the dir to store gather result, current dir by default.', default='./')
+
+        self.parser.add_option('-c', type='string', help='obdiag custom config',
+                               default=os.path.expanduser('~/.obdiag/config.yml'))
+
+    def init(self, cmd, args):
+        super(ObdiagGatherAshReportCommand, self).init(cmd, args)
+        return self
+
+    def _do_command(self, obdiag):
+        return obdiag.gather_function('gather_ash_report', self.opts)
+
+
 class ObdiagAnalyzeLogCommand(ObdiagOriginCommand):
 
     def __init__(self):
         super(ObdiagAnalyzeLogCommand, self).__init__('log', 'Analyze oceanbase log from online observer machines or offline oceanbase log files')       
         self.parser.add_option('--from', type='string', help="specify the start of the time range. format: 'yyyy-mm-dd hh:mm:ss'")
         self.parser.add_option('--to', type='string', help="specify the end of the time range. format: 'yyyy-mm-dd hh:mm:ss'")
-        self.parser.add_option('--since', type='string', help="Specify time range that from 'n' [d]ays, 'n' [h]ours or 'n' [m]inutes. before to now. format: <n> <m|h|d>. example: 1h.", default='30m')
         self.parser.add_option('--scope', type='string', help="log type constrains, choices=[observer, election, rootservice, all]", default='all')
         self.parser.add_option('--grep', action="append", type='string', help="specify keywords constrain")
         self.parser.add_option('--log_level', type='string', help="oceanbase logs greater than or equal to this level will be analyze, choices=[DEBUG, TRACE, INFO, WDIAG, WARN, EDIAG, ERROR]")
         self.parser.add_option('--files', action="append", type='string', help="specify files")
         self.parser.add_option('--store_dir', type='string', help='the dir to store gather result, current dir by default.', default='./')
         self.parser.add_option('-c', type='string', help='obdiag custom config', default=os.path.expanduser('~/.obdiag/config.yml'))
+        self.parser.add_option('--since', type='string',help="Specify time range that from 'n' [d]ays, 'n' [h]ours or 'n' [m]inutes. before to now. format: <n> <m|h|d>. example: 1h.",default='30m')
 
     def init(self, cmd, args):
         super(ObdiagAnalyzeLogCommand, self).init(cmd, args)
@@ -711,6 +744,7 @@ def __init__(self):
         self.register_command(ObdiagGatherAwrCommand())
         self.register_command(ObdiagGatherObproxyLogCommand())
         self.register_command(ObdiagGatherSceneCommand())
+        self.register_command(ObdiagGatherAshReportCommand())
 
 
 class ObdiagGatherSceneCommand(MajorCommand):

common/ob_connector.py

@@ -20,7 +20,7 @@
 
 
 class OBConnector(object):
-    def __init__(self, ip, port, username, password=None, database=None, stdio=None, timeout=10,):
+    def __init__(self, ip, port, username, password=None, database=None, stdio=None, timeout=30,):
         self.ip = str(ip)
         self.port = int(port)
         self.username = str(username)
@@ -50,6 +50,18 @@ def _connect_db(self):
             self.stdio.verbose("connect databse ...")
         except mysql.Error as e:
             self.stdio.error("connect OB: {0}:{1} with user {2} failed, error:{3}".format(self.ip, self.port, self.username, e))
+            return
+        try:
+            ob_trx_timeout=self.timeout*1000000
+            self.execute_sql("SET SESSION ob_trx_timeout={0};".format(ob_trx_timeout))
+        except Exception as e:
+            self.stdio.warn("set ob_trx_timeout failed, error:{0}".format(e))
+        try:
+            ob_query_timeout=self.timeout*1000000
+            self.execute_sql("SET SESSION ob_query_timeout={0};".format(ob_query_timeout))
+        except Exception as e:
+            self.stdio.warn("set ob_query_timeout failed, error:{0}".format(e))
+
 
     def execute_sql(self, sql):
         if self.conn is None:
@@ -102,3 +114,13 @@ def execute_sql_pretty(self, sql):
         ret = from_db_cursor(cursor)
         cursor.close()
         return ret
+
+    def callproc(self, procname, args=()):
+        if self.conn is None:
+            self._connect_db()
+        else:
+            self.conn.ping(reconnect=True)
+        cursor = self.conn.cursor()
+        cursor.callproc(procname, args)
+        ret = cursor.fetchall()
+        return ret

common/scene.py

@@ -17,7 +17,8 @@
 """
 from common.ssh import SshHelper
 from common.tool import StringUtils
-from common.command import get_observer_version, get_obproxy_version
+from common.command import get_observer_version, get_obproxy_version, get_observer_version_by_sql
+
 
 def filter_by_version(scene, cluster, stdio=None):
     try:
@@ -59,14 +60,19 @@ def filter_by_version(scene, cluster, stdio=None):
         stdio.exception("filter_by_version Exception : {0}".format(e))
         raise Exception("filter_by_version Exception : {0}".format(e))
 
-def get_version(nodes, type, stdio=None):
+def get_version(nodes, type,cluster, stdio=None):
     try:
         if len(nodes) < 1:
             raise Exception("input nodes is empty, please check your config")
         node = nodes[0]
         ssh = SshHelper(True, node.get("ip"), node.get("ssh_username"), node.get("ssh_password"), node.get("ssh_port"), node.get("ssh_key_file"), node)
+        version = ""
         if type == "observer":
-            version = get_observer_version(True, ssh, nodes[0]["home_path"], stdio)
+            try:
+                version = get_observer_version_by_sql(cluster,stdio)
+            except Exception as e:
+                stdio.warn("get observer version by sql fail, use node ssher to get. Exception:{0}".format(e))
+                version = get_observer_version(True, ssh, nodes[0]["home_path"], stdio)
         elif type == "obproxy":
             version = get_obproxy_version(True, ssh, nodes[0]["home_path"], stdio)
         return version