安装服务时md5验证失败，压缩包被修改损坏 #284

tangpf877 · 2023-06-21T05:34:55Z

tangpf877
Jun 21, 2023

我有四台服务器ddp1-4，所有的服务的安装包都放在ddp1的/opt/datasophon/DDP/packages目录下，当我安装任何服务时，如果选择安装在ddp1上，则会在ddp1上的worker的日志中出现如下信息(以安装spark为例)：

[INFO] 2023-06-15 11:32:10 com.datasophon.worker.actor.InstallServiceActor:[45] - start install package spark-3.1.3.tar.gz
[INFO] 2023-06-15 11:32:10 com.datasophon.worker.handler.InstallServiceHandler:[68] - Remote package md5 is 92553715ec374cd670c57ef0fa53b6ee
[INFO] 2023-06-15 11:32:10 com.datasophon.worker.handler.InstallServiceHandler:[74] - Local md5 is null
[INFO] 2023-06-15 11:32:10 com.datasophon.worker.handler.InstallServiceHandler:[89] - download url is http://ddp1:8081/ddh/service/install/downloadPackage?packageName=spark-3.1.3.tar.gz
[INFO] 2023-06-15 11:32:10 com.datasophon.worker.handler.InstallServiceHandler:[108] - download package spark-3.1.3.tar.gz success
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.CompressUtils:[35] - use tar -zxvf to decompress
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/pyspark.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-submit
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-submit.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-class2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-shell2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/pyspark2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/docker-image-tool.sh
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/run-example.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-submit2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/beeline.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/beeline
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-shell
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/find-spark-home
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/sparkR2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/find-spark-home.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/sparkR
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-class
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-sql2.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/load-spark-env.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/run-example
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-sql
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-class.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-sql.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/spark-shell.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/sparkR.cmd
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/load-spark-env.sh
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/bin/pyspark
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/README.md
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/DESCRIPTION
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/tests/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/tests/testthat/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/tests/testthat/test_basic.R
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/R/
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/R/SparkR
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/R/SparkR.rdx
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - spark-3.1.3/R/lib/SparkR/R/SparkR.rdb
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] -
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - gzip: stdin: unexpected end of file
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - tar: Unexpected EOF in archive
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - tar: Unexpected EOF in archive
[INFO] 2023-06-15 11:32:10 com.datasophon.common.utils.ShellUtils:[169] - tar: Error is not recoverable: exiting now
[WARN] 2023-06-15 11:32:10 com.datasophon.worker.handler.InstallServiceHandler:[132] - install package spark-3.1.3.tar.gz failed
[INFO] 2023-06-15 11:32:10 com.datasophon.worker.actor.InstallServiceActor:[71] - install package failed

并且会发现/opt/datasophon/DDP/packages目录下的spark-3.1.3.tar.gz文件会被修改。
我的操作系统版本为CentOS Linux release 7.6.1810，datasophon版本为1.1.1

如果我将服务安装在另外三台服务器上(ddp2, ddp3, dd4)，则不会出现这种问题

datasophon · 2023-06-21T06:35:23Z

datasophon
Jun 21, 2023
Maintainer

问题主要是MD5校验失败，这个可以提个issue，功能上可以优化一下，在worker与manager在同一台机器上的时候，不需要再下载安装包

1 reply

tangpf877 Jun 21, 2023
Author

好的，多谢回复

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

安装服务时md5验证失败，压缩包被修改损坏 #284

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

安装服务时md5验证失败，压缩包被修改损坏 #284

tangpf877 Jun 21, 2023

Replies: 1 comment · 1 reply

datasophon Jun 21, 2023 Maintainer

tangpf877 Jun 21, 2023 Author

tangpf877
Jun 21, 2023

Replies: 1 comment 1 reply

datasophon
Jun 21, 2023
Maintainer

tangpf877 Jun 21, 2023
Author