Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev 1.1.15 webank streamis fink load yaml #303

Merged
merged 21 commits into from
Oct 11, 2023

Conversation

yangwenzea
Copy link
Collaborator

@yangwenzea yangwenzea commented Oct 8, 2023

What is the purpose of the change

Support param 'env.java.opts' in default flink config

Brief change log

  • Refactor linkis-engingplugin-flink to support param 'env.java.opts'

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
  • If this is a code change: I have written unit tests to fully verify the new behavior.

@yangwenzea yangwenzea changed the base branch from dev-1.1.16-webank to dev-1.1.15-webank October 8, 2023 03:19
@Alexkun Alexkun changed the title Dev 1.1.14 webank streamis fink load yaml Dev 1.1.15 webank streamis fink load yaml Oct 8, 2023
@@ -232,6 +242,96 @@ class FlinkEngineConnFactory extends MultiExecutorEngineConnFactory with Logging
context
}

protected def getExtractJavaOpts(envJavaOpts: String): String = {
Copy link

@casionone casionone Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

调整为private 并添加单测
单测使用org.junit.jupiter.api.Assertions.assertEquals 来断言判断
单测目录应该和src/main 同级 即 src/test

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充必要时代码注解

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

val yamlContent = source.mkString
val yaml = new Yaml()
val configMap = yaml.loadAs(yamlContent, classOf[util.LinkedHashMap[String, Object]])
if (configMap.containsKey("env.java.opts")) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env.java.opts 多个地方使用 定义为一个全局常量

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}


protected def mergeAndDeduplicate(str1: String, str2: String): String = {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

补充必要的注解 必要的地方可以加示例
字符命名优化 str1 和str2 这种不好理解

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

val merged = mergeAndDeduplicate(defaultJavaOpts, envJavaOpts)
}

protected def mergeAndDeduplicate(str1: String, str2: String): String = {
Copy link

@casionone casionone Oct 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里重复定义funtion 的话 没有验证到实际的代码块 且后续修改需要两个地方同步修改,考虑修改为private

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

def testMergeAndDeduplicate: Unit = {
var defaultJavaOpts = "-Da=3 -Db=4 -XXc=5 -Dk=a1=b";
var envJavaOpts = "-DjobName=0607_1 -Dlog4j.configuration=./log4j.properties -Da=1 -Dk=a1=c";
val merged = mergeAndDeduplicate(defaultJavaOpts, envJavaOpts)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结果需要 断言判断

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

val yaml = new Yaml()
val configMap = yaml.loadAs(yamlContent, classOf[util.LinkedHashMap[String, Object]])
if (configMap.containsKey("env.java.opts")) {
defaultJavaOpts = configMap.get("env.java.opts").toString

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里也使用 FLINK_ENV_JAVA_OPTS

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@casionone casionone merged commit a6fa3dd into dev-1.1.15-webank Oct 11, 2023
13 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants