Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FAQ] [k8s native] dinky k8s cluster ip Submit a problem #3866

Closed
2 of 3 tasks
jianjun159 opened this issue Oct 11, 2024 · 2 comments
Closed
2 of 3 tasks

[FAQ] [k8s native] dinky k8s cluster ip Submit a problem #3866

jianjun159 opened this issue Oct 11, 2024 · 2 comments
Assignees
@gaoyan1998 @zackyoungh @Zzm0809
Labels
Bug Something isn't working Waiting for reply Waiting for reply

Comments

@jianjun159
Copy link
Contributor

jianjun159 commented Oct 11, 2024 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

When I used k8s application to submit tasks in dinky, when the exposed port mode was cluster ip mode, the flink task ran normally in k8s, but I could not get the status of the flink task，logs :
Start Process:FlinkSubmit/2
Start Process Step:SUBMIT_PRECHECK
2024-10-11 11:43:16.599 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:188 - Start check and config task, task:dinky-flink-sql-test

Process Step SUBMIT_PRECHECK exit with status:FINISHED
Start Process Step:SUBMIT_EXECUTE
Start Process Step:SUBMIT_BUILD_CONFIG
2024-10-11 11:43:16.603 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:300 - Start initialize FlinkSQLEnv:

2024-10-11 11:43:16.604 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:320 - Initializing data permissions...

2024-10-11 11:43:16.613 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:322 - Finish initialize FlinkSQLEnv.

2024-10-11 11:43:16.619 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:237 - Init gateway config, type:kubernetes-application

2024-10-11 11:43:16.625 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:250 - Init remote cluster

Process Step SUBMIT_BUILD_CONFIG exit with status:FINISHED
2024-10-11 11:43:16.646 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:67 - Initializing Flink job config...

2024-10-11 11:43:16.660 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:125 - A total of 0 UDF have been Init.

2024-10-11 11:43:16.660 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:126 - Initializing Flink UDF...Finish

2024-10-11 11:43:16.661 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:58 - Simple authentication mode

2024-10-11 11:43:16.672 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:58 - Simple authentication mode

2024-10-11 11:43:16.677 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:93 - load locale config yaml failed：Given configuration directory is null, cannot load configuration,Skip config it

2024-10-11 11:43:16.744 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:99 - Could not find main container flink-main-container in pod template, using empty one to initialize.

2024-10-11 11:43:16.755 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:83 - Start submit k8s application.

2024-10-11 11:43:16.825 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:119 - Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124

2024-10-11 11:43:16.826 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:119 - Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122

2024-10-11 11:43:16.827 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:865 - Config uses fallback configuration key 'kubernetes.pod-template-file.default' instead of key 'kubernetes.pod-template-file.jobmanager'

2024-10-11 11:43:16.831 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:865 - Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.jobmanager.service-account'

2024-10-11 11:43:16.834 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:860 - Config uses deprecated configuration key 'kubernetes.container.image' instead of proper key 'kubernetes.container.image.ref'

2024-10-11 11:43:16.843 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:865 - Config uses fallback configuration key 'kubernetes.pod-template-file.default' instead of key 'kubernetes.pod-template-file.taskmanager'

2024-10-11 11:43:16.846 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:865 - Config uses fallback configuration key 'kubernetes.pod-template-file.default' instead of key 'kubernetes.pod-template-file.taskmanager'

2024-10-11 11:43:17.026 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:112 - Ingress dinky-flink-sql-test does not exist

2024-10-11 11:43:17.027 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:66 - getRestEndpoint_Ingress_Try: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:17.028 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:88 - getRestEndpoint_Ingress_Try_Not_Found: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:17.029 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:130 - Please note that Flink client operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been set to ClusterIP.

2024-10-11 11:43:17.030 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:17.031 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:203 - Create flink application cluster dinky-flink-sql-test successfully, JobManager Web Interface: http://dinky-flink-sql-test-rest.dinky:8081

2024-10-11 11:43:17.035 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:115 - createDinkyResource

2024-10-11 11:43:17.104 INFO <lambda$createDinkyResource$1> <> <760329a4510f40af9d042eb8ff25e6e2> LINE:136 - ---
apiVersion: "v1"
kind: "ConfigMap"
metadata:
labels:
app: "dinky-flink-sql-test"
type: "flink-native-kubernetes"
name: "dinky-config-dinky-flink-sql-test"
namespace: "dinky"
ownerReferences:

  • apiVersion: "apps/v1"
    kind: "Deployment"
    blockOwnerDeletion: true
    controller: true
    name: "dinky-flink-sql-test"
    uid: "8b072218-6869-4301-a677-898c5d8fa886"
    data:
    job.sql: "CREATE TABLE datagen (\n f_sequence INT,\n f_random INT,\n f_random_str
    \ STRING,\n ts AS localtimestamp,\n WATERMARK FOR ts AS ts\n ) WITH (\n
    \ 'connector' = 'datagen',\n 'rows-per-second'='5',\n 'fields.f_sequence.kind'='sequence',\n
    \ 'fields.f_sequence.start'='1',\n 'fields.f_sequence.end'='50000',\n
    \ 'fields.f_random.min'='1',\n 'fields.f_random.max'='50000',\n 'fields.f_random_str.length'='10'\n
    \ );\nCREATE TABLE print_table (\n f_sequence INT,\n f_random INT,\n
    \ f_random_str STRING\n ) WITH (\n 'connector' = 'print'\n );\nINSERT
    \ INTO print_table select f_sequence,f_random,f_random_str from datagen;\n"

2024-10-11 11:43:17.129 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:126 - Got Flink Container State:
Pod: dinky-flink-sql-test-54b8bbc7fd-9jpxr, Ready: false, restartCount: 0, image: xxxxxx/bigdata/oversea-dinky:v1.1
------CurrentState------
waiting:
reason: ContainerCreating

------LastState------
{}

2024-10-11 11:43:17.130 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:181 - Kubernetes Pod have not ready, reTry at 5 sec later

2024-10-11 11:43:22.141 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:126 - Got Flink Container State:
Pod: dinky-flink-sql-test-54b8bbc7fd-9jpxr, Ready: true, restartCount: 0, image: xxxxxx/bigdata/oversea-dinky:v1.1
------CurrentState------
running:
startedAt: '2024-10-11T03:43:27Z'

------LastState------
{}

2024-10-11 11:43:22.163 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:112 - Ingress dinky-flink-sql-test does not exist

2024-10-11 11:43:22.164 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:66 - getRestEndpoint_Ingress_Try: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:22.164 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:88 - getRestEndpoint_Ingress_Try_Not_Found: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:22.166 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:130 - Please note that Flink client operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been set to ClusterIP.

2024-10-11 11:43:22.167 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:185 - Start get job list ....

2024-10-11 11:43:22.168 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:22.169 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:25.176 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:25.177 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:28.180 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:28.181 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:31.184 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:31.185 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:34.188 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:34.189 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:37.194 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:37.195 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:37.198 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:204 - Get job status failed,null

2024-10-11 11:43:42.209 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:126 - Got Flink Container State:
Pod: dinky-flink-sql-test-54b8bbc7fd-9jpxr, Ready: true, restartCount: 0, image: xxxxx/bigdata/oversea-dinky:v1.1
------CurrentState------
running:
startedAt: '2024-10-11T03:43:27Z'

------LastState------
{}

2024-10-11 11:43:42.233 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:112 - Ingress dinky-flink-sql-test does not exist

2024-10-11 11:43:42.234 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:66 - getRestEndpoint_Ingress_Try: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:42.234 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:88 - getRestEndpoint_Ingress_Try_Not_Found: [dinky] -> dinky-flink-sql-test

2024-10-11 11:43:42.236 WARN <> <760329a4510f40af9d042eb8ff25e6e2> LINE:130 - Please note that Flink client operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been set to ClusterIP.

2024-10-11 11:43:42.237 INFO <> <760329a4510f40af9d042eb8ff25e6e2> LINE:185 - Start get job list ....

2024-10-11 11:43:42.238 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:42.239 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:45.244 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:45.245 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:48.250 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:48.252 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:51.255 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:51.256 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

2024-10-11 11:43:54.259 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:1058 - getWebMonitorBaseUrl_direct: false

2024-10-11 11:43:54.260 ERROR <> <760329a4510f40af9d042eb8ff25e6e2> LINE:508 - Sending request of class class org.apache.flink.runtime.rest.messages.EmptyRequestBody to dinky-flink-sql-test-rest.dinky:8081/v1/jobs/overview

image
image

What you expected to happen

How do I get the program to work

How to reproduce

k8s use cluster ip

Anything else

No response

Version

1.1.0

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@jianjun159 jianjun159 added Bug Something isn't working Waiting for reply Waiting for reply labels Oct 11, 2024
@github-actions GitHub Actions
Copy link

Hello @jianjun159, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @jianjun159, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

@github-actions GitHub Actions
Copy link

Hello @jianjun159, this issue is about K8S, so I assign it to @gaoyan1998 and @zackyoungh. If you have any questions, you can comment and reply.

你好 @jianjun159, 这个 issue 是关于 K8S 的，所以我把它分配给了 @gaoyan1998 和 @zackyoungh。如有任何问题，可以评论回复。

@jianjun159 jianjun159 reopened this Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gaoyan1998 gaoyan1998

@zackyoungh zackyoungh

@Zzm0809 Zzm0809

Labels
Bug Something isn't working Waiting for reply Waiting for reply
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gaoyan1998 @zackyoungh @Zzm0809 @jianjun159