Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error has been detected by the Java Runtime Environment (SIGSEGV (0xb) at pc=0x00000000000204b6, pid=109, tid=110) #1211

Open
1 task done
conorheffron opened this issue Dec 31, 2024 · 5 comments
Open
1 task done

fatal error has been detected by the Java Runtime Environment (SIGSEGV (0xb) at pc=0x00000000000204b6, pid=109, tid=110) #1211

conorheffron opened this issue Dec 31, 2024 · 5 comments
Labels
bug Something isn't working Waiting on OP

Comments

@conorheffron
Copy link

conorheffron commented Dec 31, 2024 •
edited
Loading

Please provide a brief summary of the bug

After upgrading JDK and testing via docker and k8s, the app runs fine but deployment fails on Google Kubernetes Engine (GKE).
It may be an issue running JDK version 23 on GKE cluster noted below?

See crash log notes below, it suggested to post new issue here.

# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0x00000000000204b6, pid=109, tid=110

# JRE version: OpenJDK Runtime Environment Temurin-23.0.1+11 (23.0.1+11) (build 23.0.1+11)
# Java VM: OpenJDK 64-Bit Server VM Temurin-23.0.1+11 (23.0.1+11, mixed mode, emulated-client, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

# Problematic frame:
# C [libio_grpc_netty_shaded_netty_tcnative_linux_x86_649205427509150964151.so+0x2a154] netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4

# Core dump will be written. Default location: /core.%e.109.%t
# An error report file with more information is saved as:
# /home/gradle/hs_err_pid109.log
INFO 2024-12-31T05:19:36.537827174Z [resource.labels.containerName: ironoc-db-app-sha256-1] [5.328s][warning [os] Loading hsdis library failed

# If you would like to submit a bug report, please visit: https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Did you test with the latest update version?

  • Yes

Please provide steps to reproduce where possible

Run any release version after v5.3.9 (post JDK upgrade) here on GKE cluster.

Expected Results

Continue running without crash

Actual Results

crash SIGSEGV

What Java Version are you using?

  • JRE version: OpenJDK Runtime Environment Temurin-23.0.1+11 (23.0.1+11) (build 23.0.1+11)
  • Java VM: OpenJDK 64-Bit Server VM Temurin-23.0.1+11 (23.0.1+11, mixed mode, emulated-client, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

What is your operating system and platform?

OS: Linux/arm64

GKE / Cluster Version: 1.31.1-gke.2105000

COS Version: cos-117-18613-0-92

Docker base image - gradle:8.11.1-jdk23-alpine
docker hub details for gradle:8.11.1-jdk23-alpine

How did you install Java?

docker image

Did it work before?

Yes with JDK 21.
All versions after work locally via docker and minikube.

Did you test with other Java versions?

Yes with 21 & 23

Relevant log output

DEFAULT 2024-12-31T05:19:33.018744Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/gke-common-webhook-lock] [protoPayload.authenticationInfo.principalEmail: system:gke-common-webhooks] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:gke-common-webhooks"
DEFAULT 2024-12-31T05:19:33.027996Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/maintenance-controller] [protoPayload.authenticationInfo.principalEmail: system:maintenance-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:maintenance-controller"
INFO 2024-12-31T05:19:33.036354189Z [resource.labels.containerName: ironoc-db-app-sha256-1] ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
INFO 2024-12-31T05:19:33.039264946Z [resource.labels.containerName: ironoc-db-app-sha256-1] \\/ ___)| |_)| | | | | || (_| | ) ) ) )
INFO 2024-12-31T05:19:33.040210115Z [resource.labels.containerName: ironoc-db-app-sha256-1] ' |____| .__|_| |_|_| |_\__, | / / / /
INFO 2024-12-31T05:19:33.042541933Z [resource.labels.containerName: ironoc-db-app-sha256-1] =========|_|==============|___/=/_/_/_/
INFO 2024-12-31T05:19:33.043364018Z [resource.labels.containerName: ironoc-db-app-sha256-1] {}
INFO 2024-12-31T05:19:33.044046485Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[32m :: Spring Boot :: �[39m �[2m (v3.4.1)�[0;39m
INFO 2024-12-31T05:19:33.044220201Z [resource.labels.containerName: ironoc-db-app-sha256-1] {}
DEFAULT 2024-12-31T05:19:33.078211Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/filestore-csi-storage-gke-io] [protoPayload.authenticationInfo.principalEmail: system:filestorecsi-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:filestorecsi-controller"
DEFAULT 2024-12-31T05:19:33.085248Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:cloud-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cloud-controller-manager"
DEFAULT 2024-12-31T05:19:33.115548Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler] [protoPayload.authenticationInfo.principalEmail: system:kube-scheduler] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kube-scheduler"
INFO 2024-12-31T05:19:33.138450722Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:33.105Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mcom.ironoc.db.App �[0;39m �[2m:�[0;39m Starting App using Java 23.0.1 with PID 109 (/home/gradle/build/classes/java/main started by root in /home/gradle)
INFO 2024-12-31T05:19:33.139251082Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:33.113Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mcom.ironoc.db.App �[0;39m �[2m:�[0;39m The following 1 profile is active: "h2"
DEFAULT 2024-12-31T05:19:33.148468Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock] [protoPayload.authenticationInfo.principalEmail: system:anet-operator] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:anet-operator"
DEFAULT 2024-12-31T05:19:33.160718Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/managed-certificate-controller] [protoPayload.authenticationInfo.principalEmail: system:managed-certificate-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:managed-certificate-controller"
DEFAULT 2024-12-31T05:19:33.276853Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/ingress-gce-lock] [protoPayload.authenticationInfo.principalEmail: system:l7-lb-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:l7-lb-controller"
DEFAULT 2024-12-31T05:19:33.289176Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/external-attacher-leader-pd-csi-storage-gke-io] [protoPayload.authenticationInfo.principalEmail: system:pdcsi-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:pdcsi-controller"
DEFAULT 2024-12-31T05:19:33.321091Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-autoscaler] [protoPayload.authenticationInfo.principalEmail: system:cluster-autoscaler] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cluster-autoscaler"
DEFAULT 2024-12-31T05:19:33.335880Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/addon-manager] [protoPayload.authenticationInfo.principalEmail: system:addon-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:addon-manager"
DEFAULT 2024-12-31T05:19:33.347238Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.core.v1.configmaps.update] [protoPayload.resourceName: core/v1/namespaces/kube-system/configmaps/cluster-kubestore] [protoPayload.authenticationInfo.principalEmail: system:kubestore-collector] audit_log, method: "io.k8s.core.v1.configmaps.update", principal_email: "system:kubestore-collector"
DEFAULT 2024-12-31T05:19:33.357698Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.core.v1.configmaps.update] [protoPayload.resourceName: core/v1/namespaces/kube-system/configmaps/cluster-autoscaler-status] [protoPayload.authenticationInfo.principalEmail: system:cluster-autoscaler] audit_log, method: "io.k8s.core.v1.configmaps.update", principal_email: "system:cluster-autoscaler"
DEFAULT 2024-12-31T05:19:33.360871360Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] {"status":{…}}
DEFAULT 2024-12-31T05:19:33.360937Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-kubestore] [protoPayload.authenticationInfo.principalEmail: system:kubestore-collector] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kubestore-collector"
INFO 2024-12-31T05:19:33.464421173Z [resource.labels.containerName: gcp-filestore-driver] lock is held by gk3-ironoc-db-app-deploy-nap-10ucqffs-13a5c811-8aoj_ab5cff09-27a6-4b7a-802a-b62a9e95709b and has not yet expired
INFO 2024-12-31T05:19:33.466203949Z [resource.labels.containerName: gcp-filestore-driver] failed to acquire lease gke-managed-filestorecsi/filestore-csi-storage-gke-io-node
DEFAULT 2024-12-31T05:19:33.733267Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/gke-managed-filestorecsi/leases/filestore-csi-storage-gke-io-node] [protoPayload.authenticationInfo.principalEmail: system:serviceaccount:kube-system:filestorecsi-node-sa] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:serviceaccount:kube-system:filestorecsi-node-sa"
INFO 2024-12-31T05:19:33.734663872Z [resource.labels.containerName: gcp-filestore-driver] successfully renewed lease gke-managed-filestorecsi/filestore-csi-storage-gke-io-node
DEFAULT 2024-12-31T05:19:34.002737Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/gcp-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:gcp-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:gcp-controller-manager"
DEFAULT 2024-12-31T05:19:34.077092Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/service-steering.networking.gke.io] [protoPayload.authenticationInfo.principalEmail: system:service-steering] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:service-steering"
DEFAULT 2024-12-31T05:19:34.094856Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/ingress-gce-neg-lock] [protoPayload.authenticationInfo.principalEmail: system:l7-lb-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:l7-lb-controller"
DEFAULT 2024-12-31T05:19:34.127136Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-filestore-csi-storage-gke-io] [protoPayload.authenticationInfo.principalEmail: system:filestorecsi-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:filestorecsi-controller"
DEFAULT 2024-12-31T05:19:34.335518Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:kube-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kube-controller-manager"
DEFAULT 2024-12-31T05:19:34.337912Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-provider-extraction-migration-pt2] [protoPayload.authenticationInfo.principalEmail: system:cloud-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cloud-controller-manager"
DEFAULT 2024-12-31T05:19:34.406346Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/addon-resizer] [protoPayload.authenticationInfo.principalEmail: system:metrics-server-nanny] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:metrics-server-nanny"
DEFAULT 2024-12-31T05:19:34.580577Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-filestore-csi-storage-gke-io] [protoPayload.authenticationInfo.principalEmail: system:filestorecsi-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:filestorecsi-controller"
INFO 2024-12-31T05:19:34.635700872Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:34.581Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36m.s.d.r.c.RepositoryConfigurationDelegate�[0;39m �[2m:�[0;39m Bootstrapping Spring Data JPA repositories in DEFAULT mode.
INFO 2024-12-31T05:19:34.736468006Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:34.656Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36m.s.d.r.c.RepositoryConfigurationDelegate�[0;39m �[2m:�[0;39m Finished Spring Data repository scanning in 61 ms. Found 1 JPA repository interface.
DEFAULT 2024-12-31T05:19:34.796399Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/gke-gateway-leader-election] [protoPayload.authenticationInfo.principalEmail: service-902038140834@container-engine-robot.iam.gserviceaccount.com] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "service-902038140834@container-engine-robot.iam.gserviceaccount.com"
DEFAULT 2024-12-31T05:19:34.801025Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.core.v1.configmaps.update] [protoPayload.resourceName: core/v1/namespaces/kube-system/configmaps/clouddns-lock] [protoPayload.authenticationInfo.principalEmail: system:clouddns] audit_log, method: "io.k8s.core.v1.configmaps.update", principal_email: "system:clouddns"
DEFAULT 2024-12-31T05:19:34.809697Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/clouddns-lock] [protoPayload.authenticationInfo.principalEmail: system:clouddns] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:clouddns"
DEFAULT 2024-12-31T05:19:34.836854Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.core.v1.configmaps.update] [protoPayload.resourceName: core/v1/namespaces/kube-system/configmaps/clustermetrics] [protoPayload.authenticationInfo.principalEmail: system:clustermetrics] audit_log, method: "io.k8s.core.v1.configmaps.update", principal_email: "system:clustermetrics"
DEFAULT 2024-12-31T05:19:34.845408Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/clustermetrics] [protoPayload.authenticationInfo.principalEmail: system:clustermetrics] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:clustermetrics"
DEFAULT 2024-12-31T05:19:34.999124Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/vpa-updater] [protoPayload.authenticationInfo.principalEmail: system:vpa-updater] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:vpa-updater"
DEFAULT 2024-12-31T05:19:35.027023Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/gke-common-webhook-lock] [protoPayload.authenticationInfo.principalEmail: system:gke-common-webhooks] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:gke-common-webhooks"
DEFAULT 2024-12-31T05:19:35.033668Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/vpa-recommender] [protoPayload.authenticationInfo.principalEmail: system:vpa-recommender] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:vpa-recommender"
DEFAULT 2024-12-31T05:19:35.041504Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/maintenance-controller] [protoPayload.authenticationInfo.principalEmail: system:maintenance-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:maintenance-controller"
INFO 2024-12-31T05:19:35.063949353Z [resource.labels.containerName: ironoc-db-app-sha256-1] 2024-12-31T05:19:35.045Z INFO 109 --- [o-8080-exec-360] c.ironoc.db.controller.PersonController : Entering personController.home: map={}
INFO 2024-12-31T05:19:35.064038Z [resource.labels.containerName: ironoc-db-app-sha256-1] Hibernate: select p1_0.id,p1_0.age,p1_0.first_name,p1_0.surname,p1_0.title from person p1_0
DEFAULT 2024-12-31T05:19:35.091970Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:cloud-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cloud-controller-manager"
DEFAULT 2024-12-31T05:19:35.123300Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler] [protoPayload.authenticationInfo.principalEmail: system:kube-scheduler] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kube-scheduler"
DEFAULT 2024-12-31T05:19:35.160722Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock] [protoPayload.authenticationInfo.principalEmail: system:anet-operator] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:anet-operator"
DEFAULT 2024-12-31T05:19:35.170414Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/managed-certificate-controller] [protoPayload.authenticationInfo.principalEmail: system:managed-certificate-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:managed-certificate-controller"
DEFAULT 2024-12-31T05:19:35.284957Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/ingress-gce-lock] [protoPayload.authenticationInfo.principalEmail: system:l7-lb-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:l7-lb-controller"
DEFAULT 2024-12-31T05:19:35.328593Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-autoscaler] [protoPayload.authenticationInfo.principalEmail: system:cluster-autoscaler] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cluster-autoscaler"
DEFAULT 2024-12-31T05:19:35.343620Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/addon-manager] [protoPayload.authenticationInfo.principalEmail: system:addon-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:addon-manager"
DEFAULT 2024-12-31T05:19:35.383031Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.core.v1.configmaps.update] [protoPayload.resourceName: core/v1/namespaces/kube-system/configmaps/cluster-kubestore] [protoPayload.authenticationInfo.principalEmail: system:kubestore-collector] audit_log, method: "io.k8s.core.v1.configmaps.update", principal_email: "system:kubestore-collector"
DEFAULT 2024-12-31T05:19:35.395059Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-kubestore] [protoPayload.authenticationInfo.principalEmail: system:kubestore-collector] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kubestore-collector"
DEFAULT 2024-12-31T05:19:35.456883Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader] [protoPayload.authenticationInfo.principalEmail: system:snapshot-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:snapshot-controller"
INFO 2024-12-31T05:19:35.535633212Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.492Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mo.s.b.w.embedded.tomcat.TomcatWebServer �[0;39m �[2m:�[0;39m Tomcat initialized with port 8080 (http)
INFO 2024-12-31T05:19:35.536141847Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.512Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mo.apache.catalina.core.StandardService �[0;39m �[2m:�[0;39m Starting service [Tomcat]
INFO 2024-12-31T05:19:35.536545134Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.513Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mo.apache.catalina.core.StandardEngine �[0;39m �[2m:�[0;39m Starting Servlet engine: [Apache Tomcat/11.0.2]
DEFAULT 2024-12-31T05:19:35.918112Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/external-snapshotter-leader-pd-csi-storage-gke-io] [protoPayload.authenticationInfo.principalEmail: system:pdcsi-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:pdcsi-controller"
INFO 2024-12-31T05:19:35.935882054Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.865Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36morg.apache.jasper.servlet.TldScanner �[0;39m �[2m:�[0;39m At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
INFO 2024-12-31T05:19:35.935930661Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.877Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mo.a.c.c.C.[Tomcat].[localhost].[/] �[0;39m �[2m:�[0;39m Initializing Spring embedded WebApplicationContext
INFO 2024-12-31T05:19:35.936881930Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.879Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mw.s.c.ServletWebServerApplicationContext�[0;39m �[2m:�[0;39m Root WebApplicationContext: initialization completed in 2643 ms
DEFAULT 2024-12-31T05:19:35.947796Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/apiserver-suqqm3c42ckyy7rytf3s2nbadi] [protoPayload.authenticationInfo.principalEmail: system:apiserver] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:apiserver"
DEFAULT 2024-12-31T05:19:36.012981Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/gcp-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:gcp-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:gcp-controller-manager"
INFO 2024-12-31T05:19:36.035712435Z [resource.labels.containerName: ironoc-db-app-sha256-1] �[2m2024-12-31T05:19:35.951Z�[0;39m �[32m INFO�[0;39m �[35m109�[0;39m �[2m--- [ main] �[0;39m�[36mc.i.db.service.GoogleCloudClientImpl �[0;39m �[2m:�[0;39m Entering GoogleCloudClient.getSecret for secretVersion=projects/902038140834/secrets/MY_SQL_PASSWORD/versions/1
DEFAULT 2024-12-31T05:19:36.087022Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/service-steering.networking.gke.io] [protoPayload.authenticationInfo.principalEmail: system:service-steering] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:service-steering"
DEFAULT 2024-12-31T05:19:36.103613Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/ingress-gce-neg-lock] [protoPayload.authenticationInfo.principalEmail: system:l7-lb-controller] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:l7-lb-controller"
INFO 2024-12-31T05:19:36.217416752Z [resource.labels.containerName: gke-metadata-server] [conn-id:38ea2b168acd800b rpc-id:deb8196934a94b55 remote-addr:10.60.0.11:36276 pod:ironoc-db-ns/ironoc-db-app-deployment-6ff75565df-7wf9b] "/" HTTP/200, started at 2024-12-31 05:19:36.21554249 +0000 UTC m=+2690776.322486292
INFO 2024-12-31T05:19:36.275138Z [protoPayload.serviceName: cloudbuild.googleapis.com] [protoPayload.methodName: google.devtools.cloudbuild.v1.CloudBuild.GetBuild] [protoPayload.resourceName: projects/ironoc-db/builds/55df5166-4c61-44cf-8b54-bc95200301f9] [protoPayload.authenticationInfo.principalEmail: [email protected]] audit_log, method: "google.devtools.cloudbuild.v1.CloudBuild.GetBuild", principal_email: "[email protected]"
DEFAULT 2024-12-31T05:19:36.341874Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager] [protoPayload.authenticationInfo.principalEmail: system:kube-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:kube-controller-manager"
DEFAULT 2024-12-31T05:19:36.344747Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-provider-extraction-migration-pt2] [protoPayload.authenticationInfo.principalEmail: system:cloud-controller-manager] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:cloud-controller-manager"
DEFAULT 2024-12-31T05:19:36.417566Z [resource.labels.clusterName: ironoc-db-app-deployment-cluster] [protoPayload.serviceName: k8s.io] [protoPayload.methodName: io.k8s.coordination.v1.leases.update] [protoPayload.resourceName: coordination.k8s.io/v1/namespaces/kube-system/leases/addon-resizer] [protoPayload.authenticationInfo.principalEmail: system:metrics-server-nanny] audit_log, method: "io.k8s.coordination.v1.leases.update", principal_email: "system:metrics-server-nanny"
INFO 2024-12-31T05:19:36.535465930Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.535661634Z [resource.labels.containerName: ironoc-db-app-sha256-1] # A fatal error has been detected by the Java Runtime Environment:
INFO 2024-12-31T05:19:36.535670898Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.535691718Z [resource.labels.containerName: ironoc-db-app-sha256-1] # SIGSEGV (0xb) at pc=0x00000000000204b6, pid=109, tid=110
INFO 2024-12-31T05:19:36.535760464Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.537003009Z [resource.labels.containerName: ironoc-db-app-sha256-1] # JRE version: OpenJDK Runtime Environment Temurin-23.0.1+11 (23.0.1+11) (build 23.0.1+11)
INFO 2024-12-31T05:19:36.537344943Z [resource.labels.containerName: ironoc-db-app-sha256-1] # Java VM: OpenJDK 64-Bit Server VM Temurin-23.0.1+11 (23.0.1+11, mixed mode, emulated-client, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
INFO 2024-12-31T05:19:36.537426046Z [resource.labels.containerName: ironoc-db-app-sha256-1] # Problematic frame:
INFO 2024-12-31T05:19:36.537486721Z [resource.labels.containerName: ironoc-db-app-sha256-1] # C [libio_grpc_netty_shaded_netty_tcnative_linux_x86_649205427509150964151.so+0x2a154] netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4
INFO 2024-12-31T05:19:36.537494722Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.537676927Z [resource.labels.containerName: ironoc-db-app-sha256-1] # Core dump will be written. Default location: /core.%e.109.%t
INFO 2024-12-31T05:19:36.537686474Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.537810228Z [resource.labels.containerName: ironoc-db-app-sha256-1] # An error report file with more information is saved as:
INFO 2024-12-31T05:19:36.537821034Z [resource.labels.containerName: ironoc-db-app-sha256-1] # /home/gradle/hs_err_pid109.log
INFO 2024-12-31T05:19:36.537827174Z [resource.labels.containerName: ironoc-db-app-sha256-1] [5.328s][warning][os] Loading hsdis library failed
INFO 2024-12-31T05:19:36.635747455Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
INFO 2024-12-31T05:19:36.636293821Z [resource.labels.containerName: ironoc-db-app-sha256-1] # If you would like to submit a bug report, please visit:
INFO 2024-12-31T05:19:36.636768895Z [resource.labels.containerName: ironoc-db-app-sha256-1] # https://github.com/adoptium/adoptium-support/issues
INFO 2024-12-31T05:19:36.637246615Z [resource.labels.containerName: ironoc-db-app-sha256-1] # The crash happened outside the Java Virtual Machine in native code.
INFO 2024-12-31T05:19:36.637644404Z [resource.labels.containerName: ironoc-db-app-sha256-1] # See problematic frame for where to report the bug.
INFO 2024-12-31T05:19:36.638110052Z [resource.labels.containerName: ironoc-db-app-sha256-1] #
@conorheffron conorheffron added the bug Something isn't working label Dec 31, 2024
@conorheffron
Copy link
Author

See logs attached:

downloaded-logs-20241231-071443.csv

@conorheffron conorheffron mentioned this issue Dec 31, 2024
Open
@karianna
Copy link
Contributor

karianna commented Jan 1, 2025

@conorheffron What version of netty are you using? It looks like a native call failing and propagating back to Java.

@conorheffron
Copy link
Author

conorheffron commented Jan 2, 2025 •
edited
Loading

Hi @karianna, Thanks for response.
I am not using netty directly, However, Google Cloud secret manager (com.google.cloud:google-cloud-secretmanager:2.55.0) dependency pulls in Netty dependency (io.grpc:grpc-netty-shaded:1.69.0)

The main changes from the last stable GKE build are below.

  • JDK 21 -> 23
  • Spring Boot Version 3.4 -> 3.4.1
  • Google Cloud Secret Manager 2.49 -> 2.55
  • Tomcat Embed Jasper 10 -> 11

These changes were tested locally via Docker & Minikube without issue.
The unit & integration tests passed on CI build via GitHub workflows (gradle clean build etc.).

Pasted Graphic

Pasted Graphic 2


In addition, please see the latest dependency tree for the app below (command: gradle :dependencies):

dependencies.txt



@jarledb
Copy link

jarledb commented Jan 2, 2025

We are seeing the same issue, but with the following version
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)

Downgrading io.grpc.grpc-netty-shaded from 1.69.0 back to 1.66.0 solved the issue for now (1.67 and 1.68 did not work either)

@conorheffron
Copy link
Author

conorheffron commented Jan 2, 2025 •
edited
Loading

Yes, I have completed rollback to release with Google Cloud Secret Manager version 2.49 for now but I do not think this is the solution.

I found similar issue here but this was from 2022.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Waiting on OP
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@karianna @jarledb @conorheffron