Skip to content
This repository has been archived by the owner on Feb 14, 2023. It is now read-only.

Failing to push applications on cf-for-k8s v5.4.3 #695

Open
aveiga opened this issue Feb 22, 2022 · 1 comment
Open

Failing to push applications on cf-for-k8s v5.4.3 #695

aveiga opened this issue Feb 22, 2022 · 1 comment
Labels
unscheduled

Comments

@aveiga
Copy link

aveiga commented Feb 22, 2022

Describe the bug

We're seeing multiple, random failures when deploying applications (via cf cli v7) on one of our cf-for-k8s clusters. The cf push command fails rather quickly (around 4 seconds). This is what we can gather from our logs:

{
    "error_code"=>"UnknownError",
    "description"=>"An unknown error occurred.",
    "code"=>10001,
    "test_mode_info"=>{
        "description"=>"undefined method `spec' for nil:NilClass",
        "error_code"=>"CF-spec",
        "backtrace"=>[
            "/workspace/lib/kubernetes/route_resource_manager.rb: 51:in `block in update_destinations'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 18:in `block in apply_route_update'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 46:in `block in retry_on_conflict'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/retryable-3.0.5/lib/retryable.rb: 72:in `retryable'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 44:in `retry_on_conflict'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 17:in `apply_route_update'",
            "/workspace/lib/kubernetes/route_resource_manager.rb: 50:in `update_destinations'",
            "/workspace/app/actions/update_route_destinations.rb: 106:in `update'",
            "/workspace/app/actions/update_route_destinations.rb: 27:in `add'",
            "/workspace/app/actions/manifest_route_update.rb: 39:in `block in update'",
            "/workspace/app/actions/manifest_route_update.rb: 36:in `each'",
            "/workspace/app/actions/manifest_route_update.rb: 36:in `update'",
            "/workspace/app/actions/app_apply_manifest.rb: 98:in `update_routes'",
            "/workspace/app/actions/app_apply_manifest.rb: 63:in `apply'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 16:in `block in perform'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 15:in `each'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 15:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/timeout_job.rb: 13:in `block in perform'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 95:in `block in timeout'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `block in catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 110:in `timeout'",
            "/workspace/app/jobs/timeout_job.rb: 12:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/logging_context_job.rb: 17:in `block in perform'",
            "/workspace/app/jobs/logging_context_job.rb: 67:in `with_request_id_set'",
            "/workspace/app/jobs/logging_context_job.rb: 15:in `perform'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb: 81:in `block in invoke_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb: 78:in `invoke_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 230:in `block (2 levels) in run'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 95:in `block in timeout'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 105:in `timeout'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 230:in `block in run'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb: 308:in `realtime'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 229:in `run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 312:in `block in reserve_and_run_one_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 312:in `reserve_and_run_one_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 213:in `block in work_off'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 212:in `times'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 212:in `work_off'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 175:in `block (4 levels) in start'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb: 308:in `realtime'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 174:in `block (3 levels) in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 173:in `block (2 levels) in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 172:in `loop'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 172:in `block in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/plugins/clear_locks.rb: 7:in `block (2 levels) in <class:ClearLocks>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 79:in `block (2 levels) in add'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 79:in `block in add'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 171:in `start'",
            "/workspace/lib/tasks/jobs.rake: 76:in `start_working'",
            "/workspace/lib/tasks/jobs.rake: 49:in `block (2 levels) in <top (required)>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `block in execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `each'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/honeycomb-beeline-2.7.0/lib/honeycomb/integrations/rake.rb: 14:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 219:in `block in invoke_with_call_chain'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 199:in `synchronize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 199:in `invoke_with_call_chain'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 188:in `invoke'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 160:in `invoke_task'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `block (2 levels) in top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `each'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `block in top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 125:in `run_with_threads'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 110:in `top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 83:in `block in run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 186:in `standard_exception_handling'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 80:in `run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/exe/rake: 27:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake: 23:in `load'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake: 23:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 58:in `load'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 58:in `kernel_load'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 23:in `run'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 477:in `exec'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/command.rb: 27:in `run'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/invocation.rb: 127:in `invoke_command'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor.rb: 392:in `dispatch'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 31:in `dispatch'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/base.rb: 485:in `start'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 25:in `start'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle: 49:in `block in <top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/friendly_errors.rb: 128:in `with_friendly_errors'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle: 37:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle: 23:in `load'",
            "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle: 23:in `<main>'"
        ]
    }
}

{
    "reconciler group": "networking.cloudfoundry.org",
    "reconciler kind": "Route",
    "name": "0fbcc8c6-4e56-46e8-b3ab-64976a92dd71",
    "namespace": "cf-workloads",
    "error": "Operation cannot be fulfilled on routes.networking.cloudfoundry.org \"0fbcc8c6-4e56-46e8-b3ab-64976a92dd71\": StorageError: invalid object, Code: 4, Key: /registry/networking.cloudfoundry.org/routes/cf-workloads/0fbcc8c6-4e56-46e8-b3ab-64976a92dd71, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8df8a849-fa61-43f7-8aa7-bfc72561cbb8, UID in object meta: "
}

To Reproduce

Steps to reproduce the behavior:

  1. On a cluster with 7 CF spaces, each populated with ~50 Spring Boot applications, push multiple applications in sequence

Expected behavior

All applications are successfully pushed

Additional context

cf-for-k8s SHA

v5.4.3

Cluster information

On premises, running cf-for-k8s on top of a Rancher Labs' RKE2 cluster running on VMWare vSphere.

CLI versions

paste output of the following commands

  1. ytt --version: ytt version 0.31.0
  2. kapp --version: kapp version 0.35.0
  3. kubectl version:
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7+rke2r2", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-26T04:47:27Z", GoVersion:"go1.15.8b5", Compiler:"gc", Platform:"linux/amd64"}
  1. cf version: cf version 7.2.0+be4a5ce2b.2020-12-10
@cf-gitbot
Copy link
Collaborator

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/181345931

The labels on this github issue will be updated when the story is started.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
unscheduled
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aveiga @cf-gitbot