Reduce the infra cost for Knative

[chizhg]

Currently the infra cost for running Prow, E2E tests, release pipelines is around $50K per month. There have already been some efforts to reduce the cost. As part of the effort for CNCF infra migration, we want to further reduce the cost as much as possible before fully handing off the infra to the Knative community.

The current infra cost breakdown can be checked from #3003

Below are some tracking issues that could help:

• Disable Logs/Metrics on GKE and Run Preemptible Nodes hack#138
• Use the retry logic in kubetest2 gke deployer to retry on cluster creations #3096
• Skip Prow integration tests for PRs that only updates GitHub Actions hack#153
• Remove the build and unit tests run by Prow, and use the GitHub actions to block on presubmit #2985
• Run kubetest2 with --down arg to release the Boskos project immediately #3117
• Update heartbeat interval for holding Boskos resources to a shorter time #3214
• Update Prow build cluster node type from n1-standard-16 to e2-standard-16 #3120
• Reduce frequency of running periodic Prow jobs #3121
• Turn on optimize-utilization profile for Autoscaling for the Prow build cluster #3127
• Change Prow control plane cluster to smaller machines #3128
• Change max_pod_age to 30m and max_prowjob_age to 24h for Prow #3129
• Separate system pods and Prow job pods to different node pools for the Prow build cluster #3130
• Apply go module caching and build caching to Prow jobs #2466
• Turn off vulnerability scanning for knative-nightly images #3163

/kind cncf-infra

[mattmoor]

From the meeting: we should be very very very careful about any sort of image cleanup.

There was cleanup logic added to Knative very early on which ended up doing an rm -rf on our public release public and burned a day or so of my and others time getting the GCR/GCS teams to restore it from backups.

If release images aren't a significant cost, then please just don't touch them.

If they are, the cost is almost certainly egress, and whatever solution we implement should shadow what Kubernetes is doing to solve this (still WIP), which does NOT involve deleting things, but mirroring them to avoid egress charges.

---

On GCS object lifecycle management, I said "don't turn it on", but this was really aimed at using it to GC old blobs.

The more nuanced answer is: OLM should absolutely be turned on, but the feature around retaining older generations of objects so that we don't have to page the GCS oncall to restore things from backup. The features that allow object TTLs should absolutely never be used on a GCR bucket.

Turning on OLM to avoid paging GCS was one of the requests from the GCS team when we had to restore things from backup (above).

[chizhg]

/cc @upodroid @kvmware

This issue is tracking all the work we are doing / planning to do to reduce the infra cost. If you have other ideas, please create a new issue and link it here. If you have concerns on the existing issues, please leave comments under the corresponding issue. Thanks!

[chizhg]

As of now, we have met our goal to reduce the infra cost to below $250k/year (the cost for 2022/04 was ~13k so the year round will be ~$156k). The remaining tasks initially added to this issue are good-to-have and are already tracked in separate issues. Closing this issue to mark it as resolved.