Network binding plugin: Support compute container resource overhead #303

orelmisan · 2024-06-16T20:44:36Z

What this PR does / why we need it:
Some network binding plugins require compute resource overhead, for example the passt plugin requires additional memory overhead in the virt-launcher pod's compute container.
Suggest several alternatives to address this issue.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note:

NONE

kubevirt-bot · 2024-06-16T20:44:38Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

kubevirt-bot · 2024-06-16T20:44:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign alonakaplan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

orelmisan · 2024-06-16T20:44:53Z

/cc @EdDev @ormergi

orelmisan · 2024-06-16T20:45:22Z

/uncc @aburdenthehand @jobbler

ormergi

Thanks for the PR, overall looks good, see my inline comments.

Regarding the the PR description and second commit message, I think it should also mention that memory overhead is necessary to avoid pod eviction due to passt VMs consume more memory then expected, and improve passt VMs scheduling results; passt VMs wont be scheduled on nodes that doesnt have enough memory.

ormergi · 2024-06-18T09:04:27Z

design-proposals/network-binding-plugin/network-binding-plugin.md

@@ -1261,6 +1271,10 @@ metadata:
    k8s.v1.cni.cncf.io/networks: '[{"name":"netbindingpasst","namespace":"mynamespace","cni-args":{"logicNetworkName":"default"}}]'
 spec:
  containers:
+    - name: compute
+      resources:
+        requests:


I think resource limits should be as well, in case the pod satisfy QoS class Guaranteed the plugin overhead will not become the reason for violating it.

AFAIK, there is logic [1] to automatically add memory limits when needed (there is also an equivalent for CPU).

[1] https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L189

ormergi · 2024-06-18T09:07:11Z

design-proposals/network-binding-plugin/network-binding-plugin.md

@@ -387,6 +388,13 @@ Alternatives passing the domain attachment to sync VMI:
 Both of the options are not perfect, but the `virt-handler.SyncVMI`
 has fewer cons. Therefore, it was chosen.

+#### Additional resource requests for the virt-launcher compute container


I would rephrase this section to be vogue about the concrete plugin or dependency inside virt-launcher that requires it, saying in case a plugin requires memory overhead it should be specified in the CR, and refer to passt example.

Adjusted the wording, please tell me what you think.

design-proposals/network-binding-plugin/network-binding-plugin.md

ormergi · 2024-06-18T09:38:37Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+The passt binary is shipped and executed inside the virt-launcher pod.
+
+### Domain definition & creation


I would simplify this section saying the sidecar adds passt interface to the domain, similar to vDPA example, maybe mention it use libvirt user-space networking settings https://libvirt.org/formatdomain.html#userspace-slirp-or-passt-connection.
You can also refer to slirp example section saying passt sidecar works in similar way.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev

First commit review.

Although I placed comments inline on that commit, I think it is not a must example addition.

At the design level, we are not really interested in a specific binding, but on the general concept.
The general concept can be applied on other bindings, just to provide an example on how to use it.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev · 2024-06-18T16:38:55Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+  namespace: default
+spec:
+  config: '{
+      "cniVersion": "0.3.1",


Maybe we should use 1.0.X in the example.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:43:41Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+metadata:
+  name: virt-launcher-123
+  annotations:
+    k8s.v1.cni.cncf.io/networks: '[{"name":"netbindingpasst","namespace":"mynamespace","cni-args":{"logicNetworkName":"default"}}]'


I think the logicNetworkName is supposed to be passtnet.

This was removed.
Please ack so we can resolve the thread.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev · 2024-06-18T16:47:30Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+### Configure Pod network namespace
+
+Not required for passt binding


We do configure networking for passt.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:48:54Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+### Run services in the virt-launcher pod
+
+The passt binary is shipped and executed inside the virt-launcher pod.


This should say: Not required for passt binding.

This was removed.
Please ack so we can resolve the thread.

EdDev · 2024-06-18T16:52:11Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+    <portForward proto='udp'/>
+    <model type='virtio-non-transitional'/>
+    <backend type='passt' logFile='/var/run/kubevirt/passt.log'/>
+    <alias name='ua-default'/>


This name is not in-sync with the network name used in the VM spec.

This was removed.
Please ack so we can resolve the thread.

EdDev

Second commit:

The commit message hints and explains the passt binding and everything is driven from it. While this is correct in terms of why we do all this, I do not think we should start the story there.

The story should start from the need, mentioning passt as an example.
The fact that there is a binary alongside libvirt is not that important. If it was part of libvirt we would have needed to take that into account as well.

Also, it would be better to leave the details to the design and in the commit just provide the topic/subject. That way we can easily review it and adjusts it per that review.

EdDev · 2024-06-18T17:39:23Z

design-proposals/network-binding-plugin/network-binding-plugin.md

@@ -189,6 +189,7 @@ the binding is referenced by name with optional additional arguments.
        - domainAttachment (a standard domain definition that exists in
          the core . E.g `tap`, `sriov`).
        - downwardAPI (E.g `device-info`)
+        - resourceOverhead (currently only additional memory overhead request for the compute container in the virt-launcher pod is supported)


I would be specific here and mention the compute-container in the name.
E.g. computeResources.

This also opens up the different questions and options which should be discussed in a proposal:

Should we consider only the compute container? What about the sidecar container?

Ref the compute container, we can use a specific name for the field (e.g. computeResources) or we can make it configurable using a flag under a general resources.

What is the reason for not just doing a computeMemoryRequest needs to convince the reviewers.

Now, we can make it even more general by moving away from the specific network usage of this and look at a general sidecar hook that KubeVirt supports.
E.g.: The Kubvirt CR will have a policy for allocating resources to sidecar containers and possible the compute container. Referencing this policy from the network binding definition or from a sidecar hook. This may be an overkill, but thinking about it and examining the pors/cons of it can be useful.

EdDev · 2024-06-18T17:44:03Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+Some binding plugins may require an additional binary to be shipped inside the virt-launcher compute image.
+This binary requires additional resources that have to be explicitly stated.


I do not think we need to limit it to an additional binary. It could be just more resources from the cgroupv2 of the compute container, consumed by an existing binary (e.g. libvirt).

EdDev · 2024-06-18T17:53:07Z

design-proposals/network-binding-plugin/network-binding-plugin.md

@@ -1189,6 +1196,8 @@ spec:
        passt:
          networkAttachmentDefinition: default/netbindingpasst
          sidecarImage: quay.io/kubevirt/network-passt-binding
+          resourceOverhead:
+            memory: 800Mi


I think you will have to explain why not:

Directly memoryOverhead: 500Mi (BTW, it should be 500 and not 800).

With explicit request and being open for adding limit.

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/

ie.

overhead: podFixed: memory: "120Mi" cpu: "250m"

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea

Interesting, thanks for sharing this.
I think the API could be in sync with the general resources concept.
This allows to control the resource additions to the container, with the ability to extend it in the future to any type of resource and define both request and limits.

The oddity here, as I see it, is how we express the resources of a different container (i.e. compute) and still not lock the ability to do the same for the sidecar itself.

@EdDev I'm late to review this, but I don't understand why this API needs to add anything to the compute container and not simply advertise the resources needed to run this binding container?!
Same way as it's done for hotplug / other containers?
https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L611

runtimeClasses have a pretty similar concept, maybe worth aligning the API idea https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/

ie.

overhead: podFixed: memory: "120Mi" cpu: "250m"

@fabiand I doubt it aligns well. This overhead API was invented for kata and the overhead is being added to any pod created with this runtimeclass, I don't think it aligns.

@EdDev I'm late to review this, but I don't understand why this API needs to add anything to the compute container and not simply advertise the resources needed to run this binding container?! Same way as it's done for hotplug / other containers? https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-controller/services/renderresources.go#L611

@EdDev do we expect all network binding to affect the compute container or is it just passt?
passt specifically uses a feature in qemu there for the GetMemoryOverhead functional can identify that passt will be used and add a passt specific overhead - this is instead of a kubevirt CR API.
What do you think?

@vladikr It is unknown at this stage how future network binding plugins will affect the compute container's resource consumption.
Network binding plugins have the ability to configure the domain, thus the virt stack might consume additional compute resources which KubeVirt cannot account for.

In the past year, the passt binding was converted from being a core feature to a plugin, so KubeVirt will not know in advance that is is used.
Also, we should strive for treating all plugins as equal IMO.

@vladikr my comment was only about the API design, not about using the POD api.

@vladikr Hi
I see that according to the network binding plugin design document, one of the integration points that seems legit is the pod definition phase
There were no exceptions regarding resource rendering, and Kubevirt CR was advised as a potential API to extend in order to integrate into that point.

Given that the network binding design was accepted, and also the implementation of this design also got into the codebase, I am not sure how to proceed, I'm in favor of accepting this design.

Can you please advice @vladikr ?

orelmisan · 2024-06-23T10:52:30Z

Change: Removed the passt example, as it is not essential for to the proposal.

Addressed sidecar and compute container resource specification.

orelmisan · 2024-06-23T10:58:04Z

@EdDev @ormergi PTAL.

EdDev

First commit review.

EdDev · 2024-06-23T12:04:26Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+The sidecar container can have a memory leak and may cause node's destabilization.
+
+
+Alternatives:


Please keep here the decided solution and place the alternatives (with a ref from here) to a dedicated appendix.

EdDev · 2024-06-23T12:19:07Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+   - Coarse level of control.
+   - Only supports CPU and memory requests and limits.
+
+2. Additional API for sidecar resource configuration:


This is specific for the binding plugin, not the sidecar. The sentence is not clear about this.
The definition is per the network binding plugin, and applied if the plugin uses a sidecar.

EdDev · 2024-06-23T12:20:45Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Cons:
+ - Require an API change.
+ - The API will probably evolve as additional plugins will be created.


I do not understand this one.

Discussed this point offline.
It was removed.

EdDev · 2024-06-23T12:21:53Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+Cons:
+ - Require an API change.
+ - The API will probably evolve as additional plugins will be created.
+ - May require cluster admins to adjust plugin resources during the plugin's lifecycle.


The cluster admin is responsible to register the binding plugin in the first place, so I am unclear what this point means.

Discussed this point offline, improved the wording.

EdDev · 2024-06-23T12:25:12Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+   - Only supports CPU and memory requests and limits.
+
+2. Additional API for sidecar resource configuration:
+The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the sidecar resource requirements:


The specified resource is define for each instance of usage or per the sidecar, irrelevant of the usage?
E.g. there may be 1 interface using the plugin or there may be 3 interfaces using the plugins in the same VM.

Addressed it in the text.

EdDev · 2024-06-23T12:29:53Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+For each network binding plugin used, the VMI controller will add a label on the virt-launcher pod with the following format:
+
+`kubevirt.io/network-binding-plugin:<plugin-name>`


How will the admission webhook be able to identify the relevant container?

Addressed it in the text.

design-proposals/network-binding-plugin/network-binding-plugin.md

EdDev

Thank you for the proposal.

The following general points should be considered and added:

Effort cost estimation for each option.

EdDev · 2024-06-24T06:05:47Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+For some plugins, such as passt, there is a need to execute an additional binary in the compute container.
+Since this binary has its own CPU and memory limits, they should be somehow accounted for.


Please do not limit it to running an additional binary, the addition in memory or other resources may come from different reasons (e.g. libvirt itself requiring more memory due to the expected configuration).

EdDev · 2024-06-24T06:07:57Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Alternatives:
+1. Manually setting the VM's resources:
+The user can override KubeVirt's algorithms and set resource requirements.


The current logic that adds overhead per internal core logic is colliding with adding manually an overhead? Specifically, if one specifies explicitly resource requests, are these being increased by Kubevirt logic or something else is happening?

The way it works is as follows:
A VM could be defined with both guest memory and memory resource specification:

apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: vm-cirros spec: template: spec: domain: memory: guest: 128Mi resources: requests: memory: 640Mi # 128Mi for the guest + 512Mi for the network binding plugin

The virt-launcher pod's compute container will have a memory request which contains the sum of:

Guest VM's memory (128Mi in this example).

Memory overhead for KubeVirt's components (its size is dependent on the VMI's spec)

Arbitrary memory overhead (512Mi in this example)

The domain XML will contains the the guest's memory specification (128Mi in this example).

As a side note, it is also possible to specify a memory request with less memory than the guest requires.

EdDev · 2024-06-24T06:09:02Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+Cons:
+- Error prune
+- The user does not take into account the overhead considerations KubeVirt takes when templating a virt-launcher pod.


I guess this is related to my prev question. Can you please confirm this is indeed the case?
I would be surprise if this is how it works.

This is not true.
I removed this line.

EdDev · 2024-06-24T06:10:51Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+- The API will probably evolve as additional plugins will be created.
+- May require cluster admins to adjust plugin resources during the plugin's lifecycle.


Like with the sidecar previously, these points are not clear.

Discussed this point offline, adjusted the text.

EdDev · 2024-06-24T06:15:54Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+2. Additional API for compute container resource overhead:
+
+The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the resource requirements overhead for the compute container:


There is a need to explicitly specify if this overhead is added per plugin type or per plugin usage count (i.e. per the number of interfaces referencing it from the VM).

It is also important to specify if it is dependent on any other field/setup.
The previous sidecar resource are dependent on the existence of a sidecar, but this one may not have such a dependency.

orelmisan · 2024-07-02T06:55:26Z

Change: Moved the API alternative to be the proposed solution.

aburdenthehand · 2024-07-03T14:07:53Z

/cc @stu-gott - needs review from someone in sig-compute. Thanks!

EdDev · 2024-07-04T08:41:13Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+The requirement to maintain certificates for the webhook could be mitigated using tools such as [cert-manager](https://cert-manager.io/).
+
+# Appendix H:  Alternatives to compute container resource overhead specification


After a discussion with @enp0s3 , there is another more generic solution alternative:

Add a new resource overhead section under the KV config:

computeOverhead: selector: .... resources: requests: cpu: 200m memory: 20Mi

The selector will determine to which pod it will apply to. The binding plugin, will create pods with a label that identifies the pod as using the relevant plugin. The code that adds the overhead, will query it and determine on which pod to add the overhead.

@EdDev Hi. I think we would need to open a separate design proposal for the generic solution.

Sure, but we still want to record it here as it is related.
A remark can be added to express that it it will be presented as an independent proposal.

Would the spesifed resources apply for all containers in the selected pod?
Resources are configured on container types, it seem that it"ll require finer way to set overhead on containers, did you mean selector for containers?

FYI: I suggested something similar in this comment.
Note that the container name should be validated and allow refering to the binding sidecar and compute containers only.

Would the spesifed resources apply for all containers in the selected pod?
Resources are configured on container types, it seem that it"ll require finer way to set overhead on containers, did you mean selector for containers?

I'm sure author of the proposal will handle this detail.

FYI: I suggested something similar in this comment.
Note that the container name should be validated and allow refering to the binding sidecar and compute containers only.

The alternative here is generic, not specific to a binding plugin and requires other steps for it to work.
I think we can get into the details of what it needs to validate or not as part of a different proposal that will focus on this alternative. It should be enough to mention here the general idea, so it will be visible that we discussed it.

EdDev · 2024-07-04T08:43:40Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+          computeResourceOverhead:
+              cpu: 200m
+              memory: 20Mi


It seems this is missing the requests (and limits) level, why?

Re-added the requests field.

orelmisan · 2024-07-05T20:02:24Z

Change: Updated the compute container's manual resource specification alternative.

ormergi

Thanks for the changes

ormergi · 2024-07-07T09:01:58Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+      binding:
+        mynetbindingplugin:
+          sidecarImage: quay.io/kubevirt/mynetbindingplugin
+          sidecarResources:


Please add a note saying this field type should be the native kubernetes resources API, with validation for cpu and memory.

I agree that we should be explicit about the supported API.
We should not specify the cpu but we can add a not that the structure is the native k8s one, allowing extending the support to the other fields (cpu, custom ones).

The definition of the API should not be limited to an example.
But this is not blocker for me.

Adding the validation section can also be useful. We could specify there that we ignore any data except the memory at this stage. We should be explicit and say that we do not fail if other data is added. Again, this is not a blocker and I am good with letting this stay gray for now.

ormergi · 2024-07-07T09:12:35Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+    - Only supports CPU and memory requests and limits.
+
+2. Additional API for network binding plugin sidecar resource specification:


Yep, the container names should be validated in a way they can refer to the binding sidecar and compute container only.

ormergi · 2024-07-07T09:12:45Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+The requirement to maintain certificates for the webhook could be mitigated using tools such as [cert-manager](https://cert-manager.io/).
+
+# Appendix H:  Alternatives to compute container resource overhead specification


Would the spesifed resources apply for all containers in the selected pod?
Resources are configured on container types, it seem that it"ll require finer way to set overhead on containers, did you mean selector for containers?

FYI: I suggested something similar in this comment.
Note that the container name should be validated and allow refering to the binding sidecar and compute containers only.

ormergi · 2024-07-07T09:18:25Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+If specified, KubeVirt will add the resource overhead specified in `computeResourceOverhead` to the compute container of the virt-launcher pod.
+The specified resource overhead will be added to the compute container per unique usage of the plugin (not per NIC using the binding).
+
+For example, assuming there is a plugin registered with a 200Mi memory overhead for the `compute` container, and


Does passt require mempry overhead per interface or based on whether its being executed or not?

It is enough for the interface to be defined for the passt binary to run and consume the memory specified.

It is not clear what you mean by executed.

ormergi · 2024-07-07T09:20:36Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+          sidecarImage: quay.io/kubevirt/mynetbindingplugin
+          sidecarResources:
+            requests:


This is a mere yaml example it can be anything 🙂

ormergi · 2024-07-07T09:28:08Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+- Exposes the VM owner to unnecessary implementation details
+
+2. Mutating webhook


Is it another webook in addition to one that address the sidecar resources one?

It seem that it lacks description about what it"ll do, pros, cons section and how its going to integrate with kubevirt? ( I recall something about adding some label to launcher pods that use network binding plugins)

I've added additional details, same as for the sidecar.

EdDev · 2024-07-07T10:15:32Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+      binding:
+        mynetbindingplugin:
+          sidecarImage: quay.io/kubevirt/mynetbindingplugin
+          sidecarResources:


I agree that we should be explicit about the supported API.
We should not specify the cpu but we can add a not that the structure is the native k8s one, allowing extending the support to the other fields (cpu, custom ones).

The definition of the API should not be limited to an example.
But this is not blocker for me.

Adding the validation section can also be useful. We could specify there that we ignore any data except the memory at this stage. We should be explicit and say that we do not fail if other data is added. Again, this is not a blocker and I am good with letting this stay gray for now.

EdDev · 2024-07-07T10:25:11Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+If specified, KubeVirt will add the resource overhead specified in `computeResourceOverhead` to the compute container of the virt-launcher pod.
+The specified resource overhead will be added to the compute container per unique usage of the plugin (not per NIC using the binding).
+
+For example, assuming there is a plugin registered with a 200Mi memory overhead for the `compute` container, and


It is enough for the interface to be defined for the passt binary to run and consume the memory specified.

It is not clear what you mean by executed.

EdDev · 2024-07-07T10:26:01Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+For example, assuming there is a plugin registered with a 200Mi memory overhead for the `compute` container, and
+there are two interfaces using it.
+`virt-controller` will only add 200Mi of memory to the `compute` container.


Where is this 200Mi coming from? In the example above you used 20Mi.

It was a typo.
Fixed it in the example.

EdDev · 2024-07-07T10:34:21Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+The requirement to maintain certificates for the webhook could be mitigated using tools such as [cert-manager](https://cert-manager.io/).
+
+# Appendix H:  Alternatives to compute container resource overhead specification


Would the spesifed resources apply for all containers in the selected pod?
Resources are configured on container types, it seem that it"ll require finer way to set overhead on containers, did you mean selector for containers?

I'm sure author of the proposal will handle this detail.

FYI: I suggested something similar in this comment.
Note that the container name should be validated and allow refering to the binding sidecar and compute containers only.

The alternative here is generic, not specific to a binding plugin and requires other steps for it to work.
I think we can get into the details of what it needs to validate or not as part of a different proposal that will focus on this alternative. It should be enough to mention here the general idea, so it will be visible that we discussed it.

Signed-off-by: Orel Misan <[email protected]>

orelmisan · 2024-07-09T15:49:26Z

Change: Split the sidecar's resource specification to #309

EdDev

Thank you!

vladikr · 2024-07-25T18:09:24Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+##### Compute Container Resource Overhead
+
+For some plugins, an additional resource consumption can be expected from the virt-launcher pod compute container.
+For example, there could be need to execute an additional binary in the compute container.


Could you please explain what binary can be executed and how it gets there?
How can an external plugin rely on the fact that a specific binary is part of the virt-launcher image?

This example was derived from the passt use case, on which the passt binary is shipped as part of the virt-launcher compute image.

Another example could be that the plugin configures the domain in a way that causes the virt-stack to consume additional compute resources that KubeVirt cannot account for.

vladikr · 2024-07-25T21:57:10Z

design-proposals/network-binding-plugin/network-binding-plugin.md

+
+For some plugins, an additional resource consumption can be expected from the virt-launcher pod compute container.
+For example, there could be need to execute an additional binary in the compute container.
+Since this binary has its own CPU and memory limits, they should be somehow accounted for.


I think, we should speak of a specific functionality and not about arbitrary binaries.
virt-launcher should be well defined.
If we know what functionality we're enabling that requires additional overhead, kubevirt needs to know the resources it requires. Similar to other aspects accounted for in GetMemoryOverhead.

I would say that if we want an API for this it should be based on a functionality, not a binding.

Each resource consumption of it.

The plugin can perform actions that affect KubeVirt in ways it cannot account for.
Thus the need to externally add resources to the compute container.

orelmisan · 2024-07-30T18:07:03Z

Thank you for reviewing this proposal @vladikr, sorry it took some time to respond.

aburdenthehand · 2024-09-23T12:41:49Z

On behalf of SIG-compute, can one of @jean-edouard @stu-gott @enp0s3 @vladikr please re-review to move this forward. I can merge if I receive an approve or second lgtm from one of you. Thank you.

kubevirt-bot · 2024-09-30T13:34:53Z

Pull requests that are marked with lgtm should receive a review
from an approver within 1 week.

After that period the bot marks them with the label needs-approver-review.

/label needs-approver-review

enp0s3 · 2024-12-11T12:42:15Z

/cc

enp0s3 · 2024-12-16T19:48:00Z

Deferring the discussion for after the end year holidays

kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 16, 2024

kubevirt-bot requested review from aburdenthehand and jobbler June 16, 2024 20:44

kubevirt-bot added the size/L label Jun 16, 2024

kubevirt-bot requested review from EdDev and ormergi June 16, 2024 20:44

kubevirt-bot removed request for jobbler and aburdenthehand June 16, 2024 20:45

ormergi reviewed Jun 18, 2024

View reviewed changes

orelmisan marked this pull request as ready for review June 18, 2024 15:04

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2024

kubevirt-bot requested review from aburdenthehand and jean-edouard June 18, 2024 15:04

orelmisan force-pushed the net-binding-plugin-mem-overhead branch 2 times, most recently from ed97ca9 to 8432599 Compare June 18, 2024 17:08

EdDev reviewed Jun 18, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 8432599 to 416926c Compare June 23, 2024 10:50

EdDev reviewed Jun 23, 2024

View reviewed changes

design-proposals/network-binding-plugin/network-binding-plugin.md Show resolved Hide resolved

orelmisan changed the title ~~Network binding plugin: Support memory overhead~~ Network binding plugin: Support resource overhead Jun 23, 2024

EdDev reviewed Jun 24, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 416926c to e0b43e5 Compare June 24, 2024 15:11

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 35ddd8b to 1bf6515 Compare July 3, 2024 10:59

kubevirt-bot requested a review from stu-gott July 3, 2024 14:07

EdDev reviewed Jul 4, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch 2 times, most recently from e482700 to 848082d Compare July 5, 2024 20:00

ormergi reviewed Jul 7, 2024

View reviewed changes

EdDev reviewed Jul 7, 2024

View reviewed changes

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from 848082d to d4eb851 Compare July 7, 2024 11:57

orelmisan mentioned this pull request Jul 9, 2024

Network binding plugin: Support sidecar resource specification #309

Open

8 tasks

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from d4eb851 to c33498b Compare July 9, 2024 15:46

orelmisan changed the title ~~Network binding plugin: Support resource overhead~~ Network binding plugin: Support compute resource overhead Jul 9, 2024

Network binding plugin: Support compute container resource overhead

591c3a4

Signed-off-by: Orel Misan <[email protected]>

orelmisan changed the title ~~Network binding plugin: Support compute resource overhead~~ Network binding plugin: Support compute container resource overhead Jul 9, 2024

orelmisan force-pushed the net-binding-plugin-mem-overhead branch from c33498b to 591c3a4 Compare July 9, 2024 15:49

EdDev approved these changes Jul 10, 2024

View reviewed changes

kubevirt-bot assigned EdDev Jul 10, 2024

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 10, 2024

vladikr reviewed Jul 25, 2024

View reviewed changes

kubevirt-bot added the needs-approver-review Indicates that a PR requires a review from an approver. label Sep 30, 2024

kubevirt-bot requested a review from enp0s3 December 11, 2024 12:42


		The passt binary is shipped and executed inside the virt-launcher pod.

		### Domain definition & creation


		### Configure Pod network namespace

		Not required for passt binding


		### Run services in the virt-launcher pod

		The passt binary is shipped and executed inside the virt-launcher pod.

		Some binding plugins may require an additional binary to be shipped inside the virt-launcher compute image.
		This binary requires additional resources that have to be explicitly stated.

		The sidecar container can have a memory leak and may cause node's destabilization.


		Alternatives:

		For each network binding plugin used, the VMI controller will add a label on the virt-launcher pod with the following format:

		`kubevirt.io/network-binding-plugin:<plugin-name>`

		For some plugins, such as passt, there is a need to execute an additional binary in the compute container.
		Since this binary has its own CPU and memory limits, they should be somehow accounted for.

		- The API will probably evolve as additional plugins will be created.
		- May require cluster admins to adjust plugin resources during the plugin's lifecycle.


		2. Additional API for compute container resource overhead:

		The network binding plugin API in the KubeVirt CR could receive an additional input field to specify the resource requirements overhead for the compute container:

		The requirement to maintain certificates for the webhook could be mitigated using tools such as [cert-manager](https://cert-manager.io/).

		# Appendix H: Alternatives to compute container resource overhead specification

		- Only supports CPU and memory requests and limits.

		2. Additional API for network binding plugin sidecar resource specification:

		- Exposes the VM owner to unnecessary implementation details

		2. Mutating webhook

Network binding plugin: Support compute container resource overhead #303

Are you sure you want to change the base?

Network binding plugin: Support compute container resource overhead #303

Conversation

orelmisan commented Jun 16, 2024 • edited Loading

kubevirt-bot commented Jun 16, 2024

kubevirt-bot commented Jun 16, 2024

orelmisan commented Jun 16, 2024

orelmisan commented Jun 16, 2024

ormergi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdDev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdDev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan commented Jun 23, 2024

orelmisan commented Jun 23, 2024

EdDev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EdDev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan commented Jul 2, 2024

aburdenthehand commented Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orelmisan commented Jun 16, 2024 •

edited

Loading

orelmisan Jun 18, 2024 •

edited

Loading

orelmisan Jul 5, 2024 •

edited

Loading

aburdenthehand commented Jul 3, 2024 •

edited

Loading