Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

loadBalancerSourceRanges not supported in AntreaProxy #5493

Open
jayunit100 opened this issue Sep 18, 2023 · 6 comments · May be fixed by #6181
Open

loadBalancerSourceRanges not supported in AntreaProxy #5493

jayunit100 opened this issue Sep 18, 2023 · 6 comments · May be fixed by #6181
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. reported-by/end-user Issues reported by end users.

Comments

@jayunit100
Copy link
Contributor

Describe the bug

As of now antrea doesnt support loadBalancerSourceRanges in the antreaProxy. Since this is also not supported in the kube-proxy, we have no way to provdide this to Windows users.

To Reproduce

Similar to kubernetes/kubernetes#120033 ...

Expected

AntreaProxy would fully support the Kubernetes service spec...

Actual behavior

LoadbalancerSourceRanges that are outside of an packet's IP are allowed into antrea clusters where antreaproxy is used.

Note this isnt a HUGE bug b/c alas, even the windows service proxy doesnt yet implement this

QUESTION:

Could this be done by reusing HNS packet filtering from the HNS ACLs ? Or does it require OVS?

aclPolicy := hns.ACLPolicy{
	Type:            hns.ACL,
	Action:          hns.Block,
	Direction:       hns.In,
	RemoteAddresses: loadBalancerSourceRanges
}
```?
@jayunit100 jayunit100 added the kind/bug Categorizes issue or PR as related to a bug. label Sep 18, 2023
@tnqn
Copy link
Member

tnqn commented Sep 18, 2023

@hongliangl @wenyingd could you please take a look from Linux and Windows perspectives?

@hongliangl
Copy link
Contributor

hongliangl commented Sep 19, 2023

After discussing @wenyingd, we have two designs.

Take an example of two LoadBalancer Services

  • Service A, ingress Ips are 192.168.77.100 and 192.168.77.101, destination port 80, no source range
  • Service B, ingress Ips are 192.168.77.200 and 192.168.77.201, destination port 80, source ranges are 192.168.1.0/24 and 10.10.1.0/24

For design 1:

  • Add a new reg mark ToLoadBalancerRegMark = binding.NewOneBitRegMark(4, 28)
  • Add a new reg mark NotAllowedSourcesRegMark = binding.NewOneBitRegMark(4, 29)
  • We reuse NodePortMarkTable and rename it to ServiceMarkTable, add the following flows:
    • For Service A:
     table=ServiceMarkTable, priority=200, tcp, nw_dst=192.168.77.100, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
     table=ServiceMarkTable, priority=200, tcp, nw_dst=192.168.77.101, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
    
    • For Service B:
     table=ServiceMarkTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
     table=ServiceMarkTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
     table=ServiceMarkTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
     table=ServiceMarkTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[27]
     table=ServiceMarkTable, priority=100, tcp, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
     table=ServiceMarkTable, priority=100, tcp, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x1->NXM_NX_REG4[28]
    
  • In ServiceLBTable:
    • For Service A:
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.100, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x1->reg7,group:1
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.101, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x2->reg7,group:2
    
    • For Service B:
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.200, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x3->reg7,group:3
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[27]=0x1, nw_dst=192.168.77.201, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x4->reg7,group:4
    
    • Reject flow (fixed flow, install it when initializing flows):
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, NXM_NX_REG4[28]=0x1 actions=controller(reason=no_match,max_len=128,id=28467,userdata=04)
    

For design 2:

  • Add a new table called LoadBalancerSourceFilterTable after ServiceLBTable.
  • Add a new reg mark AllowedSourcesRegMark = binding.NewZeroBitRegMark(4, 27)
  • Add a new reg mark NotAllowedSourcesRegMark = binding.NewOneBitRegMark(4, 27)
  • In ServiceLBTable:
    • For Service A:
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.100, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x1->reg7, resubmit(,LoadBalancerSourceFilterTable),  group:1
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.101, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x2->reg7, resubmit(,LoadBalancerSourceFilterTable),  group:2
    
    • For Service B:
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000 , nw_dst=192.168.77.200, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x3->reg7, resubmit(,LoadBalancerSource), group:3
     table=ServiceLBTable, priority=200,tcp, reg4=0x10000/0x70000, nw_dst=192.168.77.201, tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x4->reg7, resubmit(,LoadBalancerSource), group:4
    
  • In LoadBalancerSourceFilterTable:
    • For Service A:
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_dst=192.168.77.100, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_dst=192.168.77.101, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    
    • For Service B:
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=192.168.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.200, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
     table=LoadBalancerSourceFilterTable, priority=200, tcp, nw_src=10.10.1.0/24, nw_dst=192.168.77.201, tp_dst=80 actions=load:0x0->NXM_NX_REG4[27]
    
    • Default drop flow (fixed flow, install it when initializing flows):
     table=LoadBalancerSourceFilterTable, priority=100, actions=load:0x1->NXM_NX_REG4[27]
    
  • In EndpointDNATTable, there is flow to match the packets which are not allowed and send them to controller to generate a reject packet
     table=EndpointDNATTable, priority=200,NXM_NX_REG4[28]=0x1 actions=controller(reason=no_match,max_len=128,id=28467,userdata=04)
    

Could you give some suggestions @tnqn? Thanks a lot.

@tnqn tnqn added the reported-by/end-user Issues reported by end users. label Dec 11, 2023
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 11, 2024
@hongliangl hongliangl removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 11, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Apr 2, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Apr 29, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Apr 29, 2024
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 10, 2024
@tnqn tnqn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 12, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 3, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 3, 2024
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 11, 2024
@hongliangl hongliangl removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 11, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 13, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 13, 2024
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 13, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services. Key changes include:

- Added a new reg mark, `LoadBalancerSourceRangesRegMark` (bit 29 of reg4),
  which is used to mark packets sourced from the CIDRs included in
  `loadBalancerSourceRanges`.
- Renamed register mark `SvcNoEpRegMark` (bit 14 of reg0) to `SvcRejectRegMark`
  to generalize its use. It is used mark packets not sourced from CIDRs included
  in `loadBalancerSourceRanges`.
- Renamed table `NodePortMark` to `ServiceMark` to generalize its use for
  installing flows for LoadBalancer Services with `loadBalancerSourceRanges`
  configured.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.152
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x20000000/0x20000000->reg4"
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x20000000/0x20000000->reg4"
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x4000/0x4000->reg0"
4. table=ServiceLB, priority=200,tcp,reg4=0x20010000/0x20070000,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x14->reg7,group:20
5. table=EndpointDNAT, priority=200,reg0=0x4000/0x4000 actions=controller(reason=no_match,id=62373,userdata=04)
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with `SvcRejectRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesRegMark` and
  performs load balancing.
- Flow 5 is to match not allowed packets with `SvcRejectRegMark`, resulting in a reject
  reply.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 13, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services. Key changes include:

- Added a new reg mark, `LoadBalancerSourceRangesRegMark` (bit 29 of reg4),
  which is used to mark packets sourced from the CIDRs included in
  `loadBalancerSourceRanges`.
- Renamed register mark `SvcNoEpRegMark` (bit 14 of reg0) to `SvcRejectRegMark`
  to generalize its use. It marks packets not sourced from CIDRs included
  in `loadBalancerSourceRanges`.
- Renamed table `NodePortMark` to `ServiceMark` to generalize its use for
  installing flows for LoadBalancer Services with `loadBalancerSourceRanges`
  configured.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.152
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x20000000/0x20000000->reg4"
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x20000000/0x20000000->reg4"
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x4000/0x4000->reg0"
4. table=ServiceLB, priority=200,tcp,reg4=0x20010000/0x20070000,nw_dst=192.168.77.152,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x14->reg7,group:20
5. table=EndpointDNAT, priority=200,reg0=0x4000/0x4000 actions=controller(reason=no_match,id=62373,userdata=04)
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with `SvcRejectRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesRegMark` and
  performs load balancing.
- Flow 5 is to match not allowed packets with `SvcRejectRegMark`, resulting in a reject
  reply.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 25, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.152
```

To implement `loadBalancerSourceRanges`, a new table LoadBalancerSourceRanges is introduced
after table PreRoutingClassifier. Here are the corresponding flows:

```text
1. table=LoadBalancerSourceRanges, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=goto_table:SessionAffinity",
2. table=LoadBalancerSourceRanges, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=goto_table:SessionAffinity",
3. table=LoadBalancerSourceRanges, priority=190,tcp,nw_dst=192.168.77.152,tp_dst=80 actions=drop",
4. table=LoadBalancerSourceRanges, priority=0 actions=goto_table:SessionAffinity
```

Flows 1-2 allow packets destined for the for sample [LoadBalancer] from CIDRs specified in the `loadBalancerSourceRanges`
of the Service.

Flow 3, with lower priority, drops packets destined for the sample [LoadBalancer] that don't match any CIDRs within the
`loadBalancerSourceRanges`.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 27, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.152
```

To implement `loadBalancerSourceRanges`, a new table LoadBalancerSourceRanges is introduced
after table PreRoutingClassifier. Here are the corresponding flows:

```text
1. table=LoadBalancerSourceRanges, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=goto_table:SessionAffinity",
2. table=LoadBalancerSourceRanges, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.152,tp_dst=80 actions=goto_table:SessionAffinity",
3. table=LoadBalancerSourceRanges, priority=190,tcp,nw_dst=192.168.77.152,tp_dst=80 actions=drop",
4. table=LoadBalancerSourceRanges, priority=0 actions=goto_table:SessionAffinity
```

Flows 1-2 allow packets destined for the for sample [LoadBalancer] from CIDRs specified in the `loadBalancerSourceRanges`
of the Service.

Flow 3, with lower priority, drops packets destined for the sample [LoadBalancer] that don't match any CIDRs within the
`loadBalancerSourceRanges`.

Signed-off-by: Hongliang Liu <[email protected]>
Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 30, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.150
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x40000000/0x60000000->reg4",
4. table=ServiceLB, priority=200,tcp,reg4=0x0x20010000/0x0x60070000,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0xe->reg7,group:14
5. table=ServiceLB, priority=190,reg4=0x40000000/0x60000000 actions=drop
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesAllowRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with
  `LoadBalancerSourceRangesDropRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesAllowRegMark` and
  perform load balancing.
- Flow 5 is to match not allowed packets with `LoadBalancerSourceRangesDropRegMark` and
  drop.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 30, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.150
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x40000000/0x60000000->reg4",
4. table=ServiceLB, priority=200,tcp,reg4=0x0x20010000/0x0x60070000,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0xe->reg7,group:14
5. table=ServiceLB, priority=190,reg4=0x40000000/0x60000000 actions=drop
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesAllowRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with
  `LoadBalancerSourceRangesDropRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesAllowRegMark` and
  perform load balancing.
- Flow 5 is to match not allowed packets with `LoadBalancerSourceRangesDropRegMark` and
  drop.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 30, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.150
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x40000000/0x60000000->reg4",
4. table=ServiceLB, priority=200,tcp,reg4=0x0x20010000/0x0x60070000,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0xe->reg7,group:14
5. table=ServiceLB, priority=190,reg4=0x40000000/0x60000000 actions=drop
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesAllowRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with
  `LoadBalancerSourceRangesDropRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesAllowRegMark` and
  perform load balancing.
- Flow 5 is to match not allowed packets with `LoadBalancerSourceRangesDropRegMark` and
  drop.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Sep 30, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.150
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x40000000/0x60000000->reg4",
4. table=ServiceLB, priority=200,tcp,reg4=0x0x20010000/0x0x60070000,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0xe->reg7,group:14
5. table=ServiceLB, priority=190,reg4=0x40000000/0x60000000 actions=drop
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesAllowRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with
  `LoadBalancerSourceRangesDropRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesAllowRegMark` and
  perform load balancing.
- Flow 5 is to match not allowed packets with `LoadBalancerSourceRangesDropRegMark` and
  drop.

Signed-off-by: Hongliang Liu <[email protected]>
hongliangl added a commit to hongliangl/antrea that referenced this issue Oct 8, 2024
For antrea-io#5493

This commit introduces support for loadBalancerSourceRanges for LoadBalancer
Services.

Here is an example of a LoadBalancer Service configuration allowing access
from specific CIDRs:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: sample-loadbalancer-source-ranges
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerSourceRanges:
    - "192.168.77.0/24"
    - "192.168.78.0/24"
status:
  loadBalancer:
    ingress:
      - ip: 192.168.77.150
```

[New] Here are the corresponding flows:

```text
1. table=ServiceMark, priority=200,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
2. table=ServiceMark, priority=200,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x20000000/0x60000000->reg4",
3. table=ServiceMark, priority=190,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x40000000/0x60000000->reg4",
4. table=ServiceLB, priority=200,tcp,reg4=0x0x20010000/0x0x60070000,nw_dst=192.168.77.150,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0xe->reg7,group:14
5. table=ServiceLB, priority=190,reg4=0x40000000/0x60000000 actions=drop
```

- Flow 1 is to match packets from allowed CIDR `192.168.77.0/24`, marking them with
  `LoadBalancerSourceRangesAllowRegMark`.
- Flow 2 is similar to flow 1 but for CIDR `192.168.78.0/24`.
- Flow 3 is to match packets not from allowed CIDRs, marking with
  `LoadBalancerSourceRangesDropRegMark`.
- Flow 4 is to match allowed packets with `LoadBalancerSourceRangesAllowRegMark` and
  perform load balancing.
- Flow 5 is to match not allowed packets with `LoadBalancerSourceRangesDropRegMark` and
  drop.

Signed-off-by: Hongliang Liu <[email protected]>
@hongliangl
Copy link
Contributor

For the current design implemented in #6181, see the git commit message. In an offline discussion, @wenyingd proposed a new design for the implementation:

Here are the flows of the latest Antrea main branch code with proxyAll enabled. I believe that everyone is very familiar with them.

...
table=ConntrackState, priority=0 actions=goto_table:PreRoutingClassifier

table=PreRoutingClassifier, priority=200,ip actions=resubmit(,NodePortMark),resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=200,ipv6 actions=resubmit(,NodePortMark),resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=0 actions=goto_table:NodePortMark

table=NodePortMark, priority=200,ip,nw_dst=10.176.27.91 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=192.168.77.100 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=172.17.0.1 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ip,nw_dst=169.254.0.252 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=2620:124:6020:1006:250:56ff:fea7:2787 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=fd00::192:168:77:100 actions=set_field:0x80000/0x80000->reg4
table=NodePortMark, priority=200,ipv6,ipv6_dst=fc01::aabb:ccdd:eefe actions=set_field:0x80000/0x80000->reg4

table=SessionAffinity, priority=0 actions=set_field:0x10000/0x70000->reg4

table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.1,tp_dst=443 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x5->reg7,group:5
table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.225.251,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x30000/0x70000->reg4,set_field:0x6->reg7,group:6
table=ServiceLB, priority=0 actions=goto_table:DSRServiceMark

Here are the draft flows of the design to implement loadBalancerSourceRanges:

...
table=ConntrackState, priority=0 actions=goto_table:LoadBalancerSourceRange

table=LoadBalancerSourceRange, priority=210,tcp,nw_src=192.168.77.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=goto_table:PreRoutingClassifier
table=LoadBalancerSourceRange, priority=210,tcp,nw_src=192.168.78.0/24,nw_dst=192.168.77.150,tp_dst=80 actions=goto_table:PreRoutingClassifier
table=LoadBalancerSourceRange, priority=200,tcp,nw_dst=192.168.77.150,tp_dst=80 actions=drop
table=LoadBalancerSourceRange, priority=0 actions=goto_table:NodePortMark

table=NodePortMark, priority=200,ip,nw_dst=169.254.0.252 actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=200,ipv6,ipv6_dst=fd00::192:168:77:100 actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=200,ipv6,ipv6_dst=fc01::aabb:ccdd:eefe actions=set_field:0x80000/0x80000->reg4, goto_table:PreRoutingClassifier
table=NodePortMark, priority=0 actions=goto_table:PreRoutingClassifier

table=PreRoutingClassifier, priority=0,ip actions=resubmit(,SessionAffinity),resubmit(,ServiceLB)
table=PreRoutingClassifier, priority=0,ipv6 actions=resubmit(,SessionAffinity),resubmit(,ServiceLB)

table=SessionAffinity, priority=0 actions=set_field:0x10000/0x70000->reg4

table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.1,tp_dst=443 actions=set_field:0x200/0x200->reg0,set_field:0x20000/0x70000->reg4,set_field:0x5->reg7,group:5
table=ServiceLB, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.225.251,tp_dst=80 actions=set_field:0x200/0x200->reg0,set_field:0x30000/0x70000->reg4,set_field:0x6->reg7,group:6
table=ServiceLB, priority=0 actions=goto_table:DSRServiceMark

We can see key changes:

  • Add a new table LoadBalancerSourceRange (this is not a great name; we may come up with another name if we use this design finally).
    • Add flows to match the packets sourced from each allowed CIDR and destined for the LoadBalancer IP and port, and forward them to the table PreRoutingClassifier.
    • Add flow to match the packets not sourced from any allowed CIDR and drop them.
    • Add a default flow to match other packets and forward them to the table NodePortMark to do NodePort traffic pre-checks.
  • Table NodePortMark is like before, marking the potential NodePort packets, and all packets will be forwarded to the table PreRoutingClassifier.
  • Table PreRoutingClassifier (maybe we should change the name, since xxxClassifier is always the first table of a stage, according to the convention of flexible pipeline design. PreRoutingClassifier is not a proper name again as it is not the first table of stage PreRouting) is like before, sending the packets to tables SessionAffinity and ServiceLB to implement Service load-balancing and sessionAffinity.

Look forwarding to your suggestions @tnqn @antoninbas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. reported-by/end-user Issues reported by end users.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants