Skip to content

Commit

Permalink
Reborn to network-node-manager
Browse files Browse the repository at this point in the history
  • Loading branch information
ssup2 committed Nov 19, 2020
1 parent d81112f commit a6e8842
Show file tree
Hide file tree
Showing 15 changed files with 1,099 additions and 481 deletions.
74 changes: 39 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Network Node Manager

network-node-manager is the kubernetes controller that solves External-IP (Load Balancer IP) issue with IPVS proxy mode. IPVS proxy mode has various problems, and one of them is that the External-IP assigned through the LoadBalancer type service with externalTrafficPolicy=Local option cannot access inside the cluster. More details on this issue can be found at [here](https://github.com/kubernetes/kubernetes/issues/75262). network-node-manager solves this issue. network-node-manager is based on [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder).
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes. By simply deploying and configuring network-node-manager, you can solve kubernetes network issues that cannot be resolved by kubernetes or resolved by the higher kubernetes Version. Below is a list of kubernetes's issues to be resolved by network-node-manager. network-node-manager is based on [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder).

* [External-IP access issue with IPVS proxy mode](issues/external_IP_access_issue_IPVS_proxy_mode.md)
* [Connection reset issue between pod and out of cluster](issues/connection_reset_issue_pod_out_cluster.md)

## Deploy

Expand All @@ -9,55 +12,56 @@ network-node-manager now supports below CPU architectures.
* amd64
* arm64

Deploy network-node-managers through below command.
Deploy network-node-managers through below command according to kube-proxy mode.

```
kubectl apply -f https://raw.githubusercontent.com/kakao/network-node-manager/master/deploy/network-node-manager.yml
iptables proxy mode : kubectl apply -f https://raw.githubusercontent.com/kakao/network-node-manager/master/deploy/network-node-manager_iptables.yml
IPVS proxy mode : kubectl apply -f https://raw.githubusercontent.com/kakao/network-node-manager/master/deploy/network-node-manager_ipvs.yml
```

## Configuration

### IPv6
### Network Stack (IPv4, IPv6)

network-node-manager supports IPv6. However, IPv6 is not enabled by default. To use IPv6, set "NET_STACK" environment in the DaemonSet manifests of network-node-manager as follows.
* Default : "ipv4"
* iptables proxy mode manifest : "ipv4"
* IPVS proxy mode manifest : "ipv4"

```
...
env:
- name: NET_STACK
value: ipv4,ipv6
- name: NODE_NAME
valueFrom:
...
IPv4 : kubectl -n kube-system set env daemonset/network-node-manager NET_STACK=ipv4
IPv6 : kubectl -n kube-system set env daemonset/network-node-manager NET_STACK=ipv6
IPv4,IPv6 : kubectl -n kube-system set env daemonset/network-node-manager NET_STACK=ipv4,ipv6
```

## How it works?
### External-IP to Cluster-IP DNAT Rule

network-node-manager works on all worker nodes and adds the DNAT rules that converts destination IP of a packet from External-IP to Cluster-IP to iptables. network-node-manager adds two DNAT rules for each LoadBalancer type service. One is added to the prerouting chain and the other is added to the output chain. The DNAT rule in the prerouting chain is for the pod that uses pod-only network namespace. On the other hand, The DNAT rule in the output chain is for the pod that uses host network namespace. All DNAT rules only target packets from pods on the host. Below is an example.
* Related issue : [External-IP access issue with IPVS proxy mode](issues/external_IP_access_issue_IPVS_proxy_mode.md)
* Default : false
* iptables proxy mode manifest : false
* IPVS proxy mode manifest : true

```
$ kubectl -n default get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
lb-service-1 LoadBalancer 10.231.42.164 10.19.20.201 80:31751/TCP,443:30126/TCP 16d
lb-service-2 LoadBalancer 10.231.2.62 10.19.22.57 80:32352/TCP,443:31549/TCP 16d
$ iptables -nvL -t nat
...
Chain NMANAGER_IPVS_LB_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 10.19.20.201 /* default/lb-service-1 */ ADDRTYPE match src-type LOCAL
0 0 DNAT all -- * * 0.0.0.0/0 10.19.20.201 /* default/lb-service-1 */ ADDRTYPE match src-type LOCAL to:10.231.42.164
2 120 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 10.19.22.57 /* default/lb-service-2 */ ADDRTYPE match src-type LOCAL
2 120 DNAT all -- * * 0.0.0.0/0 10.19.22.57 /* default/lb-service-2 */ ADDRTYPE match src-type LOCAL to:10.231.2.62
Chain NMANAGER_IPVS_LB_PREROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.240.2.128/25 10.19.20.201 /* default/lb-service-1 */
0 0 DNAT all -- * * 10.240.2.128/25 10.19.20.201 /* default/lb-service-1 */ to:10.231.42.164
1 60 KUBE-MARK-MASQ all -- * * 10.240.2.128/25 10.19.22.57 /* default/lb-service-2 */
1 60 DNAT all -- * * 10.240.2.128/25 10.19.22.57 /* default/lb-service-2 */ to:10.231.2.62
...
On : kubectl -n kube-system set env daemonset/network-node-manager RULE_EXTERNAL_CLUSTER=true
Off : kubectl -n kube-system set env daemonset/network-node-manager RULE_EXTERNAL_CLUSTER=false
```

### Drop Invalid Packet Rule in INPUT chain

* Related issue : [Connection reset issue between pod and out of cluster](issues/connection_reset_issue_pod_out_cluster.md)
* Default : true
* iptables proxy mode manifest : true
* IPVS proxy mode manifest : true

```
On : kubectl -n kube-system set env daemonset/network-node-manager RULE_DROP_INVALID_INPUT=true
Off : kubectl -n kube-system set env daemonset/network-node-manager RULE_DROP_INVALID_INPUT=false
```

## How it works?

![kpexec Architecture](img/network-node-manager_Architecture.PNG)

network-node-manager runs on all kubernetes cluster nodes in host network namespace with network privileges and manage the node network configuration. network-node-manager watches the kubernetes object through kubenetes API server like a general kubernetes controller and manage the node network configuration. Now network-node-manager only watches service object.

## License

Expand Down
116 changes: 116 additions & 0 deletions controllers/rule_common.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
package controllers

import (
"github.com/go-logr/logr"

"github.com/kakao/network-node-manager/pkg/configs"
"github.com/kakao/network-node-manager/pkg/iptables"
)

// Constants
const (
ChainFilterInput = "INPUT"
ChainNATPrerouting = "PREROUTING"
ChainNATOutput = "OUTPUT"

ChainNATKubeMarkMasq = "KUBE-MARK-MASQ"

ChainFilterBaseInput = "NMANAGER_INPUT"
ChainNATBasePrerouting = "NMANAGER_PREROUTING"
ChainNATBaseOutput = "NMANAGER_OUTPUT"

ChainFilterDropInvalidInput = "NMANAGER_DROP_INVALID_INPUT"
ChainNATExternalClusterPrerouting = "NMANAGER_EX_CLUS_PREROUTING"
ChainNATExternalClusterOutput = "NMANAGER_EX_CLUS_OUTPUT"
)

func initBaseChains(logger logr.Logger) error {
// Get network stack config
configIPv4Enabled, configIPv6Enabled, err := configs.GetConfigNetStack()
if err != nil {
return err
}

// IPv4
if configIPv4Enabled {
// Create base chain in tables
out, err := iptables.CreateChainIPv4(iptables.TableFilter, ChainFilterBaseInput)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv4(iptables.TableNAT, ChainNATBasePrerouting)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv4(iptables.TableNAT, ChainNATBaseOutput)
if err != nil {
logger.Error(err, out)
return err
}

// Create jump rule to each chain in tables
ruleJumpFilterInput := []string{"-j", ChainFilterBaseInput}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableFilter, ChainFilterInput, "", ruleJumpFilterInput...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpNATPre := []string{"-j", ChainNATBasePrerouting}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableNAT, ChainNATPrerouting, "", ruleJumpNATPre...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpNATOut := []string{"-j", ChainNATBaseOutput}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableNAT, ChainNATOutput, "", ruleJumpNATOut...)
if err != nil {
logger.Error(err, out)
return err
}

}

// IPv6
if configIPv6Enabled {
// Create base chain in nat table
out, err := iptables.CreateChainIPv6(iptables.TableFilter, ChainFilterBaseInput)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv6(iptables.TableNAT, ChainNATBasePrerouting)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv6(iptables.TableNAT, ChainNATBaseOutput)
if err != nil {
logger.Error(err, out)
return err
}

// Create jump rule to each chain in nat table
ruleJumpFilterInput := []string{"-j", ChainFilterBaseInput}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableFilter, ChainFilterInput, "", ruleJumpFilterInput...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpNATPre := []string{"-j", ChainNATBasePrerouting}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableNAT, ChainNATPrerouting, "", ruleJumpNATPre...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpNATOut := []string{"-j", ChainNATBaseOutput}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableNAT, ChainNATOutput, "", ruleJumpNATOut...)
if err != nil {
logger.Error(err, out)
return err
}
}

return nil
}
108 changes: 108 additions & 0 deletions controllers/rule_drop_invalid_input.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
package controllers

import (
"github.com/go-logr/logr"

"github.com/kakao/network-node-manager/pkg/iptables"
)

func createRulesDropInvalidInput(logger logr.Logger) error {
if err := initBaseChains(logger); err != nil {
logger.Error(err, "failed to init base chain for externalIP to clusterIP Rules")
return err
}

// IPv4
if configIPv4Enabled {
// Create chain
out, err := iptables.CreateChainIPv4(iptables.TableFilter, ChainFilterDropInvalidInput)
if err != nil {
logger.Error(err, out)
return err
}

// Set drop rule
ruleDrop := []string{"-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP"}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableFilter, ChainFilterDropInvalidInput, "", ruleDrop...)
if err != nil {
logger.Error(err, out)
return err
}

// Set jump rule
ruleJump := []string{"-j", ChainFilterDropInvalidInput}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableFilter, ChainFilterBaseInput, "", ruleJump...)
if err != nil {
logger.Error(err, out)
return err
}
}

// IPv6
if configIPv6Enabled {
// Create chain
out, err := iptables.CreateChainIPv6(iptables.TableFilter, ChainFilterDropInvalidInput)
if err != nil {
logger.Error(err, out)
return err
}

// Set drop rule
ruleDrop := []string{"-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP"}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableFilter, ChainFilterDropInvalidInput, "", ruleDrop...)
if err != nil {
logger.Error(err, out)
return err
}

// Set jump rule
ruleJump := []string{"-j", ChainFilterDropInvalidInput}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableFilter, ChainFilterBaseInput, "", ruleJump...)
if err != nil {
logger.Error(err, out)
return err
}
}

return nil
}

func deleteRulesDropInvalidInput(logger logr.Logger) error {
// IPv4
if configIPv4Enabled {
// Delete jump rule
ruleJump := []string{"-j", ChainFilterDropInvalidInput}
out, err := iptables.DeleteRuleIPv4(iptables.TableFilter, ChainFilterBaseInput, "", ruleJump...)
if err != nil {
logger.Error(err, out)
return err
}

// Delete chain
out, err = iptables.DeleteChainIPv4(iptables.TableFilter, ChainFilterDropInvalidInput)
if err != nil {
logger.Error(err, out)
return err
}
}

// IPv6
if configIPv6Enabled {
// Delete jump rule
ruleJump := []string{"-j", ChainFilterDropInvalidInput}
out, err := iptables.DeleteRuleIPv6(iptables.TableFilter, ChainFilterBaseInput, "", ruleJump...)
if err != nil {
logger.Error(err, out)
return err
}

// Delete chain
out, err = iptables.DeleteChainIPv6(iptables.TableFilter, ChainFilterDropInvalidInput)
if err != nil {
logger.Error(err, out)
return err
}
}

return nil
}
Loading

0 comments on commit a6e8842

Please sign in to comment.