Skip to content

Commit

Permalink
rename project to network-node-manager
Browse files Browse the repository at this point in the history
  • Loading branch information
ssup2 committed Nov 15, 2020
1 parent fb69d9a commit d81112f
Show file tree
Hide file tree
Showing 9 changed files with 70 additions and 69 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: Prepare
id: prepare
run: |
DOCKER_IMAGE=kakaocorp/ipvs-node-controller
DOCKER_IMAGE=kakaocorp/network-node-manager
DOCKER_PLATFORMS=linux/amd64,linux/arm64
VERSION=latest
Expand Down
10 changes: 5 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Build the ipvs-node-controller binary
# Build the network-node-manager binary
FROM golang:1.13 as builder

WORKDIR /workspace
Expand All @@ -11,13 +11,13 @@ COPY main.go main.go
COPY controllers/ controllers/
COPY pkg/ pkg/

# Build ipvs-node-controller
RUN CGO_ENABLED=0 GO111MODULE=on go build -a -o ipvs-node-controller main.go
# Build network-node-manager
RUN CGO_ENABLED=0 GO111MODULE=on go build -a -o network-node-manager main.go

# Build image
FROM alpine:3.11.6
RUN apk add iptables=1.8.3-r2 ip6tables=1.8.3-r2

WORKDIR /
COPY --from=builder /workspace/ipvs-node-controller .
ENTRYPOINT ["/ipvs-node-controller"]
COPY --from=builder /workspace/network-node-manager .
ENTRYPOINT ["/network-node-manager"]
8 changes: 4 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ else
GOBIN=$(shell go env GOBIN)
endif

all: ipvs-node-controller
all: network-node-manager

# Run tests
test: fmt vet
go test ./... -coverprofile cover.out

# Build ipvs-node-controller binary
ipvs-node-controller: fmt vet
go build -o bin/ipvs-node-controller main.go
# Build network-node-manager binary
network-node-manager: fmt vet
go build -o bin/network-node-manager main.go

# Run against the configured Kubernetes cluster in ~/.kube/config
run: fmt vet
Expand Down
2 changes: 1 addition & 1 deletion NOTICE.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# OSS Notice | ipvs-node-controller #
# OSS Notice | network-node-manager #

This application is Copyright © Kakao Corp. All rights reserved.

Expand Down
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
# IPVS Node Controller
# Network Node Manager

ipvs-node-controller is the kubernetes controller that solves External-IP (Load Balancer IP) issue with IPVS proxy mode. IPVS proxy mode has various problems, and one of them is that the External-IP assigned through the LoadBalancer type service with externalTrafficPolicy=Local option cannot access inside the cluster. More details on this issue can be found at [here](https://github.com/kubernetes/kubernetes/issues/75262). ipvs-node-controller solves this issue. ipvs-node-controller is based on [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder).
network-node-manager is the kubernetes controller that solves External-IP (Load Balancer IP) issue with IPVS proxy mode. IPVS proxy mode has various problems, and one of them is that the External-IP assigned through the LoadBalancer type service with externalTrafficPolicy=Local option cannot access inside the cluster. More details on this issue can be found at [here](https://github.com/kubernetes/kubernetes/issues/75262). network-node-manager solves this issue. network-node-manager is based on [kubebuilder](https://github.com/kubernetes-sigs/kubebuilder).

## Deploy

ipvs-node-controller now supports below CPU architectures.
network-node-manager now supports below CPU architectures.

* amd64
* arm64

Deploy ipvs-node-controllers through below command.
Deploy network-node-managers through below command.

```
kubectl apply -f https://raw.githubusercontent.com/kakao/ipvs-node-controller/master/deploy/ipvs-node-controller.yml
kubectl apply -f https://raw.githubusercontent.com/kakao/network-node-manager/master/deploy/network-node-manager.yml
```

## Configuration

### IPv6

ipvs-node-controller supports IPv6. However, IPv6 is not enabled by default. To use IPv6, set "NET_STACK" environment in the DaemonSet manifests of ipvs-node-controller as follows.
network-node-manager supports IPv6. However, IPv6 is not enabled by default. To use IPv6, set "NET_STACK" environment in the DaemonSet manifests of network-node-manager as follows.

```
...
Expand All @@ -33,7 +33,7 @@ env:

## How it works?

ipvs-node-controller works on all worker nodes and adds the DNAT rules that converts destination IP of a packet from External-IP to Cluster-IP to iptables. ipvs-node-controller adds two DNAT rules for each LoadBalancer type service. One is added to the prerouting chain and the other is added to the output chain. The DNAT rule in the prerouting chain is for the pod that uses pod-only network namespace. On the other hand, The DNAT rule in the output chain is for the pod that uses host network namespace. All DNAT rules only target packets from pods on the host. Below is an example.
network-node-manager works on all worker nodes and adds the DNAT rules that converts destination IP of a packet from External-IP to Cluster-IP to iptables. network-node-manager adds two DNAT rules for each LoadBalancer type service. One is added to the prerouting chain and the other is added to the output chain. The DNAT rule in the prerouting chain is for the pod that uses pod-only network namespace. On the other hand, The DNAT rule in the output chain is for the pod that uses host network namespace. All DNAT rules only target packets from pods on the host. Below is an example.

```
$ kubectl -n default get service
Expand All @@ -43,14 +43,14 @@ lb-service-2 LoadBalancer 10.231.2.62 10.19.22.57 80:32352/TCP,443:31
$ iptables -nvL -t nat
...
Chain IPVS_OUTPUT (1 references)
Chain NMANAGER_IPVS_LB_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 10.19.20.201 /* default/lb-service-1 */ ADDRTYPE match src-type LOCAL
0 0 DNAT all -- * * 0.0.0.0/0 10.19.20.201 /* default/lb-service-1 */ ADDRTYPE match src-type LOCAL to:10.231.42.164
2 120 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 10.19.22.57 /* default/lb-service-2 */ ADDRTYPE match src-type LOCAL
2 120 DNAT all -- * * 0.0.0.0/0 10.19.22.57 /* default/lb-service-2 */ ADDRTYPE match src-type LOCAL to:10.231.2.62
Chain IPVS_PREROUTING (1 references)
Chain NMANAGER_IPVS_LB_PREROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.240.2.128/25 10.19.20.201 /* default/lb-service-1 */
0 0 DNAT all -- * * 10.240.2.128/25 10.19.20.201 /* default/lb-service-1 */ to:10.231.42.164
Expand Down
67 changes: 34 additions & 33 deletions controllers/service_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"

"github.com/go-logr/logr"
"github.com/kakao/ipvs-node-controller/pkg/configs"
"github.com/kakao/ipvs-node-controller/pkg/ip"
"github.com/kakao/ipvs-node-controller/pkg/iptables"
"github.com/kakao/network-node-manager/pkg/configs"
"github.com/kakao/network-node-manager/pkg/ip"
"github.com/kakao/network-node-manager/pkg/iptables"
)

// ServiceReconciler reconciles a Service object
Expand All @@ -47,8 +47,9 @@ const (
ChainNATPrerouting = "PREROUTING"
ChainNATOutput = "OUTPUT"
ChainNATKubeMasquerade = "KUBE-MARK-MASQ"
ChainNATIPVSPrerouting = "IPVS_PREROUTING"
ChainNATIPVSOutput = "IPVS_OUTPUT"

ChainNATIPVSLBPrerouting = "NMANAGER_IPVS_LB_PREROUTING"
ChainNATIPVSLBOutput = "NMANAGER_IPVS_LB_OUTPUT"
)

// Variables
Expand Down Expand Up @@ -196,26 +197,26 @@ func initIptables(logger logr.Logger) error {
if configIPv4Enabled {
// Create chain in nat table
logger.Info("create the IPVS IPv4 chains")
out, err := iptables.CreateChainIPv4(iptables.TableNAT, ChainNATIPVSPrerouting)
out, err := iptables.CreateChainIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv4(iptables.TableNAT, ChainNATIPVSOutput)
out, err = iptables.CreateChainIPv4(iptables.TableNAT, ChainNATIPVSLBOutput)
if err != nil {
logger.Error(err, out)
return err
}

// Set jump rule to each chain in nat table
logger.Info("create jump rules for the IPVS IPv4 chains")
ruleJumpPre := []string{"-j", ChainNATIPVSPrerouting}
ruleJumpPre := []string{"-j", ChainNATIPVSLBPrerouting}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableNAT, ChainNATPrerouting, "", ruleJumpPre...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpOut := []string{"-j", ChainNATIPVSOutput}
ruleJumpOut := []string{"-j", ChainNATIPVSLBOutput}
out, err = iptables.CreateRuleFirstIPv4(iptables.TableNAT, ChainNATOutput, "", ruleJumpOut...)
if err != nil {
logger.Error(err, out)
Expand All @@ -226,26 +227,26 @@ func initIptables(logger logr.Logger) error {
if configIPv6Enabled {
// Create chain in nat table
logger.Info("create the IPVS IPv6 chains")
out, err := iptables.CreateChainIPv6(iptables.TableNAT, ChainNATIPVSPrerouting)
out, err := iptables.CreateChainIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting)
if err != nil {
logger.Error(err, out)
return err
}
out, err = iptables.CreateChainIPv6(iptables.TableNAT, ChainNATIPVSOutput)
out, err = iptables.CreateChainIPv6(iptables.TableNAT, ChainNATIPVSLBOutput)
if err != nil {
logger.Error(err, out)
return err
}

// Set jump rule to each chain in nat table
logger.Info("create jump rules for the IPVS IPv6 chains")
ruleJumpPre := []string{"-j", ChainNATIPVSPrerouting}
ruleJumpPre := []string{"-j", ChainNATIPVSLBPrerouting}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableNAT, ChainNATPrerouting, "", ruleJumpPre...)
if err != nil {
logger.Error(err, out)
return err
}
ruleJumpOut := []string{"-j", ChainNATIPVSOutput}
ruleJumpOut := []string{"-j", ChainNATIPVSLBOutput}
out, err = iptables.CreateRuleFirstIPv6(iptables.TableNAT, ChainNATOutput, "", ruleJumpOut...)
if err != nil {
logger.Error(err, out)
Expand All @@ -270,7 +271,7 @@ func cleanupIptables(logger logr.Logger, svcs *corev1.ServiceList, podCIDRIPv4,
}

// Cleanup prerouting chain
preRules, err := iptables.GetRulesIPv4(iptables.TableNAT, ChainNATIPVSPrerouting)
preRules, err := iptables.GetRulesIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting)
if err != nil {
return err
}
Expand Down Expand Up @@ -303,7 +304,7 @@ func cleanupIptables(logger logr.Logger, svcs *corev1.ServiceList, podCIDRIPv4,
}

// Cleanup output chain
outRules, err := iptables.GetRulesIPv4(iptables.TableNAT, ChainNATIPVSOutput)
outRules, err := iptables.GetRulesIPv4(iptables.TableNAT, ChainNATIPVSLBOutput)
if err != nil {
return err
}
Expand Down Expand Up @@ -343,7 +344,7 @@ func cleanupIptables(logger logr.Logger, svcs *corev1.ServiceList, podCIDRIPv4,
}

// Cleanup prerouting chain
preRules, err := iptables.GetRulesIPv6(iptables.TableNAT, ChainNATIPVSPrerouting)
preRules, err := iptables.GetRulesIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting)
if err != nil {
return err
}
Expand Down Expand Up @@ -376,7 +377,7 @@ func cleanupIptables(logger logr.Logger, svcs *corev1.ServiceList, podCIDRIPv4,
}

// Cleanup output
outRules, err := iptables.GetRulesIPv6(iptables.TableNAT, ChainNATIPVSOutput)
outRules, err := iptables.GetRulesIPv6(iptables.TableNAT, ChainNATIPVSLBOutput)
if err != nil {
return err
}
Expand Down Expand Up @@ -413,27 +414,27 @@ func createIptablesRules(logger logr.Logger, req *ctrl.Request, clusterIP, exter
// IPv4
// Set prerouting
rulePreMasq := []string{"-s", podCIDRIPv4, "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err := iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreMasq...)
out, err := iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreMasq...)
if err != nil {
logger.Error(err, out)
return err
}
rulePreDNAT := []string{"-s", podCIDRIPv4, "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreDNAT...)
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreDNAT...)
if err != nil {
logger.Error(err, out)
return err
}

// Set output
ruleOutMasq := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutMasq...)
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutMasq...)
if err != nil {
logger.Error(err, out)
return err
}
ruleOutDNAT := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutDNAT...)
out, err = iptables.CreateRuleLastIPv4(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutDNAT...)
if err != nil {
logger.Error(err, out)
return err
Expand All @@ -442,27 +443,27 @@ func createIptablesRules(logger logr.Logger, req *ctrl.Request, clusterIP, exter
// IPv6
// Set prerouting
rulePreMasq := []string{"-s", podCIDRIPv6, "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err := iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreMasq...)
out, err := iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreMasq...)
if err != nil {
logger.Error(err, out)
return err
}
rulePreDNAT := []string{"-s", podCIDRIPv6, "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreDNAT...)
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreDNAT...)
if err != nil {
logger.Error(err, out)
return err
}

// Set output
ruleOutMasq := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutMasq...)
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutMasq...)
if err != nil {
logger.Error(err, out)
return err
}
ruleOutDNAT := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutDNAT...)
out, err = iptables.CreateRuleLastIPv6(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutDNAT...)
if err != nil {
logger.Error(err, out)
return err
Expand All @@ -483,27 +484,27 @@ func deleteIptablesRules(logger logr.Logger, req *ctrl.Request, clusterIP, exter
// IPv4
// Unset prerouting
rulePreMasq := []string{"-s", podCIDRIPv4, "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err := iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreMasq...)
out, err := iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreMasq...)
if err != nil {
logger.Error(err, out)
return err
}
rulePreDNAT := []string{"-s", podCIDRIPv4, "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreDNAT...)
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreDNAT...)
if err != nil {
logger.Error(err, out)
return err
}

// Unset output
ruleOutMasq := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutMasq...)
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutMasq...)
if err != nil {
logger.Error(err, out)
return err
}
ruleOutDNAT := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutDNAT...)
out, err = iptables.DeleteRuleIPv4(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutDNAT...)
if err != nil {
logger.Error(err, out)
return err
Expand All @@ -512,27 +513,27 @@ func deleteIptablesRules(logger logr.Logger, req *ctrl.Request, clusterIP, exter
// IPv6
// Unset prerouting
rulePreMasq := []string{"-s", podCIDRIPv6, "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err := iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreMasq...)
out, err := iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreMasq...)
if err != nil {
logger.Error(err, out)
return err
}
rulePreDNAT := []string{"-s", podCIDRIPv6, "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSPrerouting, req.String(), rulePreDNAT...)
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSLBPrerouting, req.String(), rulePreDNAT...)
if err != nil {
logger.Error(err, out)
return err
}

// Unset output
ruleOutMasq := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", ChainNATKubeMasquerade}
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutMasq...)
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutMasq...)
if err != nil {
logger.Error(err, out)
return err
}
ruleOutDNAT := []string{"-m", "addrtype", "--src-type", "LOCAL", "-d", externalIP, "-j", "DNAT", "--to-destination", clusterIP}
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSOutput, req.String(), ruleOutDNAT...)
out, err = iptables.DeleteRuleIPv6(iptables.TableNAT, ChainNATIPVSLBOutput, req.String(), ruleOutDNAT...)
if err != nil {
logger.Error(err, out)
return err
Expand Down
Loading

0 comments on commit d81112f

Please sign in to comment.