First draft of forbidden mode for isolated clusters #172

majst01 · 2023-12-18T13:24:13Z

Related to: https://github.com/fi-ts/proxy-services/issues/4

Todos:

refactoring regarding code additions
ignore violations
send violations as event logs
update shoot client controller status
forbid service type loadbalancer outside of allowedNetworks in forbidden mode
tests

If allowed networks have been set up, we enforce all cwnps to be included. Otherwise we want them to be deleted. Use case: network isolated clusters with internet access forbidden may not add conflicting cwnps. If they somehow do exist like after migration, we reflect the changes by deletion.

majst01 · 2024-01-15T12:57:31Z

rewall monitor successfully updated, requeuing in 10s","name":"shoot--pcfgbt--forbidden-firewall-ad19e","namespace":"firewall"}                                                                                                               
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: {"level":"info","timestamp":"2024-01-15T13:57:08+01:00","caller":"controller/controller.go:118","msg":"Observed a panic in reconciler: runtime error: invalid memory address
 or nil pointer dereference","controller":"clusterwidenetworkpolicy","controllerGroup":"metal-stack.io","controllerKind":"ClusterwideNetworkPolicy","ClusterwideNetworkPolicy":{"name":"allow-to-forbidden","namespace":"firewall"},"namespace
":"firewall","name":"allow-to-forbidden","reconcileID":"2af9ae73-266a-4a63-a55d-cf8a1b0f6b49"}                                                                                                                                                
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]                                                                                         
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         panic: runtime error: invalid memory address or nil pointer dereference                                                                                             
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x168b6cd]                                                                                                    
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: goroutine 774 [running]:                                                                                                                                                    
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()                                                                                      
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0x1e5                                                                             
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: panic({0x1810a20?, 0x29dc300?})                                                                                                                                             
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         runtime/panic.go:914 +0x21f                                                                                                                                         
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: github.com/metal-stack/firewall-controller/v2/controllers.(*ClusterwideNetworkPolicyReconciler).validateCWNPEgressTargetPrefix(_, {{{0x16e22be, 0x18}, {0xc0007eb248, 0x11}}
, {{0xc0007eb1b8, 0x12}, {0x0, 0x0}, {0xc000d165b8, ...}, ...}, ...}, ...)                                                                                                                                                                    
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         github.com/metal-stack/firewall-controller/v2/controllers/clusterwidenetworkpolicy_controller.go:283 +0x30d                                                         Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: github.com/metal-stack/firewall-controller/v2/controllers.(*ClusterwideNetworkPolicyReconciler).allowedCWNPsOrDelete(0xc000281c00, {0x1cee5e8, 0xc000c8ed50}, {0xc00110aa80?
, 0x7, 0xc0011fe000?}, {0xc000d851a0?, 0x53d75a?}, {{0xc000e0e040, 0x1, ...}, ...})                                                                                                                                                           
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         github.com/metal-stack/firewall-controller/v2/controllers/clusterwidenetworkpolicy_controller.go:222 +0x24e                                                         Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: github.com/metal-stack/firewall-controller/v2/controllers.(*ClusterwideNetworkPolicyReconciler).Reconcile(0xc000281c00, {0x1cee5e8, 0xc000c8ed50}, {{{0xc000c8ed50?, 0x0?}, 
{0xc000bc5d20?, 0x4105a5?}}})                                                                                                                                                                                                                 
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         github.com/metal-stack/firewall-controller/v2/controllers/clusterwidenetworkpolicy_controller.go:101 +0x345                                                         Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1cee5e8?, {0x1cee5e8?, 0xc000c8ed50?}, {{{0xc000d165b8?, 0x1756560?}, {0xc0007eb1b8?, 0x1ce
0108?}}})                                                                                                                                                                                                                                     
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122 +0xb7                                                             
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000621360, {0x1cee620, 0xc0005730e0}, {0x1898200?, 0xc000a9f3c0?})
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323 +0x368                                  
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000621360, {0x1cee620, 0xc0005730e0})
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274 +0x1c9             
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()                        
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235 +0x79              
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]: created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 643
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e ip[2052]:         sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231 +0x565
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e systemd[1]: firewall-controller.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 15 13:57:08 shoot--pcfgbt--forbidden-firewall-ad19e systemd[1]: firewall-controller.service: Failed with result 'exit-code'.

majst01 · 2024-01-18T12:25:33Z

Error is weird:

3s          Warning   ForbiddenCIDR          service/nginx-gardener   the specified of "nginx-gardener" to address:"212.34.83.6/32" is outside of the allowed network range:"100.64.0.0/10", ignoring

…rewall-controller into isolated-clusters

controllers/clusterwidenetworkpolicy_controller.go

pkg/nftables/rendering.go

config/crd/bases/metal-stack.io_clusterwidenetworkpolicies.yaml

controllers/clusterwidenetworkpolicy_controller.go

Co-authored-by: Markus Wennrich <[email protected]>

go.mod

First draft of forbidden mode

fbcfbdb

majst01 assigned vknabel Dec 18, 2023

vknabel and others added 6 commits December 18, 2023 16:01

style: golangci-lint

db2b425

refactor(cwnp): extract cwnp validation and deletion

fd662cd

Merge branch 'master' into isolated-clusters

7babed6

feat: bump fwc-manager

15171b6

new structure for allowed networks

6035e72

ulrichSchreiner and others added 3 commits January 15, 2024 14:08

set missing recorder

55ee37d

Hint

d79151e

filter IP of loadbalancer when networkisolation is forbidden

4be5a59

ulrichSchreiner and others added 9 commits January 18, 2024 13:50

fix weird event logging message

27e2297

Only reduce cwnps for accessType forbidden

0d8abb2

Merge branch 'isolated-clusters' of https://github.com/metal-stack/fi…

c182db1

…rewall-controller into isolated-clusters

Only reduce cwnps for accessType forbidden

c49c38c

Update actions

dc523cd

Update metal-networker to get block forward

50cad61

Back to released metal-networker

56be4e2

Change metal table default drop in forward chain to accept after start

6f129de

Fix drop/accept forwarding

f2bdf2e

vknabel reviewed Jan 23, 2024

View reviewed changes

controllers/clusterwidenetworkpolicy_controller.go Outdated Show resolved Hide resolved

majst01 added 2 commits January 23, 2024 09:26

Pin metal-networker

bc963e9

Merge master

28ead28

majst01 marked this pull request as ready for review January 23, 2024 12:15

majst01 requested a review from a team as a code owner January 23, 2024 12:15

Updates

ed04054

majst01 commented Jan 23, 2024

View reviewed changes

Remove ugly code

91d952a

majst01 and others added 9 commits January 23, 2024 15:48

Prepare cwnp state

18bbe78

resolve review comments

035975e

add printer columns for state and message for cwnp

78ba051

update real status not a copy of it

12d70b1

always update; caller checks if something changed

cb21dd1

remove ugly code

c959da9

make state not required

ac01058

check for old value and only update if new values differ

3bc65fc

Update pkg/nftables/rendering.go

6e474ee

majst01 requested review from mwennrich and Gerrit91 January 29, 2024 12:16

Add api docs

564e30e

mwennrich reviewed Jan 29, 2024

View reviewed changes

config/crd/bases/metal-stack.io_clusterwidenetworkpolicies.yaml Outdated Show resolved Hide resolved

controllers/clusterwidenetworkpolicy_controller.go Outdated Show resolved Hide resolved

majst01 and others added 3 commits January 29, 2024 14:55

Update config/crd/bases/metal-stack.io_clusterwidenetworkpolicies.yaml

5b42cfb

Co-authored-by: Markus Wennrich <[email protected]>

Update controllers/clusterwidenetworkpolicy_controller.go

388e808

Co-authored-by: Markus Wennrich <[email protected]>

typo

02b097c

mwennrich approved these changes Jan 30, 2024

View reviewed changes

majst01 commented Jan 30, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Gerrit91 added 2 commits January 30, 2024 11:39

Pin.

252107a

Don't forget to save.

368a0cd

Gerrit91 approved these changes Jan 30, 2024

View reviewed changes

majst01 merged commit 118b30e into master Jan 30, 2024
2 checks passed

majst01 deleted the isolated-clusters branch January 30, 2024 10:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First draft of forbidden mode for isolated clusters #172

First draft of forbidden mode for isolated clusters #172

majst01 commented Dec 18, 2023 •

edited

Loading

majst01 commented Jan 15, 2024

majst01 commented Jan 18, 2024

First draft of forbidden mode for isolated clusters #172

First draft of forbidden mode for isolated clusters #172

Conversation

majst01 commented Dec 18, 2023 • edited Loading

majst01 commented Jan 15, 2024

majst01 commented Jan 18, 2024

majst01 commented Dec 18, 2023 •

edited

Loading