L7 Network Flow Export support in Antrea #5218

tushartathgur · 2023-07-06T22:04:53Z

L7 Network flow export enables Layer 7 flow export as we did for the l3/l4 earlier.
This feature enables the user to export L7 protocol information using:

Pod or namespace Annotations

Annotation key used is "visibility.antrea.io/l7-export" and the value is the direction for which the flow export is required, which could be (ingress/egress/both).
Based of the annotation or the NP, the fields ("appProtocolName", "httpVals") are populated and exported.

### - Design:

L7FlowExport is to be designed in order to receive all the protected and unprotected flows of Layer7 type and to be exported.
In order to complete this feature, we need the following flows.

L7FlowExport using the Annotations.
- The flows that are not part of the L7 NP, can also be exported using the pods and Namespace annotations.
- The annotation key used here is "visibility.antrea.io/l7-export".
- User can annotate pods or namespaces with the annotation key and a value for the traffic flow direction from the following (Ingress/Egress/both).
- By default, there will not be any L7 Flow information received by the exporter, i.e. none of the pods or namespaces are annotated by default.
- Based on the annotation, the L7 Listener will receive the pod or namespace event. L7 Listener would use localPodInformers and namespace informers for this task.
- Once the correct annotation key-value is received, the pod or pods in an annotated namespace would mirror their traffic to Suricata using Traffic control mark flows
- Suricata will send the traffic flows information of these mirrored traffic. We currently only support HTTP traffic flows and thus these flows will be the only flows from Suricata to the exporter.
- Exporter will then populate the necessary fields and export them to the exporters.

Deletion of the flows.

Similar to install of new Traffic control mark flows. The deletion of the mark flows will also be taken care of based of two scenarios.
User removes the annotations.
User deletes the pod or namespace. Only the ready pods or ready pods inside an annotated Namespace will have the corresponding traffic flows.
Once any of the two scenarios are met. The corresponding Traffic mark flow will be removed.

New Fields to be added in the exporter.

appProtocolName: This field will help export only the l7 flows if required. It is used to also take care of future L7 protocols implementations which may require adding new fields and "appProtocolName" to provide the differentiating field for L7 flows from L3/L4 flows.
HttpVals: This field is used and should contain all the http fields (in order) corresponding to a network flow. The fields are:
[Hostname, URL, UserAgent, ContentType, Method, Protocol, Status, ContentLength]

** GO-IPFIX changes are present in vmware/go-ipfix#315

pkg/agent/controller/networkpolicy/l7engine/l7visbility_watcher.go

pkg/agent/flowexporter/exporter/exporter.go

pkg/agent/controller/networkpolicy/l7engine/l7visbility_watcher.go

test/e2e/flowaggregator_test.go

antoninbas · 2024-01-18T19:01:19Z

pkg/agent/flowexporter/connections/conntrack_connections.go

+	for connKey, conn := range cs.connections {
+		l7event, ok := l7EventMap[connKey]


the placement of the second part of the comment is strange:

In case the L7 event is received later than the connkey become unavailable in the cs.connection, we will discard that event

When this happens, we never get to this part of the code, because the for loop (for connKey, conn := range cs.connections) only considers connections in the store. So maybe that second sentence should go before the loop?

antoninbas · 2024-01-18T19:01:53Z

pkg/agent/flowexporter/connections/conntrack_connections.go

+			}
+			// In case L7 event is received after the last planned export of the TCP connection, add
+			// the event back to the queue to be exported in next export cycle. In case the L7 event
+			// is received later than the connkey become unavailable in the cs.connection, we will


...is received after the connection is removed from the cs.connections store...

antoninbas · 2024-01-18T19:06:51Z

test/e2e/flowaggregator_test.go

@@ -1865,6 +1879,67 @@ func getAndCheckFlowAggregatorMetrics(t *testing.T, data *TestData) error {
 	return nil
 }

+func testL7FlowExporterController(t *testing.T, data *TestData, isIPv6 bool) {
+	skipIfFeatureDisabled(t, features.L7FlowExporter, true, false)
+	_, serverIPs, cleanupFunc := createAndWaitForPod(t, data, data.createNginxPodOnNode, "l7flowexportertestpodserver", nodeName(0), data.testNamespace, false)


nit: define nodeName := nodeName(0) at the beginning of the function and use this throughout the test case

antoninbas · 2024-01-18T19:08:10Z

@yuntanghsu could you take one more look at the new e2e test?

antoninbas · 2024-01-18T19:23:20Z

pkg/agent/flowexporter/connections/conntrack_connections.go

@@ -325,3 +336,27 @@ func (cs *ConntrackConnectionStore) deleteConnWithoutLock(connKey flowexporter.C
 func (cs *ConntrackConnectionStore) GetPriorityQueue() *priorityqueue.ExpirePriorityQueue {
 	return cs.connectionStore.expirePriorityQueue
 }
+
+func (cs *ConntrackConnectionStore) fillL7EventInfo(l7EventMap map[flowexporter.Tuple]L7ProtocolFields) {
+	// In case the L7 event is received after the connection is removed from the cs.connections store.


s/store./store,

test/e2e/flowaggregator_test.go

yuntanghsu · 2024-01-18T19:20:17Z

test/e2e/flowaggregator_test.go

+func checkL7FlowExporterData(t *testing.T, record, appProtocolName string) {
+	assert.Containsf(t, record, fmt.Sprintf("appProtocolName: %s", appProtocolName), "Record does not have correct Layer 7 protocol Name")
+}
+
+func checkL7FlowExporterDataClickHouse(t *testing.T, record *ClickHouseFullRow, appProtocolName string) {
+	assert.Equal(t, record.AppProtocolName, appProtocolName, "Record does not have correct Layer 7 protocol Name")
+	assert.NotEmpty(t, record.HttpVals, "Record does not have httpVals")
+}


Do we need these two functions? I think they are only used once and the value of appProtocolName is always http?

the two functions are for two different records, one for click house other one for collector

Yes. But each one of them is used only once. I was thinking maybe it's better to add line#1402 to line#1927 and line#1406-1407 to line#1939

Sure , we can do that, however, this helps in future when we may support other protocols

let's keep this as it is, this PR has been pending for a while

test/e2e/flowaggregator_test.go

yuntanghsu · 2024-01-18T19:37:16Z

test/e2e/flowaggregator_test.go

+			assert.Contains(record, testFlow1.srcPodName, "Record with srcIP does not have Pod name: %s", testFlow1.srcPodName)
+			assert.Contains(record, fmt.Sprintf("sourcePodNamespace: %s", data.testNamespace), "Record does not have correct sourcePodNamespace: %s", data.testNamespace)
+			assert.Contains(record, fmt.Sprintf("sourceNodeName: %s", nodeName), "Record does not have correct sourceNodeName: %s", nodeName)


I think you can reuse checkPodAndNodeData
Like, checkPodAndNodeData(t, record, testFlow1.srcPodName, nodeName, "", "", data.testNamespace)

those functions are specific to perftest labels, we may add a general function later!

test/e2e/flowaggregator_test.go

L7 Network flow export enables Layer 7 flow export as we did for the l3/l4 earlier. This feature enables the user to export L7 protocol information using: - Pod or namespace Annotations Annotation key used is "visibility.antrea.io/l7-export" and the value is the direction for which the flow export is required, which could be (ingress/egress/both). Based of the annotation or the NP, the fields ("appProtocolName", "httpVals") are populated and exported. Signed-off-by: Tushar Tathgur <[email protected]>

tnqn

LGTM

antoninbas · 2024-01-19T17:40:57Z

/test-all

tnqn · 2024-01-22T09:26:38Z

The new e2e test failed

Signed-off-by: Tushar Tathgur <[email protected]>

tnqn

LGTM

tnqn · 2024-01-23T11:28:57Z

/test-all

tushartathgur marked this pull request as draft July 6, 2023 22:05

tushartathgur force-pushed the L7_visibility branch 3 times, most recently from 98d13da to 6cc14bb Compare July 20, 2023 04:36

elton-furtado requested a review from yuntanghsu July 25, 2023 18:37

elton-furtado added this to the Antrea v1.14 release milestone Jul 26, 2023

tushartathgur force-pushed the L7_visibility branch 8 times, most recently from 8a11a69 to b8555a2 Compare August 3, 2023 19:41

tushartathgur force-pushed the L7_visibility branch 7 times, most recently from 57fb343 to 2799512 Compare August 4, 2023 00:01

yuntanghsu reviewed Aug 4, 2023

View reviewed changes

tushartathgur force-pushed the L7_visibility branch 6 times, most recently from ad1eb60 to 56919fe Compare August 7, 2023 23:42

tushartathgur marked this pull request as ready for review August 8, 2023 02:43

tushartathgur force-pushed the L7_visibility branch from 56919fe to 8453bcf Compare August 8, 2023 17:48

tushartathgur force-pushed the L7_visibility branch from ab34f38 to 1f4f8d1 Compare January 17, 2024 22:31

yuntanghsu reviewed Jan 17, 2024

View reviewed changes

test/e2e/flowaggregator_test.go Show resolved Hide resolved

tushartathgur force-pushed the L7_visibility branch 6 times, most recently from bb15a54 to cf33369 Compare January 18, 2024 12:20

antoninbas reviewed Jan 18, 2024

View reviewed changes

tushartathgur force-pushed the L7_visibility branch from cf33369 to c2b6b31 Compare January 18, 2024 19:17

antoninbas reviewed Jan 18, 2024

View reviewed changes

tushartathgur force-pushed the L7_visibility branch from c2b6b31 to 528f774 Compare January 18, 2024 19:25

antoninbas previously approved these changes Jan 18, 2024

View reviewed changes

yuntanghsu reviewed Jan 18, 2024

View reviewed changes

tushartathgur dismissed antoninbas’s stale review via 129529b January 18, 2024 19:53

tushartathgur force-pushed the L7_visibility branch from 528f774 to 129529b Compare January 18, 2024 19:53

tushartathgur force-pushed the L7_visibility branch from 129529b to 2c0a3bb Compare January 19, 2024 02:24

tnqn previously approved these changes Jan 19, 2024

View reviewed changes

antoninbas previously approved these changes Jan 19, 2024

View reviewed changes

Addressed new comments

9dfc6f1

Signed-off-by: Tushar Tathgur <[email protected]>

tushartathgur dismissed stale reviews from antoninbas and tnqn via 9dfc6f1 January 23, 2024 09:58

tushartathgur force-pushed the L7_visibility branch from 2c0a3bb to 9dfc6f1 Compare January 23, 2024 09:58

tnqn approved these changes Jan 23, 2024

View reviewed changes

tnqn merged commit f49a4ea into antrea-io:main Jan 24, 2024
46 of 53 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

L7 Network Flow Export support in Antrea #5218

L7 Network Flow Export support in Antrea #5218

tushartathgur commented Jul 6, 2023 •

edited

Loading

antoninbas Jan 18, 2024

antoninbas Jan 18, 2024

antoninbas Jan 18, 2024

antoninbas commented Jan 18, 2024

antoninbas Jan 18, 2024

yuntanghsu Jan 18, 2024

tushartathgur Jan 18, 2024

yuntanghsu Jan 18, 2024

tushartathgur Jan 19, 2024 •

edited

Loading

antoninbas Jan 19, 2024

yuntanghsu Jan 18, 2024

tushartathgur Jan 18, 2024 •

edited

Loading

tnqn left a comment

antoninbas commented Jan 19, 2024

tnqn commented Jan 22, 2024

tnqn left a comment

tnqn commented Jan 23, 2024

		for connKey, conn := range cs.connections {
		l7event, ok := l7EventMap[connKey]

L7 Network Flow Export support in Antrea #5218

L7 Network Flow Export support in Antrea #5218

Conversation

tushartathgur commented Jul 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas commented Jan 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tushartathgur Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tushartathgur Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

antoninbas commented Jan 19, 2024

tnqn commented Jan 22, 2024

tnqn left a comment

Choose a reason for hiding this comment

tnqn commented Jan 23, 2024

tushartathgur commented Jul 6, 2023 •

edited

Loading

tushartathgur Jan 19, 2024 •

edited

Loading

tushartathgur Jan 18, 2024 •

edited

Loading