Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DNX]- sonic-clear macsec, does not clear the macsec counters since rekey causes anomaly in the counters. #19311

Open
amitpawar12 opened this issue Jun 14, 2024 · 17 comments
Assignees
Labels
Issue for 202405 Triaged this issue has been triaged

Comments

@amitpawar12
Copy link

Description

Steps to reproduce the issue:

  1. Sent some packets over macsec.
  2. Check the macsec statistics - show macsec
  3. Issue - sonic-clear macsec command.
  4. Wait for sometime, and check the macsec statistics again with - show macsec.

Describe the results you received:

  1. Sent some packets over macsec.

  2. Check the macsec statistics:


---------------------  ---------------
	MACsec Egress SC (18cXXXXe34b20001)
	-----------  -
	encoding_an  1
	-----------  -
		MACsec Egress SA (1)
		-------------------------------------  ----------------------------------------------------------------
-------------- curtailed output -------------
		ssci                                   1
		SAI_MACSEC_SA_ATTR_CURRENT_XPN         879928291
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    854011660550696   <<<<<<<<<<
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
		SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  879928290
		SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
		-------------------------------------  ----------------------------------------------------------------
  1. Clear the macsec statistics:
sonic-clear counters; sonic-clear pfccounters; sonic-clear macsec; sudo ip netns exec asic0 sonic-clear queuecounters
  1. Check the macsec stats again:
      IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
 Ethernet32        U       64  197.76 B/s      0.00%         0        24         0       40  139.23 B/s      0.00%         0         0         0
 Ethernet40        U       63  170.31 B/s      0.00%         0        24         0       39  110.57 B/s      0.00%         0         0         0

	MACsec Egress SC (18cXXXXe34b20001)
	-----------  -
	encoding_an  1
	-----------  -
		MACsec Egress SA (0)
		-------------------------------------  ----------------------------------------------------------------
-------------- curtailed output -------------
		ssci                                   1
		SAI_MACSEC_SA_ATTR_CURRENT_XPN         2
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    488
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
		SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  2
		SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
		-------------------------------------  ----------------------------------------------------------------


      IFACE    STATE    RX_OK      RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
 Ethernet32        U       92  125.24 B/s      0.00%         0        35         0       58  100.01 B/s      0.00%         0         0         0
 Ethernet40        U       92  125.25 B/s      0.00%         0        35         0       58   99.93 B/s      0.00%         0         0         0

	MACsec Egress SC (18cXXXXe34b20001)
	-----------  -
	encoding_an  1
	-----------  -
		MACsec Egress SA (1)
		-------------------------------------  ----------------------------------------------------------------
-------------- curtailed output -------------
		ssci                                   1
		SAI_MACSEC_SA_ATTR_CURRENT_XPN         2
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    854011660554600   <<<<<<<<
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
		SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  1
		SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
		-------------------------------------  ----------------------------------------------------------------

Describe the results you expected:

'sonic-clear macsec' should have cleared the octets encrypted. The same issue is also for ingress SA as well. Hence it becomes difficult to compare the number of packets encrypted during the test, as the counter just piles up pretty quickly.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@judyjoseph
Copy link
Contributor

@amitpawar12 I checked on our lab testbed -- I see it cleans correctly.

admin@svcstr-xxxx-lc1-1:~$ show macsec 
MACsec port(Ethernet0)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 false
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
MACsec port(Ethernet8)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
                MACsec Egress SA (0)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xx
                next_pn                                1
                sak                                    xx
                salt                                   xx
                ssci                                   2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         6894
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    878831
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  6893
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (ba7422dfc4370002)
                MACsec Ingress SA (0)
                ---------------------------------------  ----------------------------------------------------------------
                active                                   true
                auth_key                                 xxx
                lowest_acceptable_pn                     1
                sak                                      xxx
                salt                                     xxx
                ssci                                     1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7336
                SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7020
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      611088
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
                ---------------------------------------  ----------------------------------------------------------------

admin@svcstr-xxxx-lc1-1:~$ sonic-clear macsec 
Clear MACsec counters
admin@svcstr-xxxx-lc1-1:~$ show macsec 
Last cached time was 2024-06-18 15:52:56.439575
MACsec port(Ethernet0)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 false
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
MACsec port(Ethernet8)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec_profile
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (185b00506bfe0001)
        -----------  -
        encoding_an  0
        -----------  -
                MACsec Egress SA (0)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xxx
                next_pn                                1
                sak                                    xxx
                salt                                   xxx
                ssci                                   2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    0
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  0
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (ba7422dfc4370002)
                MACsec Ingress SA (0)
                ---------------------------------------  ----------------------------------------------------------------
                active                                   true
                auth_key                                 xxx
                lowest_acceptable_pn                     1
                sak                                      xxx
                salt                                     xxx
                ssci                                     1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           0
                SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      0
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
                ---------------------------------------  ----------------------------------------------------------------

@judyjoseph
Copy link
Contributor

Can you share the exact sequence you tried ? Also you have the traffic stopped when you clear ?

Another option with sonic-clear macsec --clean-cache true, which will clear the cache if there.

@amitpawar12
Copy link
Author

Sure @judyjoseph. Let me also try with '--clean-cache true' and get back to you.

Thanks,
-A

@judyjoseph
Copy link
Contributor

@amitpawar12 I found an issue with "sonic-clear macsec" when the rekey is enabled. I will raise a PR to fix this. Meanwhile as a work around disable rekey by setting rekey_interval = 0 in th emacsec profile, it should work fine.

@judyjoseph
Copy link
Contributor

Working with Brcm, via CSP CS00012356026

@judyjoseph
Copy link
Contributor

I checked on this again. So according to the current implementation, when we do a rekey -- the key (combination of "PORT:SA_ID:AN_bit" changes ) and the various counters v.z IN_PKTS_OK, OCTETS_ENCRYPTED etc gets reset.

jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9056 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2010
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         53245  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         118070
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3054 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6712
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         72903  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         150805
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7795 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1619
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         67696  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         151932
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3083 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN  0
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         108594 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6757
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5622 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         237686
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         910367 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2062
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4836 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         221143
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         681157 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2115
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4184 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         231087
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         706208 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2078
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5447 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         266794
                                                              >                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2328
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9187 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            3955
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4542 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1203
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            1272 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1017
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4617 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1311
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            64   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            23
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9095 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2009
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3065 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6709
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7821 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1618
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3097 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6754
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5705 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1538
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4919 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2112
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4255 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2075
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5538 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2326
jujoseph@netjb1-westus2:~$ diff -y /tmp/a /tmp/b | less
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         164917 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         763872
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9137 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           3987
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         644661 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         245380
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4502 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1205
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         191301 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         185796
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           1213 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1020
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         835082 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         274623
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4577 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1313
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         62     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         22
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           61   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           23
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         63     |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         24
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           63   |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           25
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         23516  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         44101
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           9056 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2010
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         53245  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         118070
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3054 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6712
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         72903  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         150805
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           7795 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           1619
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         67696  |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         151932
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           3083 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN  0
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         108594 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           6757
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5622 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         237686
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         910367 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2062
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4836 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         221143
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         681157 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2115
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           4184 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         231087
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         706208 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2078
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           5447 |                 SAI_MACSEC_SA_ATTR_CURRENT_XPN         266794
                                                              >                 SAI_MACSEC_SA_ATTR_CURRENT_XPN           2328
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9187 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            3955
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4542 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1203
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            1272 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1017
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4617 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1311
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            62   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            21
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            64   |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            23
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            9095 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2009
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3065 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6709
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            7821 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1618
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            3097 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            6754
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5705 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            1538
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4919 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2112
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            4255 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2075
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            5538 |                 SAI_MACSEC_SA_STAT_IN_PKTS_OK            2326
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475375 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475960
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2333 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2335
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    538597 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    539161
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2953 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2955
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    537715 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    538327
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2711 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2713
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    571909 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    572610
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2577 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2580
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    222900 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    222900
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7817 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7817
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    228299 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    228299
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7173 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      7173
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    217302 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    217302
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      6411 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      6411
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    475926 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    502955
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2323 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2451
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    121159 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    127715
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9177 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9704
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    186399 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    196994
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2083 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      2201
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    154223 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    505675
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      9151 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1533
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    504957 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    521389
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1530 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1569
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    520766 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    516602
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1566 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1557
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    516026 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    520054
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1554 |                 SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1559
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    519365 <
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      1555 <
jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d   jujoseph@STG01-0101-0200-01T2-lc01:/usr/local/lib/python3.9/d
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  168127 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  779025
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  671066 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  246498
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  232895 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  186872
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  878004 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  275811
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  63     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  21
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  63     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  22
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  64     |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  24
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  23861  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  12
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  54061  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  73681  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  242
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  68659  |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  99
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  113031 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  353214
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  947821 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  222271
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  718088 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  232494
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  741377 |                 SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  267858

Let me know your observations.

@amitpawar12
Copy link
Author

Hi @judyjoseph , @vmittal-msft - I still continue to see this issue. The reset happens for a while. But after 2-5 mins, the counters pop up again.

@judyjoseph
Copy link
Contributor

@amitpawar12 can you check if the macsec session is getting rekeyed. This happens on rekey either from sonic/ixia end.

@amitpawar12
Copy link
Author

@judyjoseph - You are right. I configured the rekey interval to an hour to test this.

Log output:

1. Counters incremented:
admin@ixre-egl-board73:~$ show macsec
Last cached time was 2024-08-27 15:04:16.175319
MACsec port(Ethernet144)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                256_XPN_SCI
replay_window          0
send_sci               true
---------------------  ---------------
	MACsec Egress SC (xxxxxx)
	-----------  -
	encoding_an  0
	-----------  -
		MACsec Egress SA (0)
		-------------------------------------  ----------------------------------------------------------------
		auth_key                               xxxx
		next_pn                                1
		sak                                    xxxxx
		salt                                   xxx
		ssci                                   xxx
		SAI_MACSEC_SA_ATTR_CURRENT_XPN         12614100869
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    12311362440103
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
		SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  12614100869
		SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
		-------------------------------------  ----------------------------------------------------------------

2. Cleared the counter:
admin@ixre-egl-board73:~$ sonic-clear macsec
Clear MACsec counters

3. Waited for some time. Did not send any traffic and checked if the counters are getting back to old values. 

admin@ixre-egl-board73:~$ show macsec
Last cached time was 2024-08-27 15:10:43.673238
MACsec port(Ethernet144)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                256_XPN_SCI
replay_window          0
send_sci               true
---------------------  ---------------
	MACsec Egress SC (xxxxxx)
	-----------  -
	encoding_an  0
	-----------  -
		MACsec Egress SA (0)
		-------------------------------------  ----------------------------------------------------------------
		auth_key                               xxxxxxx
		next_pn                                1
		sak                                    xxxxxxx
		salt                                   xxxxxxxxx
		ssci                                   xx
		SAI_MACSEC_SA_ATTR_CURRENT_XPN         23
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    5635
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
		SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  23
		SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
		-------------------------------------  ----------------------------------------------------------------

4. The counters are not getting incremented as was observed earlier.  Rekeying is the cause which might be triggering the old stats to come back or increment along with existing values.

@abdosi
Copy link
Contributor

abdosi commented Aug 28, 2024

Will not be fix for 202205. FIx wil targetted for 202405.

@amitpawar12
Copy link
Author

@judyjoseph - as a data point, we ran a controlled test with fixed packets.

What we observed is that before the rekey, the values are correctly updated but on rekey, they just go to some junk value.

Snapshot:

Before rekeying:
	MACsec Ingress SC (XXXX0001)
		MACsec Ingress SA (1)
		---------------------------------------  ----------------------------------------------------------------
		active                                   true
		auth_key                                 XXXX
		lowest_acceptable_pn                     1
		sak                                      XXXX
		salt                                     XXXX
		ssci                                     2
		SAI_MACSEC_SA_ATTR_CURRENT_XPN           44800004
		SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
		SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
		SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
		SAI_MACSEC_SA_STAT_IN_PKTS_OK            44800006
		SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
		SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      45158401068
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0

After rekeying:
	MACsec Ingress SC (XXXX0001)
		MACsec Ingress SA (0)
		---------------------------------------  ----------------------------------------------------------------
		active                                   true
		auth_key                                 XXXX
		lowest_acceptable_pn                     1
		sak                                      XXXX
		salt                                     XXXX
		ssci                                     2
		SAI_MACSEC_SA_ATTR_CURRENT_XPN           3
		SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
		SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
		SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
		SAI_MACSEC_SA_STAT_IN_PKTS_OK            2
		SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
		SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      95693495115754
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0

Every time, after rekeying, we see different counter although there is no traffic flowing through the system:

	MACsec Ingress SC (XXXX0001)
		MACsec Ingress SA (1)
		---------------------------------------  ----------------------------------------------------------------
		active                                   true
		auth_key                                 XXXX
		lowest_acceptable_pn                     1
		sak                                      XXXX
		salt                                     XXXX
		ssci                                     2
		SAI_MACSEC_SA_ATTR_CURRENT_XPN           3
		SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
		SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
		SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
		SAI_MACSEC_SA_STAT_IN_PKTS_OK            2
		SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
		SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      95693495118958
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
		---------------------------------------  ----------------------------------------------------------------

@amitpawar12 amitpawar12 changed the title [DNX]- sonic-clear macsec, does not clear the macsec counters [DNX]- sonic-clear macsec, does not clear the macsec counters since rekey causes anomaly in the counters. Aug 30, 2024
@judyjoseph
Copy link
Contributor

Raised a CSP with Brcm (CS00012371780) to make sure all the counters are cumulative, currently few SAI counters are not cummulative across rekey

https://brcmsemiconductor-csm.wolkenservicedesk.com/wolken-support/mycases/request-details?requestId=12371780

@arlakshm
Copy link
Contributor

@judyjoseph to validate the SAI fix.

@judyjoseph
Copy link
Contributor

judyjoseph commented Dec 6, 2024

@arlakshm can you review this SAI fix ? https://dev.azure.com/mssonic/broadcom/_git/bcm_sai_external_sug/pullrequest/8578. The fix is validated.

@judyjoseph
Copy link
Contributor

The SAI fix is already present in 11.2.18.1 (#21189)

@arlakshm
Copy link
Contributor

@amitpawar12 ,can you please retest on the latest 202405 image and close this issue if the problem is fixed?

@amitpawar12
Copy link
Author

@amitpawar12 ,can you please retest on the latest 202405 image and close this issue if the problem is fixed?

@arlakshm , I am waiting for the issue #21232 to resolve before I can verify this.

Thanks,
-A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue for 202405 Triaged this issue has been triaged
Projects
Status: No status
Development

No branches or pull requests

5 participants