Linux Branch: “iptfs”:

Linux TFS development environment

There’s a Makefile that will git clone the needed repos.

Usage Notes

Building

If starting from scratch run these 2 commands:

make setup
make

This will checkout a linux kernel source tree under linux, a tagged release of buildroot under buildroot, and a modified branch of iproute2 under iproute2. Then it will build the kernel and the rootfs.

Launching a “Host <-> Router <-> Router <-> Host” setup.

You can use the tests to bring up a working iptfs ipsec tunnel. You’ll need to have installed qemu, socat, python and optional but highly recommended tmux, additionally you’ll need to install the python requirements,

python3 -m venv venv
source venv/bin/activate
pip install -r python-requirements.txt

Then run a test adding the `–pause` flag, and it will pause before running the first test, but after having configured the hosts (h1, h2) and routers (r1, r2).

NOTE: If your `sudo` command forces a restricted PATH (secure_path) then the python virtual environment may not work. In this case `sudo bash` and then activate the virtual environment as root.

NOTE: SUDO: For best results add the following to your /etc/sudoers config. This will allow tmux to continue to work inside the sudo environment.

Defaults env_keep += "TMUX"
Defaults env_keep += "TMUX_PANE"

$ sudo -E pytest -s -v tests/simplenet --pause
[...]
== PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==

or

$ sudo -E bash
# tmux # optional but highly useful
# source venv/bin/activate
# pytest -s -v tests/simplenet --pause
[...]
== PAUSING: before test 'tests/simplenet/test_simplenet.py::test_net_up' ==

You can now log into the running setup in another terminal. Use socat to log into the console of the running qemu’d linux.

$ sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 \
    unix-connect:/tmp/unet-test/tests.simplenet.test_simplenet/r1/s/console2

You can use mucmd to simply enter the namespace of the running node (e.g., to ping from h1 to h2 over the iptfs tunnel use the following command).

$ sudo mucmd -d /tmp/unet-test/tests.simplenet.test_simplenet h1 ping 10.0.2.4
PING 10.0.2.4 (10.0.2.4) 56(84) bytes of data.
64 bytes from 10.0.2.4: icmp_seq=1 ttl=62 time=1.22 ms
64 bytes from 10.0.2.4: icmp_seq=2 ttl=62 time=1.40 ms
64 bytes from 10.0.2.4: icmp_seq=3 ttl=62 time=1.25 ms
...

Qemu

consoles

3 serial consoles are created using unix sockets:

‘%RUNDIR%/s/console’
‘%RUNDIR%/s/vcon0’
‘%RUNDIR%/s/vcon1’

- These can be accessed with::

socat /dev/stdin,escape=0x1d,rawer unix-connect:%RUNDIR%/s/console

Where if the node is named r1 then %RUNDIR% is usually tmp/unet-root/r1 so:

sudo socat /dev/stdin,rawer,escape=0x1d,,echo=0,icanon=0 unix-connect:/tmp/unet-root/r1/s/console

If your socat doesn’t support rawer option replace with raw,echo=0,icanon=0.

GDB

$ sudo gdb linux/vmlinux
(gdb) target remote /tmp/unet-root/r1/s/gdbserver
...

or
(gdb)
target remote /tmp/unet-test/tests.simplenet.test_simplenet/r1/s/gdbserver
target remote /tmp/unet-test/tests.simplenet.test_simplenet/r2/s/gdbserver

target remote /tmp/unet-test/tests.errors.test_errors/r1/s/gdbserver
target remote /tmp/unet-test/tests.errors.test_errors/r2/s/gdbserver

target remote /tmp/unet-test/tests.frags.test_frags/r1/s/gdbserver
target remote /tmp/unet-test/tests.frags.test_frags/r2/s/gdbserver

target remote /tmp/unet-test/tests.phynic.test_simplenet/r1/s/gdbserver

target remote /tmp/unet-test/tests.stress.test_stress/r2/s/gdbserver

target remote /tmp/unet-test/tests.utpkt.test_utpkt/r1/s/gdbserver

target remote /tmp/unet-test/tests.verify.test_verify/r1/s/gdbserver

Building/Editing Notes

CCLS

I like to use ccls with LSP mode in emacs, which will allow you to jump to all references with semantics (i.e., it’s not just a tag search) as well as displaying documentation and tab completion of symbols etc..

In order for this to work seemlessly you need to do a couple steps:

cd linux # kernel source directory
run ./scripts/clang-tools/gen_compile_commands.py -d ../output-linux
printf “%%compile_commands.json\n-D__IN_CCLS__\n” > .ccls

CI Usage

Testing.

The tests contained herein are used directly in the iptfs-linux github project in it’s CI for testing kernel changes.

Releases

To create a new release (for artifacts) of a kernel and rootfs on github add a tag. These artifacts are used by other external project (e.g., by munet to test munet VM capabilities).

This works in the iptfs-dev directory as well as in iptfs-linux as they both use this repo for CI.

Design Notes

Config

The most basic configuration is to select “iptfs” as the mode rather than “tunnel” when configuring an IPsec SA.

Additionally there are further configuration options based on the mode the iptfs tunnel should run in.

common configuration

iptfs-dont-fragment: boolean to disable fragmenting inner packets, defaults to false
iptfs-max-queue-size: The maximum queue size for pending packets to send.
iptfs-pkt-size: the size of the outer packet (outer ip, esp, iptfs, + inner ip packets), either a value or auto to use PMTU
iptfs-reorder-window-size: The number of packets to hold waiting for re-ordered packets to arrive before considering those missing packets dropped. Default for fixed-rate send is 1. Default for demand-rate is 3. If the other endpoint is in dont-fragment mode then this value can be set to 0.
iptfs-drop-time: The number microseconds to wait unti considering the next in sequence packet as lost.

fixed-rate fixed-sized configuration

iptfs-fixed-rate: a fixed rate to send outer packets
iptfs-inner-fixed-rate: alternate form of fixed rate to sepcify inner packets datarate
iptfs-max-delay: alternate config for max-queue-size, which is based on the fixed send rate.
iptfs-disable-congestion-control: disble congestion control, should only be used when the user is in full administrative control of all paths the tunnel may take.
iptfs-no-pad-only: dont’ send all pad packets (debug option)

demand-rate configuration

iptfs-initial-delay: amount of time in microseconds to wait after before servicing the output queue when the initial packet arrives (first in queue). This time allows for collecting more packets to take advantage of IPTFS packet aggregation.

From Steffen’s Mail

[…] look at:

net/xfrm/* net/ipv4/xfrm* net/ipv4/esp4* net/ipv6/xfrm* net/ipv6/esp6*

> Anything else you think might be useful too would be much appreciated of course.

I think TFS should be a new encapsulation mode. We currently have tunnel, transport and beet mode (and some odd ipv6 modes). Adding a tfs_tunnel mode to add all the TFS special stuff would be the way to go at a first glance. The modes are implemented in:

net/xfrm/xfrm_output.c net/xfrm/xfrm_input.c

Bugs

xfrmi_rcv_cb is looking up xfrm_state from our newly created skb from decaping iptfs, but it has not xfrm_state so we panic
- Need to associate the xfrm_state with new skbs too.. is there a refcnt for this?

Miscellaneous Development Scratchpad

This is an area for development notes. Please don’t consider them current or relevant, they’re just a place to keep notes while developing IPTFS.

Coverage, missing

Large Packet Fragmenting Prior to ingress
No reordering window fast path file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(gro_cells, x, skb);
ECN and DSCP for IPv4 and IPv6 file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“dropping in-progress reassemble\n”);
drop timer, future saved in queue to output now file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“receiving ordered list of len %u\n”, count);

Reassembly

during reassembly:
- seq < want, drops packet, can only happen when reorder-window == 0 code coverage shows no code here… file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_OLD_SEQ);
- seq > want, (will need to push past window), next fragment file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“missed needed seq\n”);
- seq == want but no fragment
  - all pad, ok file:linux/net/xfrm/xfrm_iptfs.c::XFRM_INC_SA_STATS(xtfs, IPTFS_INPUT_ALL_PAD_SKIP);
  - not pad, abort and use next fragment file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“missing expected fragment\n”);
- all fragment into next packet
not during reassembly
- fragment start (unexpected), handle next fragment in packet

Reordering

reorder past file:linux/net/xfrm/xfrm_iptfs.c::return __reorder_past(xtfs, inskb, freelist, fcount);
reorder future: dup of one we’ve received file:linux/net/xfrm/xfrm_iptfs.c::/* a dup of a future */
reoder far future (larger that window)
- need something in saved window that also shifts out during middle of shifting file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“send slot0 during shift: %llu”, s0seq);
- need something at end of window that shifts into slot0 (will happend from the above). file:linux/net/xfrm/xfrm_iptfs.c::extra_drops–; /* we aren’t dropping what’s in slot0 */
was_gso == true in ingress/output collect function
zero reorder window size file:linux/net/xfrm/xfrm_iptfs.c::iptfs_input_ordered(x, skb);
free list from iptfs_input_reorder file:linux/net/xfrm/xfrm_iptfs.c::pr_devinf(“freeing list of len %u\n”, fcount);

Ingress (collecting)

input GSO packets

file:linux/net/xfrm/xfrm_iptfs.c::netdev_features_t features = netif_skb_features(skb);

Ingress (sending)

rehoming a fraglist

file:linux/net/xfrm/xfrm_iptfs.c::static struct sk_buff **iptfs_rehome_fraglist(struct sk_buff **nextp,

XXXX DO THIS NEXT

iptfs_should_fragment

file:linux/net/xfrm/xfrm_iptfs.c::iptfs_should_fragment(skb2, mtu, remaining)) { XXXX

Implementation Catalog

Utility

_proto _seq __esp_seq

State Management

xfrm_iptfs_get_rtt_and_delays xfrm_iptfs_init_state xfrm_iptfs_state_destroy xfrm_iptfs_user_init xfrm_iptfs_copy_to_user_state

Egress (receive) functions

skb_copy_bits_seq iptfs_alloc_header_skb iptfs_alloc_skb iptfs_pskb_extract_seq iptfs_input_save_runt __iptfs_iplen __iptfs_iphdrlen

Reassembly

iptfs_complete_inner_skb _iptfs_reassem_done iptfs_reassem_abort iptfs_reassem_done iptfs_reassem_cont iptfs_input_ordered

Reordering

__vec_shift __reorder_past __reorder_drop __reorder_this iptfs_set_window_drop_times __reorder_future_fits __reorder_future_shifts iptfs_input_reorder

Input

iptfs_drop_timer xfrm_iptfs_input

Ingress (sending) functions

From xfrm

iptfs_enqueue xfrm_iptfs_output_collect

Dequeue and send

iptfs_xfrm_output iptfs_output_prepare_skb iptfs_first_skb iptfs_rehome_fraglist iptfs_should_fragment iptfs_output_queued iptfs_delay_timer

header_len usages:

Initializing

AH (ah[46].c)

ah_init_state():
ah6_init_state():

if (x->props.flags & XFRM_STATE_ALIGN4)
    x->props.header_len = XFRM_ALIGN4(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
else
    x->props.header_len = XFRM_ALIGN8(sizeof(struct ip_auth_hdr) + ahp->icv_trunc_len);
if (x->props.mode == XFRM_MODE_TUNNEL)
     x->props.header_len += sizeof(struct iphdr);

     x->props.header_len += sizeof(struct ipv6hdr);

ESP (esp[46].c)

esp_init_state():
esp6_init_state():

/* TRANSPORT mode only has ESP header/trailer + crypto, no IP */
x->props.header_len = sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead);
if (x->props.mode == XFRM_MODE_TUNNEL) {
    /* v4 */
    x->props.header_len += sizeof(struct ipv6hdr);
    /* v6 */
    x->props.header_len += sizeof(struct iphdr);
    /* v4 */
} else if (x->props.mode == XFRM_MODE_BEET && x->sel.family != AF_INET6)
    x->props.header_len += IPV4_BEET_PHMAXLEN;
/* v6 */
else if (x->sel.family != AF_INET6)
    x->props.header_len += IPV4_BEET_PHMAXLEN + (sizeof(struct ipv6hdr) - sizeof(struct iphdr));
if (x->encap) { switch (encap->encap_type) {
    case UDP_ENCAP_ESPINUDP:
        x->props.header_len += sizeof(struct udphdr);
    case UDP_ENCAP_ESPINUDP_NON_IKE:
        x->props.header_len += sizeof(struct udphdr) + 2 * sizeof(u32);
    case TCP_ENCAP_ESPINTCP:
        /* only the length field, TCP encap is done by the socket */
        x->props.header_len += 2;
} }
align = ALIGN(crypto_aead_blocksize(aead), 4);
x->props.trailer_len = align + 1 + crypto_aead_authsize(aead);

ESP Offload (esp[46]_offload.c)

pcode:

static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x,
                                                   struct sk_buff *skb,
                                                   netdev_features_t features)
{
    skb->transport_header += x->props.header_len;
    ops = rcu_dereference(inet_offloads[xo->proto]);
    return ops->callbacks.gso_segment(skb, features);
}

static struct sk_buff *xfrm4_beet_gso_segment(struct xfrm_state *x,
                                              struct sk_buff *skb,
                                              netdev_features_t features)
{
    skb->transport_header += x->props.header_len;
    if (x->sel.family != AF_INET6) {
        if (proto == IPPROTO_BEETPH) {
            struct ip_beet_phdr *ph = (struct ip_beet_phdr *)skb->data;
            skb->transport_header += ph->hdrlen * 8;
            proto = ph->nexthdr;
        } else {
            skb->transport_header -= IPV4_BEET_PHMAXLEN;
        }
    } else {
        __be16 frag;
        skb->transport_header += ipv6_skip_exthdr(skb, 0, &proto, &frag);
        if (proto == IPPROTO_TCP)
            skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV4;
    }

    if (proto == IPPROTO_IPV6)
        skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP4;
    __skb_pull(skb, skb_transport_offset(skb));
    ops = rcu_dereference(inet_offloads[xo->proto]);
    return ops->callbacks.gso_segment(skb, features);
}

IPCOMP (ipcomp.c)

pcode:

static int ipcomp4_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
{
    x->props.header_len = 0;
    if (x->props.mode == XFRM_MODE_TUNNEL)
        x->props.header_len += sizeof(struct iphdr);
}

IPIP (xfrm4_tunnel.c)

pcode:

static int ipip_init_state(struct xfrm_state *x, struct netlink_ext_ack *extack)
{
    x->props.header_len = sizeof(struct iphdr);
}

Using

xfrm_state:

pcode: xfrm_state_mtu
This is used as a pad-to target maybe along with configured pad-to size

the padding added to the trailer will be (this value - skb->len)

u32 xfrm_state_mtu(struct xfrm_state *x, int mtu)
{
    if (x->km.state != XFRM_STATE_VALID || !type || type->proto != IPPROTO_ESP)
        return mtu - x->props.header_len;
    blksize = ALIGN(crypto_aead_blocksize(aead), 4);
    switch (x->props.mode) {
    case XFRM_MODE_TRANSPORT:
    case XFRM_MODE_BEET:
        if (x->props.family == AF_INET)
            net_adj = sizeof(struct iphdr);
        else if (x->props.family == AF_INET6)
            net_adj = sizeof(struct ipv6hdr);
        break;
    case XFRM_MODE_TUNNEL:
    case XFRM_MODE_IPTFS:
        break;
    default:
        WARN_ON_ONCE(1);
        break;
    }
    return ((mtu - x->props.header_len - crypto_aead_authsize(aead) - net_adj) & ~(blksize - 1))
           + net_adj - 2;
    /* TUNNEL is this right what is this value? */
    /* ((1500 - size(IP+ESP) - size(CRYPTAUTH) - 0) & 0xFFFFFFFc) + 0 - 2;

    /* TRANSPORT */
    /* ((1500 - size(ESP) - size(CRYPTAUTH) - size(IP)) & 0xFFFFFFFc) + size(IP) - 2; */
}

xfrm_device

pcode: prepping skb’s for???

This code is going to `skb_pull` `skb_data` down over the mac header and header_len of data. For IPTFS we want it to leave the IPTFS header in the data area for encryption.

static void __xfrm_transport_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb_transport_offset(skb) + x->props.header_len);
}
static void __xfrm_mode_tunnel_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb->mac_len + x->props.header_len);
}
static void __xfrm_mode_iptfs_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    pskb_pull(skb, skb->mac_len + x->props.header_len - sizeof(struct iptfs_header));
}
static void __xfrm_mode_beet_prep(struct xfrm_state *x, struct sk_buff *skb)
{
    if (xo->flags & XFRM_GSO_SEGMENT)
        skb->transport_header = skb->network_header + sizeof(struct iphdr) || sizeof(struct ipv6hdr);
    /* this is reversing something done earlier */
    if (x->sel.family != AF_INET6) {
        phlen = IPV4_BEET_PHMAXLEN;
        if (x->outer_mode.family == AF_INET6)
            phlen += sizeof(struct ipv6hdr) - sizeof(struct iphdr);
    }
    pskb_pull(skb, skb->mac_len + hsize + (x->props.header_len - phlen));
}

Call path for xfrm_outer_mode_prep:

static const struct proto_ops packet_ops = { .sendmsg = packet_sendmsg, packet_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) packet_snd(struct socket *sock, struct msghdr *msg, size_t len) || tpacket_snd(struct packet_sock *po, struct msghdr *msg) packet_xmit(const struct packet_sock *po, struct sk_buff *skb) dev_direct_xmit(struct sk_buff *skb, u16 queue_id) __dev_direct_xmit(struct sk_buff *skb, u16 queue_id) || sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q, validate_xmit_skb_list(struct sk_buff *skb, struct net_device *dev, bool *again) validate_xmit_skb(struct sk_buff *skb, struct net_device *dev, bool *again) validate_xmit_xfrm xfrm_outer_mode_prep

xfrm_policy

Offload vs No offload backtrace

No ESP offload

#0 esp_output (x=0xffff888107a8a000, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4.c:655 #1 0xffffffff827a5f8c in xfrm_output_one (err=0, skb=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:600

#2 xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40, err=<optimized out>, err@entry=1) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:635 #3 0xffffffff827a6483 in xfrm_output2 (skb=0xffff888109584b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849f7e40 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662 #4 xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109584b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826 #5 0xffffffff827b9052 in iptfs_xfrm_output (skb=skb@entry=0xffff888109584b40, remaining=remaining@entry=1358) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648 #6 0xffffffff827bc427 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080 #7 iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129

#8 0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a8f468, base=0xffff88823762b3c0, cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685 #9 __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=39634970245, flags=flags@entry=642, active_mask=active_mask@entry=240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749 #10 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766 #11 0xffffffff82b50b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #12 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #13 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #14 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #15 0xffffffff82b3b5ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #16 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #17 0x0000000000000000 in ?? () (gdb)

ESP offload (INET_ESP_OFFLOAD)

#0 esp_xmit (x=0xffff888107a8e000, skb=0xffff888109589b40, features=0) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:257 #1 0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff888109589b40, features=features@entry=1, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187

#2 0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff888109589b40, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687 #3 0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff888109589b40, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff888237609658) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709 #4 0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff888109589b40, q=q@entry=0xffff8881084e6000, dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>, validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327 #5 0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000, skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805 #6 __dev_queue_xmit (skb=skb@entry=0xffff888109589b40, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210 #7 0xffffffff82590b8d in dev_queue_xmit (skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085

#8 neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552 #9 0xffffffff826a844d in neigh_output (skip_cache=false, skb=0xffff888109589b40, n=0xffff8881092ac000) at /home/chopps/w/iptfs-dev/linux/include/net/neighbour.h:544 #10 ip_finish_output2 (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=<optimized out>, skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:229 #11 0xffffffff826aa7d6 in __ip_finish_output (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:307 #12 0xffffffff826aaa72 in ip_finish_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:317 #13 NF_HOOK_COND (pf=2 ‘\002’, hook=4, okfn=0xffffffff826aa910 <ip_finish_output>, cond=<optimized out>, out=0xffff8881053be000, in=0xffff8881053aa000, skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:292 #14 ip_output (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/ipv4/ip_output.c:431 #15 0xffffffff827b7bd9 in dst_output (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=0xffffffff849fcec0 <init_net>) at /home/chopps/w/iptfs-dev/linux/include/net/dst.h:458

#16 xfrm_output_resume (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40, err=err@entry=1) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:643 #17 0xffffffff827b97dd in xfrm_output2 (skb=0xffff888109589b40, sk=0x0 <fixed_percpu_data>, net=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:662 #18 xfrm_output (sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589b40) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_output.c:826 #19 0xffffffff827ce922 in iptfs_xfrm_output (skb=skb@entry=0xffff888109589b40, remaining=remaining@entry=1358) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:1648 #20 0xffffffff827d1cf7 in iptfs_output_queued (list=0xffff888237609db0, x=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2080 #21 iptfs_delay_timer (me=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_iptfs.c:2129

#22 0xffffffff812431fe in __run_hrtimer (flags=642, now=0xffff888237609ec0, timer=0xffff888107a91c68, base=0xffff88823762b3c0, cpu_base=0xffff88823762b240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1685 #23 __hrtimer_run_queues (cpu_base=cpu_base@entry=0xffff88823762b240, now=now@entry=123664696935, flags=flags@entry=642, active_mask=active_mask@entry=240) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1749 #24 0xffffffff8124358b in hrtimer_run_softirq (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/kernel/time/hrtimer.c:1766 #25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #31 0x0000000000000000 in ?? () (gdb) c

ARP

#0 esp_xmit (x=0xffff888107a8e000, skb=0xffff8881095893c0, features=0) at /home/chopps/w/iptfs-dev/linux/net/ipv4/esp4_offload.c:270 #1 0xffffffff827bcfd4 in validate_xmit_xfrm (skb=skb@entry=0xffff8881095893c0, features=features@entry=1, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/xfrm/xfrm_device.c:187 #2 0xffffffff825766db in validate_xmit_skb (skb=skb@entry=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3687 #3 0xffffffff82576a1b in validate_xmit_skb_list (skb=0xffff8881095893c0, dev=dev@entry=0xffff8881053be000, again=again@entry=0xffff8882376094a8) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3709 #4 0xffffffff82608321 in sch_direct_xmit (skb=skb@entry=0xffff8881095893c0, q=q@entry=0xffff8881084e6000, dev=dev@entry=0xffff8881053be000, txq=txq@entry=0xffff888104990c00, root_lock=root_lock@entry=0x0 <fixed_percpu_data>, validate=validate@entry=true) at /home/chopps/w/iptfs-dev/linux/net/sched/sch_generic.c:327 #5 0xffffffff82577b8d in __dev_xmit_skb (txq=0xffff888104990c00, dev=0xffff8881053be000, q=0xffff8881084e6000, skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:3805 #6 __dev_queue_xmit (skb=skb@entry=0xffff8881095893c0, sb_dev=sb_dev@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:4210 #7 0xffffffff82590b8d in dev_queue_xmit (skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/include/linux/netdevice.h:3085 #8 neigh_resolve_output (neigh=0xffff8881092ac000, skb=0xffff8881095893c0) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1552 #9 0xffffffff82590fa0 in __neigh_update (neigh=neigh@entry=0xffff8881092ac000, lladdr=<optimized out>, lladdr@entry=0xffff8881058ff056 “\002\252\252\252\002\003\n”, new=<optimized out>, flags=0, nlmsg_pid=nlmsg_pid@entry=0, extack=extack@entry=0x0 <fixed_percpu_data>) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1450 #10 0xffffffff82591c93 in neigh_update (neigh=neigh@entry=0xffff8881092ac000, lladdr=lladdr@entry=0xffff8881058ff056 “\002\252\252\252\002\003\n”, new=<optimized out>, flags=<optimized out>, nlmsg_pid=nlmsg_pid@entry=0) at /home/chopps/w/iptfs-dev/linux/net/core/neighbour.c:1477 #11 0xffffffff8271fbc5 in arp_process (net=net@entry=0xffffffff849fcec0 <init_net>, sk=sk@entry=0x0 <fixed_percpu_data>, skb=skb@entry=0xffff888109589640) at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:932 #12 0xffffffff827206d4 in NF_HOOK (pf=3 ‘\003’, sk=0x0 <fixed_percpu_data>, okfn=0xffffffff8271f700 <arp_process>, out=0x0 <fixed_percpu_data>, in=0xffff8881053be000, skb=0xffff888109589640, net=0xffffffff849fcec0 <init_net>, hook=0) at /home/chopps/w/iptfs-dev/linux/include/linux/netfilter.h:303 #13 arp_rcv (skb=0xffff888109589640, dev=0xffff8881053be000, pt=<optimized out>, orig_dev=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/ipv4/arp.c:988 #14 0xffffffff8257b1bc in __netif_receive_skb_list_ptype (orig_dev=0xffff8881053be000, pt_prev=0xffffffff842766c0 <arp_packet_type>, head=0xffff888237609a48) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5539 #15 __netif_receive_skb_list_core (head=head@entry=0xffff8881053c2110, pfmemalloc=pfmemalloc@entry=false) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5582 #16 0xffffffff8257bd97 in __netif_receive_skb_list (head=0xffff8881053c2110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5634 #17 netif_receive_skb_list_internal (head=head@entry=0xffff8881053c2110) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:5725 #18 0xffffffff8257c983 in gro_normal_list (napi=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/include/net/gro.h:433 #19 napi_complete_done (n=n@entry=0xffff8881053c2008, work_done=work_done@entry=1) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6065 #20 0xffffffff81f70300 in virtqueue_napi_complete (processed=<optimized out>, vq=0xffff888105357800, napi=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:406 #21 virtnet_poll (napi=<optimized out>, budget=<optimized out>) at /home/chopps/w/iptfs-dev/linux/drivers/net/virtio_net.c:1965 #22 0xffffffff8257cc91 in __napi_poll (n=n@entry=0xffff8881053c2008, repoll=repoll@entry=0xffff888237609ed0) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6496 #23 0xffffffff8257d964 in napi_poll (repoll=0xffff888237609f00, n=0xffff8881053c2008) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6563 #24 net_rx_action (h=<optimized out>) at /home/chopps/w/iptfs-dev/linux/net/core/dev.c:6696 #25 0xffffffff82b67b0c in __do_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:571 #26 0xffffffff8113a70c in invoke_softirq () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:445 #27 __irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:650 #28 irq_exit_rcu () at /home/chopps/w/iptfs-dev/linux/kernel/softirq.c:662 #29 0xffffffff82b525ca in common_interrupt (regs=0xffffffff83a07d58, error_code=<optimized out>) at /home/chopps/w/iptfs-dev/linux/arch/x86/kernel/irq.c:240 #30 0xffffffff82c0146b in asm_common_interrupt () at /home/chopps/w/iptfs-dev/linux/arch/x86/include/asm/idtentry.h:636 #31 0x0000000000000000 in ?? ()

nr_frags large packet

(gdb) p *skb $25 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, {dev = 0xffff888109008000, dev_scratch = 18446612686516027392}}, rbnode = { __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>, rb_left = 0xffff888109008000}, list = {next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, {sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0, skb_mstamp_ns = 0}, cb = “\b”, ‘\000’ <repeats 19 times>, “\001\000\000\000ȉ\373\377\000\000\000\000\316\000\000\000\024\000\000\000ʠ\377\377\342\000\377\377”, {{ _skb_refdst = 18446612686612525184, destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff88810ec0f080, prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686612525184}, _nfct = 0, len = 8040, data_len = 4982, mac_len = 0, hdr_len = 0, queue_mapping = 1, __cloned_offset = 0xffff88810e8fb43e “”, cloned = 0 ‘\000’, nohdr = 0 ‘\000’, fclone = 0 ‘\000’, peeked = 0 ‘\000’, head_frag = 0 ‘\000’, pfmemalloc = 0 ‘\000’, pp_recycle = 0 ‘\000’, active_extensions = 0 ‘\000’, {{__pkt_type_offset = 0xffff88810e8fb440 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff88810e8fb441 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 1 ‘\001’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048802, { csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69, sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}, headers = { __pkt_type_offset = 0xffff88810e8fb440 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff88810e8fb441 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 1 ‘\001’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048802, { csum_start = 226, csum_offset = 16}}, priority = 0, skb_iif = 8, hash = 2159314812, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 69, sender_cpu = 69}, secmark = 0, {mark = 0, reserved_tailroom = 0}, {inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 226, network_header = 206, mac_header = 206}}, tail = 3264, end = 3776, head = 0xffff88810ed64000 “”, data = 0xffff88810ed640ce “E”, truesize = 9334, users = {refs = {counter = 1}}, extensions = 0x0 <fixed_percpu_data>} (gdb) p *(struct skb_shared_info *)(skb->head + skb->end) $26 = {flags = 0 ‘\000’, meta_len = 0 ‘\000’, nr_frags = 2 ‘\002’, tx_flags = 0 ‘\000’, gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>, hwtstamps = {{hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0, dataref = {counter = 1}, xdp_frags_size = 0, destructor_arg = 0x0 <fixed_percpu_data>, frags = {{bv_page = 0xffffea000432cb00, bv_len = 3072, bv_offset = 192}, {bv_page = 0xffffea000432cb80, bv_len = 1910, bv_offset = 192}, {bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

These 2 pages are 0x80 apart (sizeof *page == 0x40 so 2 pages), but the offset is 192 (0xc0) on both with lengths of 3072 and 1910 these regions overlap. That’s b/c the actual page address is not the page struct.

Here’s a SW based 2 frag version: (gdb) p *skb $5 = {{{next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>, { dev = 0xffff888109210000, dev_scratch = 18446612686518157312}}, rbnode = { __rb_parent_color = 0, rb_right = 0x0 <fixed_percpu_data>, rb_left = 0xffff888109210000}, list = {next = 0x0 <fixed_percpu_data>, prev = 0x0 <fixed_percpu_data>}, ll_node = {next = 0x0 <fixed_percpu_data>}}, { sk = 0x0 <fixed_percpu_data>, ip_defrag_offset = 0}, {tstamp = 0, skb_mstamp_ns = 0}, cb = “\004”, ‘\000’ <repeats 19 times>, “\001”, ‘\000’ <repeats 11 times>, “N\000\000\000\024\000\000\000\306\340\377\377b\000\000”, {{_skb_refdst = 18446612686654115968, destructor = 0x0 <fixed_percpu_data>}, tcp_tsorted_anchor = {next = 0xffff8881113b9080, prev = 0x0 <fixed_percpu_data>}, _sk_redir = 18446612686654115968}, _nfct = 0, len = 9000, data_len = 8948, mac_len = 0, hdr_len = 0, queue_mapping = 1, __cloned_offset = 0xffff8881112f657e “”, cloned = 0 ‘\000’, nohdr = 0 ‘\000’, fclone = 0 ‘\000’, peeked = 0 ‘\000’, head_frag = 0 ‘\000’, pfmemalloc = 0 ‘\000’, pp_recycle = 0 ‘\000’, active_extensions = 0 ‘\000’, {{ __pkt_type_offset = 0xffff8881112f6580 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff8881112f6581 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 0 ‘\000’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0, skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66, sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, { inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98, network_header = 78, mac_header = 78}, headers = { __pkt_type_offset = 0xffff8881112f6580 “”, pkt_type = 0 ‘\000’, ignore_df = 0 ‘\000’, dst_pending_confirm = 0 ‘\000’, ip_summed = 0 ‘\000’, ooo_okay = 0 ‘\000’, __mono_tc_offset = 0xffff8881112f6581 “”, mono_delivery_time = 0 ‘\000’, tc_at_ingress = 0 ‘\000’, tc_skip_classify = 0 ‘\000’, remcsum_offload = 0 ‘\000’, csum_complete_sw = 0 ‘\000’, csum_level = 0 ‘\000’, inner_protocol_type = 0 ‘\000’, l4_hash = 0 ‘\000’, sw_hash = 0 ‘\000’, wifi_acked_valid = 0 ‘\000’, wifi_acked = 0 ‘\000’, no_fcs = 0 ‘\000’, encapsulation = 0 ‘\000’, encap_hdr_csum = 0 ‘\000’, csum_valid = 0 ‘\000’, ndisc_nodetype = 0 ‘\000’, redirected = 0 ‘\000’, nf_skip_egress = 0 ‘\000’, slow_gro = 1 ‘\001’, tc_index = 0, alloc_cpu = 1, {csum = 1048674, {csum_start = 98, csum_offset = 16}}, priority = 0, skb_iif = 4, hash = 0, {vlan_all = 0, {vlan_proto = 0, vlan_tci = 0}}, {napi_id = 66, sender_cpu = 66}, secmark = 0, {mark = 0, reserved_tailroom = 0}, { inner_protocol = 0, inner_ipproto = 0 ‘\000’}, inner_transport_header = 0, inner_network_header = 0, inner_mac_header = 0, protocol = 8, transport_header = 98, network_header = 78, mac_header = 78}}, tail = 130, end = 320, head = 0xffff88810fa37c00 “”, data = 0xffff88810fa37c4e “E”, truesize = 9844, users = { refs = {counter = 1}}, extensions = 0x0 <fixed_percpu_data>} (gdb) p *((struct skb_shared_info *)(skb->head + skb->end)) $6 = {flags = 0 ‘\000’, meta_len = 0 ‘\000’, nr_frags = 2 ‘\002’, tx_flags = 0 ‘\000’, gso_size = 0, gso_segs = 0, frag_list = 0x0 <fixed_percpu_data>, hwtstamps = {{ hwtstamp = 0, netdev_data = 0x0 <fixed_percpu_data>}}, gso_type = 0, tskey = 0, dataref = {counter = 1}, xdp_frags_size = 0, destructor_arg = 0x0 <fixed_percpu_data>, frags = {{bv_page = 0xffffea0004387000, bv_len = 3506, bv_offset = 29262}, { bv_page = 0xffffea0004387200, bv_len = 5442, bv_offset = 0}, { bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0} <repeats 15 times>}}

$3 = { […] frags = {{ bv_page = 0xffffea0004387000, bv_len = 3506, bv_offset = 29262 }, { bv_page = 0xffffea0004387200, bv_len = 5442, bv_offset = 0 }, { bv_page = 0x0 <fixed_percpu_data>, bv_len = 0, bv_offset = 0 } <repeats 15 times>} }

(gdb) p *$7->frags[0].bv_page $18 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>, mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0, share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122, _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = { counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>, pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, { pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, { _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 1}} (gdb) p/x *$7->frags[0].bv_page $20 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, { __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0, prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}}, mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0, pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0, pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = { counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = { counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x1}}

(gdb) p *$7->frags[1].bv_page $19 = {flags = 144115188075921408, {{{lru = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, {__filler = 0x0 <fixed_percpu_data>, mlock_count = 290}, buddy_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}, pcp_list = {next = 0x0 <fixed_percpu_data>, prev = 0xdead000000000122}}, mapping = 0x0 <fixed_percpu_data>, {index = 0, share = 0}, private = 0}, {pp_magic = 0, pp = 0xdead000000000122, _pp_mapping_pad = 0, dma_addr = 0, {dma_addr_upper = 0, pp_frag_count = { counter = 0}}}, {compound_head = 0}, {_pt_pad_1 = 0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0, {pt_mm = 0x0 <fixed_percpu_data>, pt_frag_refcount = {counter = 0}}, ptl = 0x0 <fixed_percpu_data>}, { pgmap = 0x0 <fixed_percpu_data>, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0 <fixed_percpu_data>, func = 0xdead000000000122}}, { _mapcount = {counter = -1}, page_type = 4294967295}, _refcount = {counter = 4}} (gdb) p/x *$7->frags[1].bv_page $21 = {flags = 0x200000000010000, {{{lru = {next = 0x0, prev = 0xdead000000000122}, { __filler = 0x0, mlock_count = 0x122}, buddy_list = {next = 0x0, prev = 0xdead000000000122}, pcp_list = {next = 0x0, prev = 0xdead000000000122}}, mapping = 0x0, {index = 0x0, share = 0x0}, private = 0x0}, {pp_magic = 0x0, pp = 0xdead000000000122, _pp_mapping_pad = 0x0, dma_addr = 0x0, {dma_addr_upper = 0x0, pp_frag_count = {counter = 0x0}}}, {compound_head = 0x0}, {_pt_pad_1 = 0x0, pmd_huge_pte = 0xdead000000000122, _pt_pad_2 = 0x0, {pt_mm = 0x0, pt_frag_refcount = { counter = 0x0}}, ptl = 0x0}, {pgmap = 0x0, zone_device_data = 0xdead000000000122}, callback_head = {next = 0x0, func = 0xdead000000000122}}, {_mapcount = { counter = 0xffffffff}, page_type = 0xffffffff}, _refcount = {counter = 0x4}}

Sandbox

CONFIG options for tracing

CONFIG_DEBUG_INFO=y CONFIG_FRAME_POINTER=y CONFIG_FTRACE=y CONFIG_KALLSYMS=y CONFIG_KPROBES=y CONFIG_KPROBE_EVENTS=y CONFIG_LOCKDEP=y CONFIG_LOCK_STAT=y CONFIG_PERF_EVENTS=y CONFIG_TRACEPOINTS=y CONFIG_UPROBES=y CONFIG_UPROBE_EVENTS=y

Sample PPS and packet send times for 1500B IP packets

#include <stdio.h>
#define ENET_OHEAD (14 + 4 + 8 + 12)
#define _1GE_PPS(iptfs_ip_mtu) ((1e9 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _10GE_PPS(iptfs_ip_mtu) ((1e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _40GE_PPS(iptfs_ip_mtu) ((4e10 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _100GE_PPS(iptfs_ip_mtu) ((1e11 / 8) / ((iptfs_ip_mtu) + ENET_OHEAD))
#define _1GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _1GE_PPS(iptfs_ip_mtu))
#define _10GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _10GE_PPS(iptfs_ip_mtu))
#define _40GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _40GE_PPS(iptfs_ip_mtu))
#define _100GE_PP_NANOS(iptfs_ip_mtu) (1e9 / _100GE_PPS(iptfs_ip_mtu))

int mtu = 64;

printf("+ 1GE 10GE 40GE 100GE\n");
printf("PPS %lu %lu %lu %lu\n", (uint64_t)_1GE_PPS(mtu),(uint64_t)_10GE_PPS(mtu),(uint64_t)_40GE_PPS(mtu),(uint64_t)_100GE_PPS(mtu));
printf("packet-time %luns %luns %luns %luns\n", (uint64_t)_1GE_PP_NANOS(mtu),(uint64_t)_10GE_PP_NANOS(mtu),(uint64_t)_40GE_PP_NANOS(mtu),(uint64_t)_100GE_PP_NANOS(mtu));

Checking uncommon flag setting code disassembly

No idea how to get this emacs-babel to work, but this saves the code for now. Babel is always evaluating the C block and saving those empty results in the file not the `code` as instructed.

echo "$filename"
gcc -O2 -c -o ${filename%.c}.o $filename
echo objdump -S -d "${filename%.c}.o"

#define SETF (1<<12)
#define CHECKF (1ull << 38)

unsigned long long is_setdm(unsigned long long bits, unsigned long long result)
{
    result |= (bits & CHECKF) / CHECKF * SETF;
    return result;
}

unsigned long long is_setto(unsigned long long bits, unsigned long long result)
{
    result |= bits & CHECKF ? SETF : 0;
    return result;
}

unsigned long long is_setif(unsigned long long bits, unsigned long long result)
{
    if (!!(bits & CHECKF))
        result |= SETF;
    return result;
}

echo "Hello World"

Performance Triaging

Testing done on a single server with 3 networks cards using munet and wiring the ports to each other. [h1] - [r1] - [r2] - [h2] ===========

Qemu emulated - single socket/core

Routed from h1 to h2 iperf bidir: ~2000 Mbps
IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~120 Mbps
IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir ~2 Mbps

Qemu -accel kvm - single socket/core

Routed from h1 to h2 iperf bidir: ~9400 Mbps
IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~920 Mbps
IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir ~2 Mbps

Qemu -accel kvm - 4 sockets

Routed from h1 to h2 iperf bidir: ~9400 Mbps
IPsec [r1,r2] tunnel, from h1 to h2 iperf bidir ~7200 Mbps
IPTFS [r1,r2] tunnel, from h1 to h2 iperf bidir 700Kbps-3.87Mbps

Examples

These are single Core

tests/stress/trex_stress_phy.py::test_policy_imix

mode == tunnel (i.e., normal ipsec) (50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.83 Gbps version : STL @ v2.98 total_tx_L1 : 3.93 Gbps cpu_util. : 6.52% @ 2 cores (2 per dual port) total_rx : 3.04 Gbps rx_cpu_util. : 8.26% / 0 pps total_pps : 645.97 Kpps async_util. : 0% / 30.41 bps drop_rate : 792.36 Mbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.52% | 6.52% | – | | | Tx bps L2 | 1.92 Gbps | 1.91 Gbps | 3.83 Gbps Tx bps L1 | 1.97 Gbps | 1.96 Gbps | 3.93 Gbps Tx pps | 323.81 Kpps | 322.16 Kpps | 645.97 Kpps Line Util. | 4.93 % | 4.9 % | — | | | Rx bps | 1.52 Gbps | 1.51 Gbps | 3.04 Gbps Rx pps | 256.81 Kpps | 255.5 Kpps | 512.31 Kpps ---- | | | opackets | 7479318 | 7511222 | 14990540 ipackets | 5941015 | 5965888 | 11906903 obytes | 5542174638 | 5565815502 | 11107990140 ibytes | 4402286022 | 4421020888 | 8823306910 tx-pkts | 7.48 Mpkts | 7.51 Mpkts | 14.99 Mpkts rx-pkts | 5.94 Mpkts | 5.97 Mpkts | 11.91 Mpkts tx-bytes | 5.54 GB | 5.57 GB | 11.11 GB rx-bytes | 4.4 GB | 4.42 GB | 8.82 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix new 50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.84 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 6.39% @ 2 cores (2 per dual port) total_rx : 2.9 Gbps rx_cpu_util. : 6.74% / 0 pps total_pps : 647.42 Kpps async_util. : 0% / 34.3 bps drop_rate : 942.81 Mbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.39% | 6.39% | – | | | Tx bps L2 | 1.92 Gbps | 1.92 Gbps | 3.84 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 323.87 Kpps | 324.12 Kpps | 647.99 Kpps Line Util. | 4.93 % | 4.93 % | — | | | Rx bps | 1.45 Gbps | 1.45 Gbps | 2.9 Gbps Rx pps | 244.29 Kpps | 244.59 Kpps | 488.88 Kpps ---- | | | opackets | 6113860 | 6158922 | 12272782 ipackets | 4643520 | 4678016 | 9321536 obytes | 4530370260 | 4563761202 | 9094131462 ibytes | 3440826656 | 3466377360 | 6907204016 tx-pkts | 6.11 Mpkts | 6.16 Mpkts | 12.27 Mpkts rx-pkts | 4.64 Mpkts | 4.68 Mpkts | 9.32 Mpkts tx-bytes | 4.53 GB | 4.56 GB | 9.09 GB rx-bytes | 3.44 GB | 3.47 GB | 6.91 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.98 Gbps version : STL @ v2.98 total_tx_L1 : 4.2 Gbps cpu_util. : 12.07% @ 2 cores (2 per dual port) total_rx : 2.02 Gbps rx_cpu_util. : 9.33% / 0 pps total_pps : 1.41 Mpps async_util. : 0% / 36.53 bps drop_rate : 1.95 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 12.07% | 12.07% | – | | | Tx bps L2 | 2 Gbps | 1.98 Gbps | 3.98 Gbps Tx bps L1 | 2.11 Gbps | 2.09 Gbps | 4.2 Gbps Tx pps | 706.35 Kpps | 700.51 Kpps | 1.41 Mpps Line Util. | 5.28 % | 5.23 % | — | | | Rx bps | 1.02 Gbps | 1.01 Gbps | 2.02 Gbps Rx pps | 359.91 Kpps | 356.86 Kpps | 716.76 Kpps ---- | | | opackets | 10937602 | 10984861 | 21922463 ipackets | 5620736 | 5645504 | 11266240 obytes | 3866441526 | 3883148074 | 7749589600 ibytes | 1985575700 | 1995323444 | 3980899144 tx-pkts | 10.94 Mpkts | 10.98 Mpkts | 21.92 Mpkts rx-pkts | 5.62 Mpkts | 5.65 Mpkts | 11.27 Mpkts tx-bytes | 3.87 GB | 3.88 GB | 7.75 GB rx-bytes | 1.99 GB | 2 GB | 3.98 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.94 Gbps version : STL @ v2.98 total_tx_L1 : 4.16 Gbps cpu_util. : 11.57% @ 2 cores (2 per dual port) total_rx : 1.5 Gbps rx_cpu_util. : 8.49% / 0 pps total_pps : 1.39 Mpps async_util. : 0% / 0 bps drop_rate : 2.43 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 11.57% | 11.57% | – | | | Tx bps L2 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx bps L1 | 2.08 Gbps | 2.08 Gbps | 4.16 Gbps Tx pps | 694.97 Kpps | 697.46 Kpps | 1.39 Mpps Line Util. | 5.19 % | 5.21 % | — | | | Rx bps | 750.55 Mbps | 754.34 Mbps | 1.5 Gbps Rx pps | 265.62 Kpps | 266.53 Kpps | 532.14 Kpps ---- | | | opackets | 313742270 | 313798138 | 627540408 ipackets | 120083200 | 120102542 | 240185742 obytes | 110907888436 | 110927643186 | 221835531622 ibytes | 42428351448 | 42428427258 | 84856778706 tx-pkts | 313.74 Mpkts | 313.8 Mpkts | 627.54 Mpkts rx-pkts | 120.08 Mpkts | 120.1 Mpkts | 240.19 Mpkts tx-bytes | 110.91 GB | 110.93 GB | 221.84 GB rx-bytes | 42.43 GB | 42.43 GB | 84.86 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

These are 3 core

NOTE: for Qemu if multiple cores are used ipsec/iptfs seems to only use 1 thread (core). If multple sockets are used then multiple threads (sockets) are utilized.

tests/stress/trex_stress_phy.py::test_policy_imix

New IMIX (50/50)

mode == tunnel (i.e., normal ipsec) (50%/50%)

===== 0 DROP May go faster =====

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.87 Gbps version : STL @ v2.98 total_tx_L1 : 3.98 Gbps cpu_util. : 7.67% @ 2 cores (2 per dual port) total_rx : 3.87 Gbps rx_cpu_util. : 10.18% / 0.12 pps total_pps : 653.14 Kpps async_util. : 0% / 0 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 7.67% | 7.67% | – | | | Tx bps L2 | 1.95 Gbps | 1.92 Gbps | 3.87 Gbps Tx bps L1 | 2 Gbps | 1.97 Gbps | 3.98 Gbps Tx pps | 328.91 Kpps | 324.28 Kpps | 653.19 Kpps Line Util. | 5.01 % | 4.94 % | — | | | Rx bps | 1.95 Gbps | 1.92 Gbps | 3.87 Gbps Rx pps | 328.91 Kpps | 324.25 Kpps | 653.16 Kpps ---- | | | opackets | 37006482 | 37035660 | 74042142 ipackets | 37006464 | 37035618 | 74042082 obytes | 27421803162 | 27443424060 | 54865227222 ibytes | 27421789824 | 27443391520 | 54865181344 tx-pkts | 37.01 Mpkts | 37.04 Mpkts | 74.04 Mpkts rx-pkts | 37.01 Mpkts | 37.04 Mpkts | 74.04 Mpkts tx-bytes | 27.42 GB | 27.44 GB | 54.87 GB rx-bytes | 27.42 GB | 27.44 GB | 54.87 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (imix new 50%/50%)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.85 Gbps version : STL @ v2.98 total_tx_L1 : 3.95 Gbps cpu_util. : 6.35% @ 2 cores (2 per dual port) total_rx : 2.7 Gbps rx_cpu_util. : 6.64% / 0 pps total_pps : 649 Kpps async_util. : 0% / 33.17 bps drop_rate : 1.14 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 6.35% | 6.35% | – | | | Tx bps L2 | 1.93 Gbps | 1.93 Gbps | 3.86 Gbps Tx bps L1 | 1.98 Gbps | 1.99 Gbps | 3.97 Gbps Tx pps | 325.41 Kpps | 326.34 Kpps | 651.75 Kpps Line Util. | 4.95 % | 4.97 % | — | | | Rx bps | 1.35 Gbps | 1.36 Gbps | 2.71 Gbps Rx pps | 228.17 Kpps | 228.88 Kpps | 457.05 Kpps ---- | | | opackets | 7075748 | 7094883 | 14170631 ipackets | 5048896 | 5062638 | 10111534 obytes | 5243129268 | 5257307626 | 10500436894 ibytes | 3741435036 | 3751414630 | 7492849666 tx-pkts | 7.08 Mpkts | 7.09 Mpkts | 14.17 Mpkts rx-pkts | 5.05 Mpkts | 5.06 Mpkts | 10.11 Mpkts tx-bytes | 5.24 GB | 5.26 GB | 10.5 GB rx-bytes | 3.74 GB | 3.75 GB | 7.49 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

Old Imix 7,4,1

mode == iptfs (imix legacy/firewall 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.93 Gbps version : STL @ v2.98 total_tx_L1 : 4.16 Gbps cpu_util. : 14.76% @ 2 cores (2 per dual port) total_rx : 3.56 Gbps rx_cpu_util. : 18.21% / 0 pps total_pps : 1.39 Mpps async_util. : 0% / 30.14 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 14.76% | 14.76% | – | | | Tx bps L2 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx bps L1 | 2.08 Gbps | 2.09 Gbps | 4.17 Gbps Tx pps | 696.39 Kpps | 697.98 Kpps | 1.39 Mpps Line Util. | 5.2 % | 5.21 % | — | | | Rx bps | 1.86 Gbps | 1.71 Gbps | 3.58 Gbps Rx pps | 658.15 Kpps | 606.26 Kpps | 1.26 Mpps ---- | | | opackets | 14995714 | 15099834 | 30095548 ipackets | 14231296 | 13124265 | 27355561 obytes | 5300982114 | 5337789222 | 10638771336 ibytes | 5030619874 | 4638994972 | 9669614846 tx-pkts | 15 Mpkts | 15.1 Mpkts | 30.1 Mpkts rx-pkts | 14.23 Mpkts | 13.12 Mpkts | 27.36 Mpkts tx-bytes | 5.3 GB | 5.34 GB | 10.64 GB rx-bytes | 5.03 GB | 4.64 GB | 9.67 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == tunnel (i.e., normal ipsec) (imix legacy 7,4,1)

Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3.95 Gbps version : STL @ v2.98 total_tx_L1 : 4.17 Gbps cpu_util. : 10.12% @ 2 cores (2 per dual port) total_rx : 1.41 Gbps rx_cpu_util. : 5.47% / 0 pps total_pps : 1.4 Mpps async_util. : 0% / 34.58 bps drop_rate : 2.54 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 10.12% | 10.12% | – | | | Tx bps L2 | 1.97 Gbps | 1.98 Gbps | 3.95 Gbps Tx bps L1 | 2.08 Gbps | 2.09 Gbps | 4.17 Gbps Tx pps | 697.28 Kpps | 698.79 Kpps | 1.4 Mpps Line Util. | 5.21 % | 5.22 % | — | | | Rx bps | 704.24 Mbps | 703.04 Mbps | 1.41 Gbps Rx pps | 247.81 Kpps | 248.36 Kpps | 496.17 Kpps ---- | | | opackets | 11321860 | 11419036 | 22740896 ipackets | 4065024 | 4100397 | 8165421 obytes | 4002278236 | 4036629952 | 8038908188 ibytes | 1453675184 | 1455668512 | 2909343696 tx-pkts | 11.32 Mpkts | 11.42 Mpkts | 22.74 Mpkts rx-pkts | 4.07 Mpkts | 4.1 Mpkts | 8.17 Mpkts tx-bytes | 4 GB | 4.04 GB | 8.04 GB rx-bytes | 1.45 GB | 1.46 GB | 2.91 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

mode == iptfs (i.e., normal ipsec) 1G 40 octet packet

mode == tunnel (i.e., normal ipsec) 1G 40 octet packet

tunnel 1G 40b pkt
Global Statistics

connection : localhost, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 36.57% @ 2 cores (2 per dual port) total_rx : 494 Mbps rx_cpu_util. : 13.65% / 0 pps total_pps : 5.86 Mpps async_util. : 0% / 35.1 bps drop_rate : 2.51 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 36.57% | 36.57% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.93 % | 4.92 % | — | | | Rx bps | 250.51 Mbps | 243.49 Mbps | 494 Mbps Rx pps | 489.27 Kpps | 475.57 Kpps | 964.84 Kpps ---- | | | opackets | 89007203 | 89443956 | 178451159 ipackets | 14835660 | 14522048 | 29357708 obytes | 5696460992 | 5724413184 | 11420874176 ibytes | 949482112 | 929411072 | 1878893184 tx-pkts | 89.01 Mpkts | 89.44 Mpkts | 178.45 Mpkts rx-pkts | 14.84 Mpkts | 14.52 Mpkts | 29.36 Mpkts tx-bytes | 5.7 GB | 5.72 GB | 11.42 GB rx-bytes | 949.48 MB | 929.41 MB | 1.88 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

browse: ‘q’ - quit, ‘d’ - dashboard, ‘u’ - util, ‘s’ - streams, ‘l’ - latency, dashboard: ‘n’ - reset view, ‘o’ - owned ports, ‘a’ - all ports, ‘c’ - clear,

More performance

routed, no tunnel

UPKT 1400, rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.17 Gbps version : STL @ v2.98 total_tx_L1 : 15.39 Gbps cpu_util. : 4.14% @ 4 cores (4 per dual port) total_rx : 14.14 Gbps rx_cpu_util. : 19.92% / 0 pps total_pps : 1.34 Mpps async_util. : 0% / 0 bps drop_rate : 0 bps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.14% | 4.14% | – | | | Tx bps L2 | 7.57 Gbps | 7.61 Gbps | 15.18 Gbps Tx bps L1 | 7.68 Gbps | 7.71 Gbps | 15.39 Gbps Tx pps | 667.53 Kpps | 670.59 Kpps | 1.34 Mpps Line Util. | 19.2 % | 19.29 % | — | | | Rx bps | 6.55 Gbps | 7.6 Gbps | 14.15 Gbps Rx pps | 577.24 Kpps | 670.25 Kpps | 1.25 Mpps ---- | | | opackets | 16176393 | 16229842 | 32406235 ipackets | 14131488 | 16224312 | 30355800 obytes | 22938123860 | 23013915956 | 45952039816 ibytes | 20038449984 | 23006072998 | 43044522982 tx-pkts | 16.18 Mpkts | 16.23 Mpkts | 32.41 Mpkts rx-pkts | 14.13 Mpkts | 16.22 Mpkts | 30.36 Mpkts tx-bytes | 22.94 GB | 23.01 GB | 45.95 GB rx-bytes | 20.04 GB | 23.01 GB | 43.04 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 1K, rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.35 Gbps version : STL @ v2.98 total_tx_L1 : 15.66 Gbps cpu_util. : 4.76% @ 4 cores (4 per dual port) total_rx : 11.28 Gbps rx_cpu_util. : 20.4% / 0 pps total_pps : 1.89 Mpps async_util. : 0% / 0 bps drop_rate : 4.08 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.76% | 4.76% | – | | | Tx bps L2 | 7.7 Gbps | 7.65 Gbps | 15.35 Gbps Tx bps L1 | 7.85 Gbps | 7.8 Gbps | 15.66 Gbps Tx pps | 945.93 Kpps | 939.49 Kpps | 1.89 Mpps Line Util. | 19.64 % | 19.5 % | — | | | Rx bps | 6.09 Gbps | 5.19 Gbps | 11.28 Gbps Rx pps | 747.53 Kpps | 637.13 Kpps | 1.38 Mpps ---- | | | opackets | 39428467 | 39494064 | 78922531 ipackets | 31190218 | 26685961 | 57876179 obytes | 40138179406 | 40204954110 | 80343133516 ibytes | 31751641924 | 27166307280 | 58917949204 tx-pkts | 39.43 Mpkts | 39.49 Mpkts | 78.92 Mpkts rx-pkts | 31.19 Mpkts | 26.69 Mpkts | 57.88 Mpkts tx-bytes | 40.14 GB | 40.2 GB | 80.34 GB rx-bytes | 31.75 GB | 27.17 GB | 58.92 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: /

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 512 rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 15.51 Gbps version : STL @ v2.98 total_tx_L1 : 16.1 Gbps cpu_util. : 4.93% @ 4 cores (4 per dual port) total_rx : 3.21 Gbps rx_cpu_util. : 10.02% / 0 pps total_pps : 3.66 Mpps async_util. : 0% / 0 bps drop_rate : 12.3 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.93% | 4.93% | – | | | Tx bps L2 | 7.79 Gbps | 7.72 Gbps | 15.51 Gbps Tx bps L1 | 8.09 Gbps | 8.01 Gbps | 16.1 Gbps Tx pps | 1.84 Mpps | 1.82 Mpps | 3.66 Mpps Line Util. | 20.22 % | 20.03 % | — | | | Rx bps | 1.61 Gbps | 1.6 Gbps | 3.21 Gbps Rx pps | 380.82 Kpps | 377.25 Kpps | 758.07 Kpps ---- | | | opackets | 63793853 | 63793814 | 127587667 ipackets | 13199104 | 13199116 | 26398220 obytes | 33810742616 | 33810724050 | 67621466666 ibytes | 6995525120 | 6995532010 | 13991057130 tx-pkts | 63.79 Mpkts | 63.79 Mpkts | 127.59 Mpkts rx-pkts | 13.2 Mpkts | 13.2 Mpkts | 26.4 Mpkts tx-bytes | 33.81 GB | 33.81 GB | 67.62 GB rx-bytes | 7 GB | 7 GB | 13.99 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

UPKT 256 rate=8G, 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 16.08 Gbps version : STL @ v2.98 total_tx_L1 : 17.26 Gbps cpu_util. : 7.38% @ 4 cores (4 per dual port) total_rx : 1.66 Gbps rx_cpu_util. : 10.18% / 0 pps total_pps : 7.34 Mpps async_util. : 0% / 0 bps drop_rate : 14.42 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 7.38% | 7.38% | – | | | Tx bps L2 | 8.04 Gbps | 8.04 Gbps | 16.08 Gbps Tx bps L1 | 8.63 Gbps | 8.63 Gbps | 17.26 Gbps Tx pps | 3.67 Mpps | 3.67 Mpps | 7.34 Mpps Line Util. | 21.57 % | 21.57 % | — | | | Rx bps | 830.78 Mbps | 830.68 Mbps | 1.66 Gbps Rx pps | 379.01 Kpps | 378.96 Kpps | 757.97 Kpps ---- | | | opackets | 105193737 | 105523395 | 210717132 ipackets | 10867507 | 10901568 | 21769075 obytes | 28823082318 | 28913409150 | 57736491468 ibytes | 2977696918 | 2987029632 | 5964726550 tx-pkts | 105.19 Mpkts | 105.52 Mpkts | 210.72 Mpkts rx-pkts | 10.87 Mpkts | 10.9 Mpkts | 21.77 Mpkts tx-bytes | 28.82 GB | 28.91 GB | 57.74 GB rx-bytes | 2.98 GB | 2.99 GB | 5.96 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

iptfs

1400UPKT 5G rate 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 9.47 Gbps version : STL @ v2.98 total_tx_L1 : 9.61 Gbps cpu_util. : 1.25% @ 4 cores (4 per dual port) total_rx : 2.86 Gbps rx_cpu_util. : 3.06% / 0 pps total_pps : 835.12 Kpps async_util. : 0% / 7.62 bps drop_rate : 6.62 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 1.25% | 1.25% | – | | | Tx bps L2 | 4.72 Gbps | 4.75 Gbps | 9.47 Gbps Tx bps L1 | 4.79 Gbps | 4.82 Gbps | 9.6 Gbps Tx pps | 416.15 Kpps | 418.74 Kpps | 834.89 Kpps Line Util. | 11.97 % | 12.04 % | — | | | Rx bps | 1.4 Gbps | 1.46 Gbps | 2.85 Gbps Rx pps | 123.1 Kpps | 128.47 Kpps | 251.58 Kpps ---- | | | opackets | 17519987 | 17561715 | 35081702 ipackets | 5292758 | 5474423 | 10767181 obytes | 24843341566 | 24902511870 | 49745853436 ibytes | 7505130844 | 7762730396 | 15267861240 tx-pkts | 17.52 Mpkts | 17.56 Mpkts | 35.08 Mpkts rx-pkts | 5.29 Mpkts | 5.47 Mpkts | 10.77 Mpkts tx-bytes | 24.84 GB | 24.9 GB | 49.75 GB rx-bytes | 7.51 GB | 7.76 GB | 15.27 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18908723 (75.29220704883792%) p1missed: 18699872 (74.46058807941534%)

ipsec

1400UPKT 5G rate 4core

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 9.48 Gbps version : STL @ v2.98 total_tx_L1 : 9.61 Gbps cpu_util. : 1.17% @ 4 cores (4 per dual port) total_rx : 2.47 Gbps rx_cpu_util. : 2.57% / 0 pps total_pps : 835.27 Kpps async_util. : 0% / 0 bps drop_rate : 7.01 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 1.17% | 1.17% | – | | | Tx bps L2 | 4.75 Gbps | 4.74 Gbps | 9.49 Gbps Tx bps L1 | 4.82 Gbps | 4.8 Gbps | 9.62 Gbps Tx pps | 418.75 Kpps | 417.45 Kpps | 836.21 Kpps Line Util. | 12.04 % | 12.01 % | — | | | Rx bps | 1.24 Gbps | 1.23 Gbps | 2.47 Gbps Rx pps | 109.15 Kpps | 108.8 Kpps | 217.94 Kpps ---- | | | opackets | 10810806 | 10844294 | 21655100 ipackets | 2844712 | 2853184 | 5697896 obytes | 15329724322 | 15377208892 | 30706933214 ibytes | 4033801616 | 4045814912 | 8079616528 tx-pkts | 10.81 Mpkts | 10.84 Mpkts | 21.66 Mpkts rx-pkts | 2.84 Mpkts | 2.85 Mpkts | 5.7 Mpkts tx-bytes | 15.33 GB | 15.38 GB | 30.71 GB rx-bytes | 4.03 GB | 4.05 GB | 8.08 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 18559218 (73.90052116795617%) p1missed: 18557335 (73.89302329378069%)

but then another run got:

and was showing more like 2G

FAILED tests/stress/test_stress_phy.py::test_policy_small_pkt - Exception: FAILED: p0missed: 14448450 (57.53194908692576%) p1missed: 14270021 (56.82146677611519%)

ipsec tunnel

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 4.32% @ 4 cores (4 per dual port) total_rx : 147.22 Mbps rx_cpu_util. : 3.51% / 0 pps total_pps : 5.86 Mpps async_util. : 0% / 0 bps drop_rate : 2.85 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.32% | 4.32% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.92 % | 4.92 % | — | | | Rx bps | 73.59 Mbps | 73.63 Mbps | 147.22 Mbps Rx pps | 143.74 Kpps | 143.8 Kpps | 287.54 Kpps ---- | | | opackets | 595000923 | 595235414 | 1190236337 ipackets | 29586114 | 29596946 | 59183060 obytes | 38080059072 | 38095066496 | 76175125568 ibytes | 1893511296 | 1894204544 | 3787715840 tx-pkts | 595 Mpkts | 595.24 Mpkts | 1.19 Gpkts rx-pkts | 29.59 Mpkts | 29.6 Mpkts | 59.18 Mpkts tx-bytes | 38.08 GB | 38.1 GB | 76.18 GB rx-bytes | 1.89 GB | 1.89 GB | 3.79 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: |

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

R1 HTOP

0[||| 2.7%] Tasks: 17, 0 thr, 67 kthr; 2 running 1[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Load average: 0.99 0.59 0.25 2[ 0.0%] Uptime: 00:04:31 Mem[||||||| 55.0M/3.83G] Swp[ 0K/0K]

[Main] [I/O] PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command 256 root 20 0 3164 2272 2116 S 0.7 0.1 0:00.02 /usr/sbin/dropbear -R -2 260 root 20 0 4356 3132 2572 R 0.7 0.1 0:00.34 htop 1 root 20 0 3432 648 576 S 0.0 0.0 0:00.15 init 142 root 20 0 3432 520 460 S 0.0 0.0 0:00.00 /sbin/syslogd -n 146 root 20 0 3432 492 428 S 0.0 0.0 0:00.00 /sbin/klogd -n 171 root 20 0 3164 1768 1636 S 0.0 0.0 0:00.00 /usr/sbin/dropbear -R 172 root 20 0 3432 512 444 S 0.0 0.0 0:00.00 /sbin/getty -L console 0 vt100 173 root 20 0 4528 3620 3268 S 0.0 0.1 0:00.00 /bin/sh – 175 root 20 0 3432 524 460 S 0.0 0.0 0:00.00 /sbin/getty -L ttyS0 115200 vt100 176 root 20 0 4660 4020 3472 S 0.0 0.1 0:00.04 -bash 177 root 20 0 4528 3604 3256 S 0.0 0.1 0:00.00 /bin/sh – 178 root 20 0 4528 3656 3308 S 0.0 0.1 0:00.00 /bin/sh – 254 root 20 0 3164 2276 2120 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 255 root 20 0 20904 14368 9300 S 0.0 0.4 0:00.21 perf record -F 997 -a -g -o /tmp/perf.data – sleep 1200 257 root 20 0 4660 3740 3396 S 0.0 0.1 0:00.00 bash 258 root 20 0 3300 508 448 S 0.0 0.0 0:00.00 sleep 1200 283 root 20 0 3432 524 456 S 0.0 0.0 0:00.00 /sbin/getty -n -l /bin/sh -L ttyS3 115200 vt100

F

IPTFS

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 2.98 Gbps version : STL @ v2.98 total_tx_L1 : 3.91 Gbps cpu_util. : 4.91% @ 4 cores (4 per dual port) total_rx : 275.2 Mbps rx_cpu_util. : 7.96% / 0 pps total_pps : 5.82 Mpps async_util. : 0% / 0 bps drop_rate : 2.7 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 4.91% | 4.91% | – | | | Tx bps L2 | 1.48 Gbps | 1.5 Gbps | 2.98 Gbps Tx bps L1 | 1.94 Gbps | 1.97 Gbps | 3.91 Gbps Tx pps | 2.89 Mpps | 2.93 Mpps | 5.82 Mpps Line Util. | 4.86 % | 4.91 % | — | | | Rx bps | 136.81 Mbps | 138.39 Mbps | 275.2 Mbps Rx pps | 267.2 Kpps | 270.29 Kpps | 537.49 Kpps ---- | | | opackets | 47519368 | 47490145 | 95009513 ipackets | 4431611 | 4424832 | 8856443 obytes | 3041239252 | 3039369040 | 6080608292 ibytes | 283623104 | 283189248 | 566812352 tx-pkts | 47.52 Mpkts | 47.49 Mpkts | 95.01 Mpkts rx-pkts | 4.43 Mpkts | 4.42 Mpkts | 8.86 Mpkts tx-bytes | 3.04 GB | 3.04 GB | 6.08 GB rx-bytes | 283.62 MB | 283.19 MB | 566.81 MB ----- | | | oerrors | 0 | 0 | 0 ierrors | 0 | 0 | 0

status: \

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

0[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 17, 0 thr, 67 kthr; 2 running 1[||||||||||||||||||||||||||||||||||||||||||||||||||||||| 66.2%] Load average: 0.64 0.18 0.06 2[||||| 5.4%] Uptime: 00:01:05 Mem[||||| 54.2M/3.83G] Swp[ 0K/0K]

[Main] [I/O] PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command 1 root 20 0 3432 656 580 S 0.0 0.0 0:00.11 init 142 root 20 0 3432 520 460 S 0.0 0.0 0:00.00 /sbin/syslogd -n 146 root 20 0 3432 520 452 S 0.0 0.0 0:00.00 /sbin/klogd -n 171 root 20 0 3164 1788 1656 S 0.0 0.0 0:00.00 /usr/sbin/dropbear -R 172 root 20 0 3432 516 456 S 0.0 0.0 0:00.00 /sbin/getty -L console 0 vt100 173 root 20 0 4528 3588 3236 S 0.0 0.1 0:00.00 /bin/sh – 175 root 20 0 3432 516 452 S 0.0 0.0 0:00.00 /sbin/getty -L ttyS0 115200 vt100 176 root 20 0 4660 3844 3292 S 0.0 0.1 0:00.03 -bash 177 root 20 0 4528 3756 3408 S 0.0 0.1 0:00.00 /bin/sh – 178 root 20 0 4528 3576 3216 S 0.0 0.1 0:00.00 /bin/sh – 254 root 20 0 3164 2268 2112 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 255 root 20 0 20904 14340 9276 S 0.0 0.4 0:00.12 perf record -F 997 -a -g -o /tmp/perf.data – sleep 1200 256 root 20 0 3300 520 460 S 0.0 0.0 0:00.00 sleep 1200 257 root 20 0 3164 2236 2080 S 0.0 0.1 0:00.00 /usr/sbin/dropbear -R -2 258 root 20 0 4660 3676 3184 S 0.0 0.1 0:00.00 bash

Poor performance 150Kpps @ 40 w 4 cores:

Global Statistics

connection : 192.168.0.1, Port 4501 total_tx_L2 : 3 Gbps version : STL @ v2.98 total_tx_L1 : 3.94 Gbps cpu_util. : 13.12% @ 2 cores (2 per dual port) total_rx : 144.04 Mbps rx_cpu_util. : 9.97% / 0 pps total_pps : 5.87 Mpps async_util. : 0% / 23.93 bps drop_rate : 2.86 Gbps total_cps. : 0 cps queue_full : 0 pkts

Port Statistics

port | 0 | 1 | total -----------+-------------------+-------------------+------------------ owner | root | root | link | UP | UP | state | TRANSMITTING | TRANSMITTING | speed | 40 Gb/s | 40 Gb/s | CPU util. | 13.12% | 13.12% | – | | | Tx bps L2 | 1.5 Gbps | 1.5 Gbps | 3 Gbps Tx bps L1 | 1.97 Gbps | 1.97 Gbps | 3.94 Gbps Tx pps | 2.93 Mpps | 2.93 Mpps | 5.86 Mpps Line Util. | 4.92 % | 4.93 % | — | | | Rx bps | 71.98 Mbps | 72.02 Mbps | 144 Mbps Rx pps | 140.58 Kpps | 140.66 Kpps | 281.24 Kpps ---- | | | opackets | 1380900371 | 1381337221 | 2762237592 ipackets | 69166674 | 69183583 | 138350257 obytes | 88377623744 | 88405582144 | 176783205888 ibytes | 4426667136 | 4427749312 | 8854416448 tx-pkts | 1.38 Gpkts | 1.38 Gpkts | 2.76 Gpkts rx-pkts | 69.17 Mpkts | 69.18 Mpkts | 138.35 Mpkts tx-bytes | 88.38 GB | 88.41 GB | 176.78 GB rx-bytes | 4.43 GB | 4.43 GB | 8.85 GB ----- | | | oerrors | 0 | 0 | 0 ierrors | 1,052 | 1,116 | 2,168

status: -

Press ‘ESC’ for navigation panel… status:

tui(read-only)>

bash-5.1# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 14 0 0 0 IO-APIC 2-edge timer 1: 0 9 0 0 IO-APIC 1-edge i8042 3: 452 0 0 0 IO-APIC 3-edge ttyS1 4: 0 0 39 0 IO-APIC 4-edge ttyS0 8: 0 0 1 0 IO-APIC 8-edge rtc0 9: 0 0 0 0 IO-APIC 9-fasteoi acpi 12: 125 0 0 0 IO-APIC 12-edge i8042 16: 0 0 0 0 IO-APIC 16-fasteoi i801_smbus 24: 0 0 0 0 PCI-MSI 32768-edge virtio0-config 25: 0 0 0 26 PCI-MSI 32769-edge virtio0-virtqueues 26: 20 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2] 27: 0 0 0 0 PCI-MSI 49152-edge virtio1-config 28: 0 0 116 0 PCI-MSI 49153-edge virtio1-input.0 29: 0 0 0 126 PCI-MSI 49154-edge virtio1-output.0 30: 0 41 0 0 PCI-MSI 65536-edge eth1-TxRx-0 31: 0 0 147347 0 PCI-MSI 65537-edge eth1-TxRx-1 32: 0 0 0 147444 PCI-MSI 65538-edge eth1-TxRx-2 33: 39 0 0 0 PCI-MSI 65539-edge eth1-TxRx-3 34: 0 2 0 0 PCI-MSI 65540-edge eth1 35: 0 0 0 0 PCI-MSI 98304-edge virtio2-config 36: 0 3 0 0 PCI-MSI 98305-edge virtio2-requests 37: 0 0 0 42 PCI-MSI 81920-edge iavf-0000:00:05.0:mbx 38: 0 0 2 0 PCI-MSI 81921-edge iavf-eth2-TxRx-0 39: 0 0 0 333179 PCI-MSI 81922-edge iavf-eth2-TxRx-1 40: 0 0 0 0 PCI-MSI 81923-edge iavf-eth2-TxRx-2 41: 0 0 0 0 PCI-MSI 81924-edge iavf-eth2-TxRx-3 NMI: 0 0 0 0 Non-maskable interrupts LOC: 70266 70258 143816 70294 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 13 20 21 18 Rescheduling interrupts CAL: 72 103 73 88 Function call interrupts TLB: 1 2 1 1 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 Machine check polls HYP: 1 1 1 1 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event bash-5.1#

bash-5.1# htop bash-5.1# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 14 0 0 0 IO-APIC 2-edge timer 1: 0 9 0 0 IO-APIC 1-edge i8042 3: 606 0 0 0 IO-APIC 3-edge ttyS1 4: 0 0 49 0 IO-APIC 4-edge ttyS0 8: 0 0 1 0 IO-APIC 8-edge rtc0 9: 0 0 0 0 IO-APIC 9-fasteoi acpi 12: 125 0 0 0 IO-APIC 12-edge i8042 16: 0 0 0 0 IO-APIC 16-fasteoi i801_smbus 24: 0 0 0 0 PCI-MSI 32768-edge virtio0-config 25: 0 0 0 26 PCI-MSI 32769-edge virtio0-virtqueues 26: 29 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2] 27: 0 0 0 0 PCI-MSI 49152-edge virtio1-config 28: 0 0 167 0 PCI-MSI 49153-edge virtio1-input.0 29: 0 0 0 123 PCI-MSI 49154-edge virtio1-output.0 30: 0 46 0 0 PCI-MSI 65536-edge eth1-TxRx-0 31: 0 0 46 0 PCI-MSI 65537-edge eth1-TxRx-1 32: 0 0 0 161945 PCI-MSI 65538-edge eth1-TxRx-2 33: 193538 0 0 0 PCI-MSI 65539-edge eth1-TxRx-3 34: 0 3 0 0 PCI-MSI 65540-edge eth1 35: 0 0 0 0 PCI-MSI 98304-edge virtio2-config 36: 0 3 0 0 PCI-MSI 98305-edge virtio2-requests 37: 0 0 0 46 PCI-MSI 81920-edge iavf-0000:00:05.0:mbx 38: 0 0 2 0 PCI-MSI 81921-edge iavf-eth2-TxRx-0 39: 0 0 0 1 PCI-MSI 81922-edge iavf-eth2-TxRx-1 40: 364801 0 0 0 PCI-MSI 81923-edge iavf-eth2-TxRx-2 41: 0 0 0 0 PCI-MSI 81924-edge iavf-eth2-TxRx-3 NMI: 0 0 0 0 Non-maskable interrupts LOC: 76411 76355 76435 172972 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 14 9 34 23 Rescheduling interrupts CAL: 75 107 97 67 Function call interrupts TLB: 2 2 1 2 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 Machine check polls HYP: 1 1 1 1 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event b but even moving the IRQ didn’t change anything maybe b/c rx-2 is being shared?

New Perfs