failed to run ./nettrace -p icmp in loongarch 4.19 kernel

[georgejguo]

nettrace version： v1.2.6.
cpu：loongarch
kernel：4.19 version

./nettrace -p icmp --debug

DEBUG: command: mount | grep debugfs, status:0
DEBUG: command: cat /sys/kernel/debug/tracing/events/skb/kfree_skb/format 2>/dev/null | grep NOT_SPECIFIED, status:0
DEBUG: trace name=tcp_ack_probe, prog=__trace_tcp_ack_probe is made invalid for: sock or sk mode
DEBUG: trace name=tcp_probe_timer, prog=__trace_tcp_probe_timer is made invalid for: sock or sk mode
DEBUG: trace name=tcp_send_probe0, prog=__trace_tcp_send_probe0 is made invalid for: sock or sk mode
DEBUG: trace name=__tcp_transmit_skb, prog=__trace___tcp_transmit_skb is made invalid for: sock or sk mode
DEBUG: trace name=tcp_sendmsg_locked, prog=__trace_tcp_sendmsg_locked is made invalid for: sock or sk mode
DEBUG: trace name=__tcp_push_pending_frames, prog=__trace___tcp_push_pending_frames is made invalid for: sock or sk mode
DEBUG: trace name=inet_listen, prog=__trace_inet_listen is made invalid for: sock or sk mode
DEBUG: trace name=tcp_v4_destroy_sock, prog=__trace_tcp_v4_destroy_sock is made invalid for: sock or sk mode
DEBUG: trace name=tcp_close, prog=__trace_tcp_close is made invalid for: sock or sk mode
DEBUG: trace name=tcp_write_timer_handler, prog=__trace_tcp_write_timer_handler is made invalid for: sock or sk mode
DEBUG: trace name=tcp_retransmit_timer, prog=__trace_tcp_retransmit_timer is made invalid for: sock or sk mode
DEBUG: trace name=tcp_enter_recovery, prog=__trace_tcp_enter_recovery is made invalid for: sock or sk mode
DEBUG: trace name=tcp_enter_loss, prog=__trace_tcp_enter_loss is made invalid for: sock or sk mode
DEBUG: trace name=tcp_try_keep_open, prog=__trace_tcp_try_keep_open is made invalid for: sock or sk mode
DEBUG: trace name=tcp_enter_cwr, prog=__trace_tcp_enter_cwr is made invalid for: sock or sk mode
DEBUG: trace name=tcp_fastretrans_alert, prog=__trace_tcp_fastretrans_alert is made invalid for: sock or sk mode
DEBUG: trace name=tcp_rearm_rto, prog=__trace_tcp_rearm_rto is made invalid for: sock or sk mode
DEBUG: trace name=tcp_event_new_data_sent, prog=__trace_tcp_event_new_data_sent is made invalid for: sock or sk mode
DEBUG: trace name=tcp_schedule_loss_probe, prog=__trace_tcp_schedule_loss_probe is made invalid for: sock or sk mode
DEBUG: command: verlte() { [ "$1" = "$2" ] && echo 0 && return; [ "$1" = "$(/bin/echo -e "$1\n$2" | sort -V | head -n1)" ] && echo -1 && return; echo 1; }; [ ! $(verlte "$(uname -r)" "5.4") -eq -1 ], status:1
DEBUG: trace name=__netif_receive_skb_core, prog=__trace___netif_receive_skb_core_pskb is made invalid for: cond
DEBUG: command: verlte() { [ "$1" = "$2" ] && echo 0 && return; [ "$1" = "$(/bin/echo -e "$1\n$2" | sort -V | head -n1)" ] && echo -1 && return; echo 1; }; [ $(verlte "$(uname -r)" "5.16") -eq -1 ], status:0
DEBUG: begin to resolve kernel symbol...
kernel function netif_receive_generic_xdp not founded, skipped
DEBUG: trace name=netif_receive_generic_xdp, prog=__trace_netif_receive_generic_xdp is made invalid for: not found
kernel function cls_bpf_classify not founded, skipped
DEBUG: trace name=cls_bpf_classify, prog=__trace_cls_bpf_classify is made invalid for: not found
kernel function tcf_bpf_act not founded, skipped
DEBUG: trace name=tcf_bpf_act, prog=__trace_tcf_bpf_act is made invalid for: not found
kernel function bond_dev_queue_xmit not founded, skipped
DEBUG: trace name=bond_dev_queue_xmit, prog=__trace_bond_dev_queue_xmit is made invalid for: not found
kernel function nft_do_chain not founded, skipped
DEBUG: trace name=nft_do_chain, prog=__trace_nft_do_chain is made invalid for: not found
DEBUG: ip_rcv_core.isra.2 is made manual attach
DEBUG: ip6_rcv_core.isra.2 is made manual attach
kernel function ip_finish_output_gso not founded, skipped
DEBUG: trace name=ip_finish_output_gso, prog=__trace_ip_finish_output_gso is made invalid for: not found
DEBUG: xfrm_output_gso.isra.1 is made manual attach
kernel function xfrm4_transport_output not founded, skipped
DEBUG: trace name=xfrm4_transport_output, prog=__trace_xfrm4_transport_output is made invalid for: not found
kernel function xfrm4_policy_check not founded, skipped
DEBUG: trace name=xfrm4_policy_check, prog=__trace_xfrm4_policy_check is made invalid for: not found
kernel function xfrm4_transport_input not founded, skipped
DEBUG: trace name=xfrm4_transport_input, prog=__trace_xfrm4_transport_input is made invalid for: not found
kernel function ah_output not founded, skipped
DEBUG: trace name=ah_output, prog=__trace_ah_output is made invalid for: not found
kernel function esp_output not founded, skipped
DEBUG: trace name=esp_output, prog=__trace_esp_output is made invalid for: not found
kernel function esp_output_tail not founded, skipped
DEBUG: trace name=esp_output_tail, prog=__trace_esp_output_tail is made invalid for: not found
kernel function ah_input not founded, skipped
DEBUG: trace name=ah_input, prog=__trace_ah_input is made invalid for: not found
kernel function esp_input not founded, skipped
DEBUG: trace name=esp_input, prog=__trace_esp_input is made invalid for: not found
DEBUG: udp_unicast_rcv_skb.isra.14 is made manual attach
kernel function __udp_queue_rcv_skb not founded, skipped
DEBUG: trace name=__udp_queue_rcv_skb, prog=__trace___udp_queue_rcv_skb is made invalid for: not found
DEBUG: icmp_reply.constprop.12 is made manual attach
kernel function __ping_queue_rcv_skb not founded, skipped
DEBUG: trace name=__ping_queue_rcv_skb, prog=__trace___ping_queue_rcv_skb is made invalid for: not found
DEBUG: ping_lookup.isra.3 is made manual attach
DEBUG: finished to resolve kernel symbol
DEBUG: kernel feature probe begin
DEBUG: kernel feature probe end
DEBUG: backup: valid prog for __netif_receive_skb_core is __trace___netif_receive_skb_core
DEBUG: backup: valid prog for ipt_do_table is __trace_ipt_do_table_legacy
DEBUG: trace name=ipt_do_table, prog=__trace_ipt_do_table is made invalid for: backup
following traces are enabled and valid:
tracepoint: napi_gro_receive_entry, prog: __trace_napi_gro_receive_entry
kprobe: dev_gro_receive, prog: __trace_dev_gro_receive
kprobe: enqueue_to_backlog, prog: __trace_enqueue_to_backlog
kprobe: xdp_do_generic_redirect, prog: __trace_xdp_do_generic_redirect
kprobe: __netif_receive_skb_core, prog: __trace___netif_receive_skb_core
kprobe: __dev_queue_xmit, prog: __trace___dev_queue_xmit
kprobe: dev_hard_start_xmit, prog: __trace_dev_hard_start_xmit
kprobe: pfifo_enqueue, prog: __trace_pfifo_enqueue
kprobe: sch_direct_xmit, prog: __trace_sch_direct_xmit
kprobe: tcf_classify, prog: __trace_tcf_classify
kprobe: arp_rcv, prog: __trace_arp_rcv
kprobe: arp_process, prog: __trace_arp_process
kprobe: packet_rcv, prog: __trace_packet_rcv
kprobe: tpacket_rcv, prog: __trace_tpacket_rcv
kprobe: packet_direct_xmit, prog: __trace_packet_direct_xmit
kprobe: nf_nat_manip_pkt, prog: __trace_nf_nat_manip_pkt
kprobe: nf_hook_slow, prog: __trace_nf_hook_slow
kprobe: ipt_do_table, prog: __trace_ipt_do_table_legacy
kprobe: tcp_in_window, prog: __trace_tcp_in_window
kprobe: ip_rcv, prog: __trace_ip_rcv
kprobe: ip_rcv_core.isra.2, prog: __trace_ip_rcv_core
kprobe: ip_rcv_finish, prog: __trace_ip_rcv_finish
kprobe: ip_local_deliver, prog: __trace_ip_local_deliver
kprobe: ip_local_deliver_finish, prog: __trace_ip_local_deliver_finish
kprobe: ip_forward, prog: __trace_ip_forward
kprobe: ip_forward_finish, prog: __trace_ip_forward_finish
kprobe: ip6_forward, prog: __trace_ip6_forward
kprobe: ip6_rcv_finish, prog: __trace_ip6_rcv_finish
kprobe: ip6_rcv_core.isra.2, prog: __trace_ip6_rcv_core
kprobe: ipv6_rcv, prog: __trace_ipv6_rcv
kprobe: ip_output, prog: __trace_ip_output
kprobe: ip_finish_output, prog: __trace_ip_finish_output
kprobe: ip_finish_output2, prog: __trace_ip_finish_output2
kprobe: ip6_output, prog: __trace_ip6_output
kprobe: ip6_finish_output, prog: __trace_ip6_finish_output
kprobe: ip6_finish_output2, prog: __trace_ip6_finish_output2
kprobe: ip6_send_skb, prog: __trace_ip6_send_skb
kprobe: ip6_local_out, prog: __trace_ip6_local_out
kprobe: xfrm4_output, prog: __trace_xfrm4_output
kprobe: xfrm_output, prog: __trace_xfrm_output
kprobe: xfrm_output2, prog: __trace_xfrm_output2
kprobe: xfrm_output_gso.isra.1, prog: __trace_xfrm_output_gso
kprobe: xfrm_output_resume, prog: __trace_xfrm_output_resume
kprobe: xfrm4_prepare_output, prog: __trace_xfrm4_prepare_output
kprobe: xfrm4_rcv, prog: __trace_xfrm4_rcv
kprobe: xfrm_input, prog: __trace_xfrm_input
kprobe: fib_validate_source, prog: __trace_fib_validate_source
kprobe: ip_route_input_slow, prog: __trace_ip_route_input_slow
kprobe: tcp_v4_rcv, prog: __trace_tcp_v4_rcv
kprobe: tcp_v6_rcv, prog: __trace_tcp_v6_rcv
kprobe: tcp_filter, prog: __trace_tcp_filter
kprobe: tcp_child_process, prog: __trace_tcp_child_process
kprobe: tcp_v4_send_reset, prog: __trace_tcp_v4_send_reset
kprobe: tcp_v6_send_reset, prog: __trace_tcp_v6_send_reset
kprobe: tcp_v4_do_rcv, prog: __trace_tcp_v4_do_rcv
kprobe: tcp_v6_do_rcv, prog: __trace_tcp_v6_do_rcv
kprobe: tcp_rcv_established, prog: __trace_tcp_rcv_established
kprobe: tcp_rcv_state_process, prog: __trace_tcp_rcv_state_process
kprobe: tcp_queue_rcv, prog: __trace_tcp_queue_rcv
kprobe: tcp_ack, prog: __trace_tcp_ack
kprobe: __inet_lookup_listener, prog: __trace___inet_lookup_listener
kprobe: inet6_lookup_listener, prog: __trace_inet6_lookup_listener
kprobe: udp_rcv, prog: __trace_udp_rcv
kprobe: udp_unicast_rcv_skb.isra.14, prog: __trace_udp_unicast_rcv_skb
kprobe: udp_queue_rcv_skb, prog: __trace_udp_queue_rcv_skb
kprobe: xfrm4_udp_encap_rcv, prog: __trace_xfrm4_udp_encap_rcv
kprobe: xfrm4_rcv_encap, prog: __trace_xfrm4_rcv_encap
kprobe: __udp_enqueue_schedule_skb, prog: __trace___udp_enqueue_schedule_skb
kprobe: icmp_rcv, prog: __trace_icmp_rcv
kprobe: icmp_echo, prog: __trace_icmp_echo
kprobe: icmp_reply.constprop.12, prog: __trace_icmp_reply
kprobe: icmpv6_rcv, prog: __trace_icmpv6_rcv
kprobe: icmpv6_echo_reply, prog: __trace_icmpv6_echo_reply
kprobe: ping_rcv, prog: __trace_ping_rcv
kprobe: ping_queue_rcv_skb, prog: __trace_ping_queue_rcv_skb
kprobe: ping_lookup.isra.3, prog: __trace_ping_lookup
kprobe: __kfree_skb, prog: __trace___kfree_skb
kprobe/kretprobe: skb_clone, prog: __trace_skb_clone
tracepoint: consume_skb, prog: __trace_consume_skb
tracepoint: kfree_skb, prog: __trace_kfree_skb
libbpf: failed to find skeleton program 'ret__trace___netif_receive_skb_core_pskb'
ERROR: failed to open kprobe-based eBPF
ERROR: failed to load kprobe-based bpf

想知道这里为何libbpf: failed to find skeleton program 'ret__trace___netif_receive_skb_core_pskb'没有找到这个程序呢？

[menglongdong]

你这里用的是自己采用compat模式编译的nettrace吗？从报错信息上来看，我猜测是编译环境上所采用的libbpf和bpftool的版本的问题。

目前还没有loongarch架构的docker编译环境，这里你可以尝试使用较高版本的libbpf以及bpftool，看看能不能解决问题。

[georgejguo]

libbpf version： v1.1.0
bpftool：7.1.0
nettrace：1.2.6

$ make COMPAT=1 all
现在编译没有问题，

$ ./src/nettrace -p icmp --debug
ERROR: failed to load kprobe-based eBPF
ERROR: failed to load kprobe-based bpf
但是执行的时候报错：log如下：

DEBUG: prog: __trace_ah_output is madlibbpf: prog '__trace_dev_gro_receive': BPF program load failed: Permission denied
libbpf: prog '__trace_dev_gro_receive': -- BEGIN PROG LOAD LOG --
Unrecognized arg#0 type PTR
; DEFINE_ALL_PROBES(KPROBE_DEFAULT, TP_DEFAULT, FNC)
0: (bf) r8 = r1
; DEFINE_ALL_PROBES(KPROBE_DEFAULT, TP_DEFAULT, FNC)
1: (7b) *(u64 *)(r10 -200) = r8

...
805: (79) r1 = *(u64 *)(r6 +55)
misaligned value access off (0x0; 0x0)+0+55 size 8
processed 1461 insns (limit 1000000) max_states_per_insn 2 total_states 117 peak_states 117 mark_read 28
-- END PROG LOAD LOG --
libbpf: prog '__trace_dev_gro_receive': failed to load: -13
libbpf: failed to load object 'kprobe'
libbpf: failed to load BPF skeleton 'kprobe': -13

[menglongdong]

这个我先分析一下，感觉像是loongarch对BPF的支持问题

[georgejguo]

这个是完整的日志：
fail-log.txt
我主要是没有思路。不知道怎么debug了，应该和内核相关。