Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Warnings during h2 message exchanges #1387

Closed
4 of 5 tasks
vankoven opened this issue Mar 4, 2020 · 10 comments · Fixed by #1705
Closed
4 of 5 tasks

Warnings during h2 message exchanges #1387

vankoven opened this issue Mar 4, 2020 · 10 comments · Fixed by #1705
Assignees
@s0nx
Labels
bug h2
Milestone
0.7 - HTTP/2

Comments

@vankoven
Copy link
Contributor

vankoven commented Mar 4, 2020 •
edited by s0nx
Loading

Caught a few warnings during testing h2 with h2spec:

  • 1: Can be reproduced with ./h2spec http2/5.1.1/3 -h <host> -p 443 -t -k
[92286.039416] ------------[ cut here ]------------
[92286.041515] WARNING: CPU: 0 PID: 0 at /home/user/qtc/release/tempesta/tempesta_fw/http.c:5722 tfw_http_msg_process_generic+0x7d0/0xd10 [tempesta_fw]
[92286.047585] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92286.061263]  [last unloaded: tempesta_lib]
[92286.061982] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92286.063774] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92286.065379] task: ffffffff83e12480 task.stack: ffffffff83e00000
[92286.066766] RIP: 0010:tfw_http_msg_process_generic+0x7d0/0xd10 [tempesta_fw]
[92286.070566] RSP: 0018:ffff917165403910 EFLAGS: 00010246
[92286.071722] RAX: 0000000000000000 RBX: ffff9170da1821a0 RCX: 0000000000000000
[92286.073611] RDX: ffff917165403a20 RSI: 0000000000000000 RDI: ffff9170da1821a0
[92286.075043] RBP: 0000000000000000 R08: ffff917165403940 R09: ffff9170da182908
[92286.076385] R10: 00000000ffffffff R11: 000000000000001d R12: ffff9170da1828c0
[92286.077589] R13: 0000000000000000 R14: ffff9170da1829c8 R15: 0000000000000000
[92286.078756] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92286.080118] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92286.081061] CR2: 00007f5cd152c000 CR3: 000000000520a004 CR4: 00000000003606f0
[92286.082235] Call Trace:
[92286.082667]  <IRQ>
[92286.084194]  ? gcmaes_decrypt.constprop.13+0x11e/0x2a0 [aesni_intel]
[92286.085448]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92286.086582]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92286.087444]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92286.088410]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92286.089340]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92286.090138]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92286.090979]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92286.093377]  ? lock_timer_base+0x74/0x90
[92286.094168]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92286.095463]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92286.097068]  ? kvm_sched_clock_read+0x1a/0x30
[92286.099107]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92286.100439]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92286.102181]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92286.103946]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92286.107233]  tcp_rcv_established+0x4d2/0x570
[92286.109332]  tcp_v4_do_rcv+0x129/0x1d0
[92286.111040]  tcp_v4_rcv+0x947/0xa50
[92286.112838]  ip_local_deliver_finish+0x9a/0x1c0
[92286.114862]  ip_local_deliver+0x6b/0xe0
[92286.116076]  ? tcp_v4_early_demux+0x112/0x150
[92286.117433]  ? ip_rcv_finish+0x17a/0x400
[92286.118383]  ip_rcv+0x289/0x3c0
[92286.119066]  ? inet_del_offload+0x40/0x40
[92286.120930]  __netif_receive_skb_core+0x84f/0xb30
[92286.124403]  ? enqueue_entity+0x110/0x7a0
[92286.126422]  ? process_backlog+0xa3/0x160
[92286.127427]  process_backlog+0xa3/0x160
[92286.129167]  net_rx_action+0x28e/0x3f0
[92286.130859]  __do_softirq+0x10f/0x2a8
[92286.131956]  irq_exit+0xae/0xb0
[92286.132633]  call_function_single_interrupt+0x7d/0x90
[92286.134064]  </IRQ>
[92286.134666] RIP: 0010:native_safe_halt+0x2/0x10
[92286.136918] RSP: 0018:ffffffff83e03e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92286.139947] RAX: ffffffff8349a7a0 RBX: ffffffff83e12480 RCX: 0000000000000000
[92286.142009] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[92286.144613] RBP: 0000000000000000 R08: 0000000000000002 R09: 000000000000b7bf
[92286.146234] R10: ffffb17ec0663d78 R11: 0000000000000000 R12: ffffffff83e12480
[92286.147420] R13: ffffffff83e12480 R14: 0000000000000000 R15: 0000000000000000
[92286.150573]  ? __sched_text_end+0x3/0x3
[92286.152340]  default_idle+0x1a/0xf0
[92286.153914]  do_idle+0x16e/0x1f0
[92286.155376]  cpu_startup_entry+0x6f/0x80
[92286.157434]  start_kernel+0x467/0x487
[92286.158812]  secondary_startup_64+0xa5/0xb0
[92286.160053] Code: 00 f6 c4 08 0f 85 51 04 00 00 48 8b 44 24 10 48 89 43 40 49 89 c6 e9 de f9 ff ff 48 83 7f 08 00 0f 85 8a f8 ff ff e9 87 f8 ff ff <0f> 0b 41 bf ea ff ff ff e9 73 f9 ff ff 49 89 7c 24 10 48 89 3f 
[92286.165460] ---[ end trace ab4f9128609f842b ]---

[92308.274202] ------------[ cut here ]------------
[92308.275475] WARNING: CPU: 1 PID: 16 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:1199 tfw_hpack_decode+0x1c32/0x1cc0 [tempesta_fw]
[92308.278623] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.294190]  [last unloaded: tempesta_lib]
[92308.294924] CPU: 1 PID: 16 Comm: ksoftirqd/1 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.296969] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.298523] task: ffff9171415d3c00 task.stack: ffffb17ec06b0000
[92308.299771] RIP: 0010:tfw_hpack_decode+0x1c32/0x1cc0 [tempesta_fw]
[92308.300828] RSP: 0018:ffffb17ec06b36c0 EFLAGS: 00010206
[92308.301697] RAX: 0000000000000008 RBX: ffff9170e2511020 RCX: 0000000000000000
[92308.302886] RDX: 0000000000000000 RSI: 00000000000000e0 RDI: ffff9170dd5cf518
[92308.304062] RBP: ffffffffc08bcb00 R08: ffff9170e2eb802a R09: 0000000000000020
[92308.305817] R10: ffffffffc08bcda0 R11: 000000000000000a R12: ffff9170d713e6a8
[92308.307010] R13: 0000000000000000 R14: ffff9170d713e6a8 R15: ffff9170dd5cf4d0
[92308.308187] FS:  0000000000000000(0000) GS:ffff917165500000(0000) knlGS:0000000000000000
[92308.309617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.310565] CR2: 00007f5cd1539020 CR3: 000000000520a002 CR4: 00000000003606e0
[92308.311741] Call Trace:
[92308.312176]  ? tfw_h2_add_stream+0x5a/0xf0 [tempesta_fw]
[92308.313124]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.314147]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92308.315096]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.316598]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.317477]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92308.319349]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.321086]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.323719]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.326659]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.327982]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.329865]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.330764]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.331456]  ? lock_timer_base+0x74/0x90
[92308.332013]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.333345]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.335232]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.337458]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.338769]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.341303]  ? __skb_flow_dissect+0x659/0x10c0
[92308.342467]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.344613]  tcp_rcv_established+0x4d2/0x570
[92308.346180]  tcp_v4_do_rcv+0x129/0x1d0
[92308.347768]  tcp_v4_rcv+0x947/0xa50
[92308.348579]  ip_local_deliver_finish+0x9a/0x1c0
[92308.349812]  ip_local_deliver+0x6b/0xe0
[92308.350690]  ? tcp_v4_early_demux+0x112/0x150
[92308.352299]  ? ip_rcv_finish+0x17a/0x400
[92308.353356]  ip_rcv+0x289/0x3c0
[92308.354171]  ? inet_del_offload+0x40/0x40
[92308.356063]  __netif_receive_skb_core+0x84f/0xb30
[92308.357912]  ? vring_unmap_one+0x16/0x70 [virtio_ring]
[92308.359478]  ? virtqueue_napi_complete+0x39/0x70 [virtio_net]
[92308.360340]  ? process_backlog+0xa3/0x160
[92308.360930]  process_backlog+0xa3/0x160
[92308.361507]  net_rx_action+0x28e/0x3f0
[92308.362052]  __do_softirq+0x10f/0x2a8
[92308.362679]  run_ksoftirqd+0x1c/0x40
[92308.363754]  smpboot_thread_fn+0x10e/0x160
[92308.364496]  kthread+0xff/0x130
[92308.365048]  ? sort_range+0x20/0x20
[92308.365647]  ? kthread_create_on_node+0x70/0x70
[92308.366405]  ret_from_fork+0x35/0x40
[92308.367003] Code: 89 74 24 48 e9 4c fe ff ff 4d 89 e6 e9 e5 f3 ff ff 45 89 ea 49 89 ee 31 c9 e9 9b f6 ff ff 45 89 ea 31 c9 e9 98 fc ff ff 4d 89 e6 <0f> 0b 41 ba fe ff ff ff e9 75 ef ff ff 49 c1 e5 05 48 8b 74 24 
[92308.370154] ---[ end trace ab4f9128609f842c ]---
  • 3: Can be reproduced with ./h2spec http2/8.1.2.1/3 -h <host> -p 443 -t -k
[92308.408692] ------------[ cut here ]------------
[92308.410253] WARNING: CPU: 0 PID: 15856 at /home/user/qtc/release/tempesta/tempesta_fw/http_msg.c:528 tfw_http_msg_hdr_close+0x128/0x200 [tempesta_fw]
[92308.414407] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.427320]  [last unloaded: tempesta_lib]
[92308.428045] CPU: 0 PID: 15856 Comm: systemd-journal Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.430095] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.432347] task: ffff917133d1cb00 task.stack: ffffb17ec0a94000
[92308.433969] RIP: 0010:tfw_http_msg_hdr_close+0x128/0x200 [tempesta_fw]
[92308.435336] RSP: 0018:ffff917165403790 EFLAGS: 00010293
[92308.437801] RAX: 0000000000008800 RBX: ffff9170dd4e2680 RCX: 0000000000000000
[92308.440525] RDX: 000000000000000b RSI: 0000000000000040 RDI: ffff9170df80b020
[92308.443259] RBP: ffff9170df80b020 R08: ffff9171654038c4 R09: 0000000000000001
[92308.446107] R10: 00000000ffffffff R11: 000000000000000a R12: ffff9170d713c099
[92308.447880] R13: 0000000000000000 R14: ffff9170df80b2c0 R15: ffff9170df80b2c8
[92308.449896] FS:  00007f5cd3e7c8c0(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92308.451945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.453580] CR2: 00007f5cd153b000 CR3: 0000000081152001 CR4: 00000000003606f0
[92308.455056] Call Trace:
[92308.455612]  <IRQ>
[92308.456009]  tfw_hpack_decode+0xc2d/0x1cc0 [tempesta_fw]
[92308.456954]  ? gcmaes_decrypt.constprop.13+0x73/0x2a0 [aesni_intel]
[92308.458501]  ? __save_stack_trace+0x82/0x100
[92308.459339]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.460384]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92308.461277]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.462293]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.463143]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92308.465235]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.466305]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.467172]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.468128]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.469067]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.469874]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.470721]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.471621]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.472561]  ? ip_output+0x71/0xe0
[92308.473174]  ? ip_queue_xmit+0x5c/0x3b0
[92308.473835]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.474661]  ? kvm_sched_clock_read+0x1a/0x30
[92308.475460]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.476384]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.477364]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.478232]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.479632]  tcp_rcv_established+0x4d2/0x570
[92308.480501]  tcp_v4_do_rcv+0x129/0x1d0
[92308.481246]  tcp_v4_rcv+0x947/0xa50
[92308.481906]  ip_local_deliver_finish+0x9a/0x1c0
[92308.482771]  ip_local_deliver+0x6b/0xe0
[92308.483341]  ? tcp_v4_early_demux+0x112/0x150
[92308.484009]  ? ip_rcv_finish+0x17a/0x400
[92308.484613]  ip_rcv+0x289/0x3c0
[92308.485173]  ? inet_del_offload+0x40/0x40
[92308.486567]  __netif_receive_skb_core+0x84f/0xb30
[92308.488025]  ? tcp_write_xmit+0x2a7/0x11c0
[92308.489469]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[92308.491686]  ? process_backlog+0xa3/0x160
[92308.492666]  process_backlog+0xa3/0x160
[92308.493544]  net_rx_action+0x28e/0x3f0
[92308.494455]  __do_softirq+0x10f/0x2a8
[92308.495968]  irq_exit+0xae/0xb0
[92308.497052]  call_function_single_interrupt+0x7d/0x90
[92308.498692]  </IRQ>
[92308.500423] RIP: 0010:___bpf_prog_run+0x851/0x1270
[92308.501774] RSP: 0018:ffffb17ec0a97cd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92308.503917] RAX: ffffffff82f6ebd1 RBX: ffffb17ec0749068 RCX: 0000000000000000
[92308.505571] RDX: 0000000000000000 RSI: ffffb17ec0749030 RDI: 00000000c000003e
[92308.507677] RBP: ffffb17ec0a97d30 R08: 0000000000000000 R09: 0000000000000000
[92308.509918] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83a21c80
[92308.511673] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[92308.514356]  ? ___bpf_prog_run+0x851/0x1270
[92308.515945]  ? ___bpf_prog_run+0x851/0x1270
[92308.517570]  ? path_openat+0x725/0x1460
[92308.518877]  ? __bpf_prog_run32+0x33/0x50
[92308.520039]  ? filename_parentat+0x193/0x1b0
[92308.521641]  ? kernel_text_address+0xd4/0xe0
[92308.522801]  ? seccomp_run_filters+0x58/0xb0
[92308.523935]  ? do_filp_open+0xad/0x110
[92308.525740]  ? __seccomp_filter+0x43/0x510
[92308.527803]  ? create_object+0x249/0x300
[92308.529580]  ? __check_object_size+0xb3/0x190
[92308.531520]  ? syscall_trace_enter+0x117/0x2c0
[92308.532975]  ? do_syscall_64+0xf3/0x120
[92308.533612]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[92308.535412] Code: 74 3f 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 41 83 fd 0a 77 32 48 8b 87 98 00 00 00 f6 c4 08 74 06 41 83 fd 08 74 28 <0f> 0b 48 83 c4 08 b8 fe ff ff ff 5b 5d 41 5c 41 5d 41 5e 41 5f 
[92308.542976] ---[ end trace ab4f9128609f842d ]---
  • 4:
[92308.544840] ------------[ cut here ]------------
[92308.545935] WARNING: CPU: 0 PID: 15856 at /home/user/qtc/release/tempesta/tempesta_fw/http_stream.c:81 tfw_h2_stream_fsm+0x5f/0x250 [tempesta_fw]
[92308.548677] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.576894]  [last unloaded: tempesta_lib]
[92308.577910] CPU: 0 PID: 15856 Comm: systemd-journal Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.581157] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.584468] task: ffff917133d1cb00 task.stack: ffffb17ec0a94000
[92308.586079] RIP: 0010:tfw_h2_stream_fsm+0x5f/0x250 [tempesta_fw]
[92308.588556] RSP: 0018:ffff9171654036f0 EFLAGS: 00010202
[92308.590730] RAX: 0000000000000002 RBX: ffff9170dd4e2680 RCX: 0000000000000001
[92308.593355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9170dd4e26b0
[92308.594990] RBP: 0000000000000001 R08: ffff917165403744 R09: 0000000000000001
[92308.597271] R10: 00000000fffffffe R11: 000000000000000a R12: ffff9170dd4e26b0
[92308.600091] R13: ffff917165403744 R14: 0000000000000001 R15: 0000000000000001
[92308.602733] FS:  00007f5cd3e7c8c0(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92308.605873] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.607523] CR2: 00007f5cd153b000 CR3: 0000000081152001 CR4: 00000000003606f0
[92308.609962] Call Trace:
[92308.610834]  <IRQ>
[92308.611500]  tfw_h2_stream_id_close+0xbd/0x130 [tempesta_fw]
[92308.613525]  tfw_h2_send_resp+0x209/0x560 [tempesta_fw]
[92308.614998]  tfw_http_cli_error_resp_and_log+0x2a6/0x3f0 [tempesta_fw]
[92308.617120]  tfw_http_msg_process_generic+0x24b/0xd10 [tempesta_fw]
[92308.619229]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.621348]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.623220]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.625554]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.627792]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.629914]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.631888]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.633974]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.636239]  ? ip_output+0x71/0xe0
[92308.637665]  ? ip_queue_xmit+0x5c/0x3b0
[92308.638797]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.640371]  ? kvm_sched_clock_read+0x1a/0x30
[92308.642042]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.643372]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.645247]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.646535]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.648482]  tcp_rcv_established+0x4d2/0x570
[92308.650281]  tcp_v4_do_rcv+0x129/0x1d0
[92308.651848]  tcp_v4_rcv+0x947/0xa50
[92308.653317]  ip_local_deliver_finish+0x9a/0x1c0
[92308.655204]  ip_local_deliver+0x6b/0xe0
[92308.656468]  ? tcp_v4_early_demux+0x112/0x150
[92308.657850]  ? ip_rcv_finish+0x17a/0x400
[92308.659390]  ip_rcv+0x289/0x3c0
[92308.660753]  ? inet_del_offload+0x40/0x40
[92308.662408]  __netif_receive_skb_core+0x84f/0xb30
[92308.664041]  ? tcp_write_xmit+0x2a7/0x11c0
[92308.665785]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[92308.667863]  ? process_backlog+0xa3/0x160
[92308.669028]  process_backlog+0xa3/0x160
[92308.670149]  net_rx_action+0x28e/0x3f0
[92308.671405]  __do_softirq+0x10f/0x2a8
[92308.673343]  irq_exit+0xae/0xb0
[92308.674218]  call_function_single_interrupt+0x7d/0x90
[92308.675138]  </IRQ>
[92308.675643] RIP: 0010:___bpf_prog_run+0x851/0x1270
[92308.676544] RSP: 0018:ffffb17ec0a97cd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92308.677923] RAX: ffffffff82f6ebd1 RBX: ffffb17ec0749068 RCX: 0000000000000000
[92308.679128] RDX: 0000000000000000 RSI: ffffb17ec0749030 RDI: 00000000c000003e
[92308.681737] RBP: ffffb17ec0a97d30 R08: 0000000000000000 R09: 0000000000000000
[92308.684932] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83a21c80
[92308.687591] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[92308.689666]  ? ___bpf_prog_run+0x851/0x1270
[92308.690379]  ? ___bpf_prog_run+0x851/0x1270
[92308.691074]  ? path_openat+0x725/0x1460
[92308.692209]  ? __bpf_prog_run32+0x33/0x50
[92308.692929]  ? filename_parentat+0x193/0x1b0
[92308.693652]  ? kernel_text_address+0xd4/0xe0
[92308.694376]  ? seccomp_run_filters+0x58/0xb0
[92308.695101]  ? do_filp_open+0xad/0x110
[92308.695766]  ? __seccomp_filter+0x43/0x510
[92308.696518]  ? create_object+0x249/0x300
[92308.697268]  ? __check_object_size+0xb3/0x190
[92308.697989]  ? syscall_trace_enter+0x117/0x2c0
[92308.698958]  ? do_syscall_64+0xf3/0x120
[92308.699889]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[92308.700789] Code: c7 c2 45 84 ff 74 46 44 89 f2 83 e2 01 41 f6 c6 01 75 0a 40 80 fd 03 0f 85 f7 01 00 00 8b 43 2c 83 f8 05 7f 42 41 f6 c6 01 74 3c <0f> 0b be 03 00 00 00 4c 89 e7 c6 07 00 0f 1f 40 00 48 83 c4 08 
[92308.705372] ---[ end trace ab4f9128609f842e ]---
  • 5:
[92315.518389] ------------[ cut here ]------------
[92315.519660] WARNING: CPU: 0 PID: 7 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:678 tfw_huffman_decode+0x400/0x580 [tempesta_fw]
[92315.521791] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92315.535992]  [last unloaded: tempesta_lib]
[92315.536615] CPU: 0 PID: 7 Comm: ksoftirqd/0 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92315.538757] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92315.540680] task: ffff917141510f00 task.stack: ffffb17ec0660000
[92315.542231] RIP: 0010:tfw_huffman_decode+0x400/0x580 [tempesta_fw]
[92315.544094] RSP: 0018:ffffb17ec0663660 EFLAGS: 00010286
[92315.545925] RAX: 000000000000034e RBX: ffff9170df80b020 RCX: 0000000000000002
[92315.547085] RDX: 000000000000027f RSI: 000000000000fffa RDI: ffff9170df80b160
[92315.548449] RBP: ffff9170da180788 R08: 0000000000001000 R09: 0000000000000508
[92315.550519] R10: ffffb17ec0663660 R11: 0000000000000001 R12: 00000000fffffffa
[92315.552705] R13: 00000000fffffff9 R14: ffff9170d71cfeb5 R15: 0000000000000000
[92315.554893] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92315.557684] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92315.560548] CR2: 00007f5cd153b000 CR3: 000000000520a005 CR4: 00000000003606f0
[92315.563856] Call Trace:
[92315.564976]  tfw_hpack_decode+0x388/0x1cc0 [tempesta_fw]
[92315.567665]  ? tfw_h2_add_stream+0x5a/0xf0 [tempesta_fw]
[92315.568962]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92315.571487]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92315.572419]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92315.573424]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92315.574277]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92315.575998]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92315.577312]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92315.578186]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92315.579135]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92315.581173]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92315.583108]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92315.585878]  ? lock_timer_base+0x74/0x90
[92315.587503]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92315.589880]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92315.591428]  ? kvm_sched_clock_read+0x1a/0x30
[92315.592406]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92315.593627]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92315.595792]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92315.597060]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92315.598209]  tcp_rcv_established+0x4d2/0x570
[92315.600088]  tcp_v4_do_rcv+0x129/0x1d0
[92315.601105]  tcp_v4_rcv+0x947/0xa50
[92315.601931]  ip_local_deliver_finish+0x9a/0x1c0
[92315.603461]  ip_local_deliver+0x6b/0xe0
[92315.604661]  ? tcp_v4_early_demux+0x112/0x150
[92315.605592]  ? ip_rcv_finish+0x17a/0x400
[92315.607372]  ip_rcv+0x289/0x3c0
[92315.608114]  ? inet_del_offload+0x40/0x40
[92315.609901]  __netif_receive_skb_core+0x84f/0xb30
[92315.611689]  ? process_backlog+0xa3/0x160
[92315.612827]  process_backlog+0xa3/0x160
[92315.613937]  net_rx_action+0x28e/0x3f0
[92315.615406]  __do_softirq+0x10f/0x2a8
[92315.616682]  run_ksoftirqd+0x1c/0x40
[92315.617652]  smpboot_thread_fn+0x10e/0x160
[92315.619315]  kthread+0xff/0x130
[92315.620343]  ? sort_range+0x20/0x20
[92315.621933]  ? kthread_create_on_node+0x70/0x70
[92315.623498]  ret_from_fork+0x35/0x40
[92315.625109] Code: 5c fa ff 48 85 c0 0f 84 45 fe ff ff 48 8b 93 60 01 00 00 48 c7 40 10 01 00 00 00 48 89 10 48 83 83 50 01 00 00 01 e9 e4 fe ff ff <0f> 0b b8 fe ff ff ff e9 1f fe ff ff 48 83 7d 78 00 0f 85 64 fe
@vankoven vankoven added this to the 0.7 HTTP/2 milestone Mar 4, 2020
@vankoven vankoven self-assigned this Mar 4, 2020
@vankoven
Copy link
Contributor Author

vankoven commented Mar 4, 2020 •
edited by krizhanovsky
Loading

  • Also caught when configured Tempesta to stand before Buildbot:
[ 4557.232912] ------------[ cut here ]------------
[ 4557.235488] WARNING: CPU: 0 PID: 0 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:2392 tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[ 4557.239945] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[ 4557.266558]  [last unloaded: tempesta_lib]
[ 4557.268074] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[ 4557.272649] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[ 4557.276491] task: ffffffff83e12480 task.stack: ffffffff83e00000
[ 4557.278266] RIP: 0010:tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[ 4557.280814] RSP: 0018:ffff917165403748 EFLAGS: 00010293
[ 4557.282119] RAX: 0000000000000003 RBX: 0000000000000000 RCX: 0000000000000032
[ 4557.284592] RDX: ffff917165403890 RSI: 0000000000000032 RDI: 0000000000000035
[ 4557.286068] RBP: 0000000000000000 R08: 0000000000000003 R09: ffff917165403868
[ 4557.287926] R10: 0000000000000003 R11: ffff9171062e803a R12: 0000000000003530
[ 4557.292976] R13: 0000000000007400 R14: ffff91710679d020 R15: ffff917165403828
[ 4557.295560] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[ 4557.297592] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4557.298691] CR2: 0000557d663629f8 CR3: 000000000520a003 CR4: 00000000003606f0
[ 4557.300134] Call Trace:
[ 4557.300796]  <IRQ>
[ 4557.301283]  ? irq_work_interrupt+0x7d/0x90
[ 4557.302154]  tfw_h2_resp_status_write+0x100/0x250 [tempesta_fw]
[ 4557.303433]  tfw_h2_send_resp+0x150/0x560 [tempesta_fw]
[ 4557.305719]  tfw_http_cli_error_resp_and_log+0x2a6/0x3f0 [tempesta_fw]
[ 4557.308818]  ? kmem_cache_free+0x1a7/0x1d0
[ 4557.310749]  tfw_http_msg_process_generic+0x150/0xd10 [tempesta_fw]
[ 4557.313608]  ? native_safe_halt+0x2/0x10
[ 4557.315383]  ? __kmalloc+0x18c/0x200
[ 4557.317032]  tfw_http_msg_process+0x84/0xc0 [tempesta_fw]
[ 4557.319054]  ? virtqueue_add_outbuf+0x27d/0x30a [virtio_ring]
[ 4557.320701]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[ 4557.326824]  ? minmax_running_min+0x3b/0x90
[ 4557.329635]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[ 4557.331402]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[ 4557.333951]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[ 4557.335301]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[ 4557.336298]  tcp_data_queue+0x4f5/0xc50
[ 4557.337388]  tcp_rcv_established+0x27c/0x570
[ 4557.338845]  ? tcp_v4_inbound_md5_hash+0x62/0x1b0
[ 4557.340617]  tcp_v4_do_rcv+0x129/0x1d0
[ 4557.341928]  tcp_v4_rcv+0x947/0xa50
[ 4557.343604]  ip_local_deliver_finish+0x9a/0x1c0
[ 4557.345648]  ip_local_deliver+0x6b/0xe0
[ 4557.347407]  ? tcp_v4_early_demux+0x112/0x150
[ 4557.349056]  ? ip_rcv_finish+0x17a/0x400
[ 4557.350862]  ip_rcv+0x289/0x3c0
[ 4557.351874]  ? inet_del_offload+0x40/0x40
[ 4557.352605]  __netif_receive_skb_core+0x84f/0xb30
[ 4557.354041]  ? process_backlog+0xa3/0x160
[ 4557.358165]  process_backlog+0xa3/0x160
[ 4557.360881]  net_rx_action+0x28e/0x3f0
[ 4557.361923]  __do_softirq+0x10f/0x2a8
[ 4557.362642]  irq_exit+0xae/0xb0
[ 4557.363260]  call_function_single_interrupt+0x7d/0x90
[ 4557.365250]  </IRQ>
[ 4557.365937] RIP: 0010:native_safe_halt+0x2/0x10
[ 4557.366983] RSP: 0018:ffffffff83e03e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[ 4557.369610] RAX: ffffffff8349a7a0 RBX: ffffffff83e12480 RCX: 0000000000000000
[ 4557.372141] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 4557.375379] RBP: 0000000000000000 R08: 0000048deed6578e R09: ffff91713e9c4300
[ 4557.378307] R10: 0000000000000000 R11: 000004f26258ebfb R12: ffffffff83e12480
[ 4557.381539] R13: ffffffff83e12480 R14: 0000000000000000 R15: 0000000000000000
[ 4557.384942]  ? __sched_text_end+0x3/0x3
[ 4557.385566]  default_idle+0x1a/0xf0
[ 4557.386384]  do_idle+0x16e/0x1f0
[ 4557.387427]  cpu_startup_entry+0x6f/0x80
[ 4557.389219]  start_kernel+0x467/0x487
[ 4557.392020]  secondary_startup_64+0xa5/0xb0
[ 4557.393435] Code: fb 7f 48 8d 42 05 0f 86 7b ff ff ff 89 de 48 c1 eb 07 48 83 c0 01 83 ce 80 83 c1 01 48 83 fb 7f 40 88 70 ff 77 e6 e9 5c ff ff ff <0f> 0b 48 8b 44 24 58 e9 08 f7 ff ff 41 89 dc e9 9a f0 ff ff 0f

@vankoven
Copy link
Contributor Author

vankoven commented Apr 10, 2020 •
edited by krizhanovsky
Loading

  • One more warning:
[78406.287899] ------------[ cut here ]------------
[78406.289256] WARNING: CPU: 1 PID: 584 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:2391 tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[78406.291874] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm iTCO_wdt iTCO_vendor_support kvm_intel snd_hda_codec_generic binfmt_misc kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_intel snd_hda_codec snd_hda_core ghash_clmulni_intel snd_hwdep snd_pcm snd_timer evdev serio_raw pcspkr snd soundcore virtio_balloon sg virtio_gpu virtio_console ttm lpc_ich mfd_core drm_kms_helper drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse ahci libahci sym53c8xx scsi_transport_spi uhci_hcd ehci_pci ehci_hcd libata i2c_i801 scsi_mod usbcore virtio_pci usb_common virtio_ring virtio
[78406.317135]  [last unloaded: tempesta_lib]
[78406.319725] CPU: 1 PID: 584 Comm: nginx Tainted: G    B   W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[78406.327777] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[78406.330875] task: ffff9c40fcd9e900 task.stack: ffffbab680db8000
[78406.332466] RIP: 0010:tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[78406.336057] RSP: 0018:ffff9c41655037b0 EFLAGS: 00010293
[78406.337360] RAX: 000000000000001f RBX: 0000000000000000 RCX: 0000000000000029
[78406.339593] RDX: ffff9c41655240bf RSI: 0000000000000029 RDI: 000000000000002e
[78406.341660] RBP: 0000000000000000 R08: 0000000000000003 R09: ffff9c41655038f0
[78406.346238] R10: 000000000000001f R11: ffff9c413e1f61ee R12: 0000000000002e30
[78406.348124] R13: 0000000000002e00 R14: 0000000000000002 R15: ffff9c4165503890
[78406.351018] FS:  00007fde01a1c740(0000) GS:ffff9c4165500000(0000) knlGS:0000000000000000
[78406.356923] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[78406.363536] CR2: 000055aa84341ec8 CR3: 000000007cee8002 CR4: 00000000003606e0
[78406.369647] Call Trace:
[78406.371385]  <IRQ>
[78406.372800]  tfw_h2_add_hdr_via+0xef/0x140 [tempesta_fw]
[78406.374991]  tfw_http_resp_cache_cb+0x4ac/0x5f0 [tempesta_fw]
[78406.376532]  ? __tfw_http_msg_add_str_data+0x5e/0x100 [tempesta_fw]
[78406.378624]  ? tfw_http_conn_drop+0x280/0x280 [tempesta_fw]
[78406.380504]  tfw_cache_process+0x72/0x2a0 [tempesta_fw]
[78406.385721]  ? tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[78406.393488]  ? kvm_clock_get_cycles+0x1a/0x20
[78406.397727]  tfw_http_resp_cache+0xe1/0x150 [tempesta_fw]
[78406.401197]  tfw_http_msg_process_generic+0x893/0xd10 [tempesta_fw]
[78406.406241]  ? ip_finish_output2+0x2be/0x3b0
[78406.408679]  ? pg_skb_alloc+0x253/0x470
[78406.410788]  tfw_http_msg_process+0x84/0xc0 [tempesta_fw]
[78406.413430]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[78406.415835]  ? consume_skb+0x27/0x80
[78406.418778]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[78406.424415]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[78406.428091]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[78406.440355]  ? mod_timer+0xb0/0x3b0
[78406.441168]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[78406.442478]  tcp_rcv_established+0x4d2/0x570
[78406.443676]  tcp_v4_do_rcv+0x129/0x1d0
[78406.444619]  tcp_v4_rcv+0x947/0xa50
[78406.446035]  ip_local_deliver_finish+0x9a/0x1c0
[78406.447149]  ip_local_deliver+0x6b/0xe0
[78406.448230]  ? nf_hook_slow+0x43/0xc0
[78406.449447]  ip_rcv+0x289/0x3c0
[78406.450441]  ? inet_del_offload+0x40/0x40
[78406.451605]  __netif_receive_skb_core+0x84f/0xb30
[78406.453209]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[78406.456620]  ? process_backlog+0xa3/0x160
[78406.464121]  process_backlog+0xa3/0x160
[78406.467997]  net_rx_action+0x28e/0x3f0
[78406.471433]  __do_softirq+0x10f/0x2a8
[78406.473576]  do_softirq_own_stack+0x2a/0x40
[78406.476495]  </IRQ>
[78406.477463]  do_softirq.part.17+0x3d/0x50
[78406.479665]  __local_bh_enable_ip+0x55/0x60
[78406.480820]  ip_finish_output2+0x1b3/0x3b0
[78406.483415]  ? ip_output+0x71/0xe0
[78406.484672]  ip_output+0x71/0xe0
[78406.486736]  ? ip_queue_xmit+0x5c/0x3b0
[78406.489085]  tcp_transmit_skb+0x535/0x9c0
[78406.490854]  tcp_write_xmit+0x223/0x11c0
[78406.492331]  ? _copy_from_iter_full+0x93/0x230
[78406.493030]  __tcp_push_pending_frames+0x2d/0xd0
[78406.494068]  tcp_sendmsg_locked+0x68e/0xe60
[78406.495932]  tcp_sendmsg+0x27/0x40
[78406.497243]  sock_sendmsg+0x36/0x40
[78406.498259]  sock_write_iter+0x87/0x100
[78406.499226]  do_iter_readv_writev+0x11b/0x190
[78406.501233]  do_iter_write+0x80/0x190
[78406.502836]  vfs_writev+0x98/0x110
[78406.504009]  ? do_writev+0x5e/0xf0
[78406.505266]  do_writev+0x5e/0xf0
[78406.507133]  ? SyS_pread64+0x90/0xb0
[78406.509205]  do_syscall_64+0x8d/0x120
[78406.510761]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[78406.512833] RIP: 0033:0x7fde00354fe4
[78406.513707] RSP: 002b:00007ffcac2496f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
[78406.517015] RAX: ffffffffffffffda RBX: 00007fde019f02c8 RCX: 00007fde00354fe4
[78406.520071] RDX: 0000000000000001 RSI: 00007ffcac2497b0 RDI: 000000000000000b
[78406.526661] RBP: 000055bf73ed0cb8 R08: 000055bf73f4e460 R09: 00007ffcac2497b0
[78406.530225] R10: 000055bf73eb2d50 R11: 0000000000000246 R12: 00007ffcac249790
[78406.536453] R13: 000055bf7221548c R14: 000000007fffefff R15: 0000000000005db7
[78406.538593] Code: fb 7f 48 8d 42 05 0f 86 7b ff ff ff 89 de 48 c1 eb 07 48 83 c0 01 83 ce 80 83 c1 01 48 83 fb 7f 40 88 70 ff 77 e6 e9 5c ff ff ff <0f> 0b 48 8b 44 24 58 e9 08 f7 ff ff 41 89 dc e9 9a f0 ff ff 0f 
[78406.546574] ---[ end trace d2f49a8c8d368827 ]---

@vankoven vankoven mentioned this issue Apr 16, 2020
Closed
4 tasks
@krizhanovsky krizhanovsky assigned avbelov23 and unassigned vankoven May 5, 2020
@vankoven vankoven mentioned this issue May 13, 2020
Closed
@krizhanovsky
Copy link
Contributor

krizhanovsky commented Nov 17, 2020 •
edited by s0nx
Loading

  • Just tried to load our web site trough Tempesta FW current master as of date of the message using Firefox browser and got this warning:
[129225.889634] ------------[ cut here ]------------
[129225.892203] WARNING: CPU: 0 PID: 3257 at /root/tempesta/tempesta_fw/hpack.c:2401 tfw_hpack_node_compare+0xc08/0xc30 [tempesta_fw]
[129225.896297] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) tcp_diag inet_diag bochs_drm ttm crct10dif_pclmul drm_kms_helper ppdev drm sg button parport_pc parport fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw binfmt_misc ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto sr_mod sd_mod cdrom ata_generic virtio_net ata_piix virtio_pci virtio_ring virtio psmouse libata i2c_piix4 scsi_mod [last unloaded: tempesta_lib]
[129225.899082] CPU: 0 PID: 3257 Comm: nginx Tainted: G           O    4.14.32-kdump+ #158
[129225.899082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-2.fc32 04/01/2014
[129225.899082] task: ffff9a3cdc268140 task.stack: ffff9a3cd6290000
[129225.899082] RIP: 0010:tfw_hpack_node_compare+0xc08/0xc30 [tempesta_fw]
[129225.899082] RSP: 0018:ffff9a3cffc03718 EFLAGS: 00010293
[129225.899082] RAX: ffff9a3cb377816e RBX: 000000000000001f RCX: 0000000000000029
[129225.899082] RDX: ffff9a3cffc227ff RSI: 0000000000000029 RDI: ffff9a3cffc03848
[129225.899082] RBP: 000000000000001f R08: 0000000000000029 R09: 0000000000002e30
[129225.899082] R10: 0000000000000003 R11: ffff9a3cffc038a8 R12: 0000000000000000
[129225.899082] R13: 0000000000002e30 R14: 0000000000002030 R15: 0000000000000000
[129225.899082] FS:  00007efd12e5ab80(0000) GS:ffff9a3cffc00000(0000) knlGS:0000000000000000
[129225.899082] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[129225.899082] CR2: 000056294c2cb1c8 CR3: 000000005c32e001 CR4: 00000000003606f0
[129225.899082] Call Trace:
[129225.899082]  <IRQ>
[129225.899082]  tfw_hpack_rbtree_find.isra.1+0x4a/0x100 [tempesta_fw]
[129225.899082]  tfw_hpack_encode+0x13d/0x570 [tempesta_fw]
[129225.899082]  tfw_h2_add_hdr_via+0xea/0x130 [tempesta_fw]
[129225.899082]  tfw_http_resp_cache_cb+0x4a9/0x5f0 [tempesta_fw]
[129225.899082]  ? tfw_http_conn_drop+0x250/0x250 [tempesta_fw]
[129225.899082]  tfw_cache_process+0x6d/0x290 [tempesta_fw]
[129225.899082]  ? tfw_gfsm_move+0x146/0x180 [tempesta_fw]
[129225.899082]  ? efi_native_runtime_setup+0xa0/0xa0
[129225.899082]  tfw_http_resp_cache+0x141/0x180 [tempesta_fw]
[129225.899082]  tfw_http_msg_process_generic+0x839/0xce0 [tempesta_fw]
[129225.899082]  ? dev_hard_start_xmit+0x9e/0x1e0
[129225.899082]  ? pg_skb_alloc+0x223/0x420
[129225.899082]  tfw_http_msg_process+0x7f/0xc0 [tempesta_fw]
[129225.899082]  __gfsm_fsm_exec+0x51/0x90 [tempesta_fw]
[129225.899082]  ? consume_skb+0x22/0x80
[129225.899082]  tfw_connection_recv+0x3c/0x60 [tempesta_fw]
[129225.899082]  ? tfw_connection_send+0x20/0x20 [tempesta_fw]
[129225.899082]  ss_tcp_process_data+0x1e5/0x480 [tempesta_fw]
[129225.899082]  ss_tcp_data_ready+0x3e/0x90 [tempesta_fw]
[129225.899082]  tcp_data_queue+0x4f0/0xc50
[129225.899082]  tcp_rcv_established+0x277/0x570
[129225.899082]  tcp_v4_do_rcv+0x124/0x1c0
[129225.899082]  tcp_v4_rcv+0x93f/0xa40
[129225.899082]  ip_local_deliver_finish+0x95/0x1c0

Firefox didn't show the header image
Screen Shot 2020-11-17 at 23 11 15-fullpage

Tempesta config:

listen 192.168.100.4:443 proto=h2;
listen 192.168.100.4:80;

frang_limits {
	client_header_timeout 20;
	client_body_timeout 10;
	http_header_chunk_cnt 10;
	http_body_chunk_cnt 0;
	http_methods GET;
	http_uri_len 512;
	http_resp_code_block 400 403 404 3 10;
}

# Allow only following characters in URI (no '%'): /a-zA-Z0-9&?:-._=
http_uri_brange 0x2f 0x41-0x5a 0x61-0x7a 0x30-0x39 0x26 0x3f 0x3a 0x2d 0x2e 0x5f 0x3d;

block_action attack reply;
block_action error reply;

srv_group default {
	server 127.0.0.1:8080 conns_n=4;
}

vhost debian {
	tls_certificate /root/tempesta/etc/tfw-root.crt;
	tls_certificate_key /root/tempesta/etc/tfw-root.key;

	resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains";

	proxy_pass default;
}

cache 0;
#cache_fulfill * *;

http_chain {
	-> debian;
}

@krizhanovsky krizhanovsky assigned i-rinat and unassigned avbelov23 Nov 18, 2020
@krizhanovsky krizhanovsky mentioned this issue May 26, 2022
Closed
@krizhanovsky
Copy link
Contributor

Warnings on tfw_http_msg_process_generic() and tfw_http_msg_hdr_close() warnings raised in tempesta-tech/tempesta-test#254

@krizhanovsky krizhanovsky mentioned this issue Jul 26, 2022
Merged
@nickzaev nickzaev mentioned this issue Aug 25, 2022
Merged
@krizhanovsky
Copy link
Contributor

Warnings for tfw_http_msg_process_generic are covered by the disabled test https://github.com/tempesta-tech/tempesta-test/pull/296/files

@krizhanovsky krizhanovsky assigned s0nx and unassigned krizhanovsky Sep 16, 2022
This was referenced Sep 28, 2022
Merged
Closed
@s0nx s0nx linked a pull request Sep 28, 2022 that will close this issue
http2: Fix various warnings #1705
Merged
@s0nx
Copy link
Contributor

s0nx commented Sep 28, 2022 •
edited
Loading

Found some more:

  • ./h2spec http2/8.1.2.1/3 -h f35tfw.local -p 443 -t -k
[tempesta fw] Warning: failed to parse request: 192.168.122.1
------------[ cut here ]------------
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/fw/http.c:5053 tfw_http_cli_error_resp_and_log+0x285/0xb80 [tempesta_fw]
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJEC
T nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_f
ilter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus
pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virt
io_blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:tfw_http_cli_error_resp_and_log+0x285/0xb80 [tempesta_fw]
Code: 00 48 8d 54 24 40 31 f6 48 89 54 24 10 48 83 c7 14 e8 bf 7d fb ff 48 8b 54 24 10 4c 89 ee 48 c7 c7 00 e3 d9 c0 e8 f4 57 8e c2 <0f> 0b e9 3e fe ff ff 80 3
c 02 00 0f 85 d8 06 00 00 49 8b 87 a0 00
RSP: 0018:ffffc900001f7e08 EFLAGS: 00010286
RAX: 000000000000003d RBX: 1ffff9200003efc5 RCX: 0000000000000000
RDX: 1ffffffff08507a0 RSI: 00000000000000f6 RDI: fffff5200003efb8
RBP: 0000000000000000 R08: 000000000000003d R09: ffffffff84cdcf87
R10: fffffbfff099b9f0 R11: 0000000000000001 R12: 0000000000000001
R13: ffffffffc0da07e0 R14: 0000000000000001 R15: ffff88801dd2a020
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc68afc9018 CR3: 000000012c71e003 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <IRQ>
 ? tfw_h1_send_err_resp+0xe0/0xe0 [tempesta_fw]
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_http_req_process+0x938/0x15a0 [tempesta_fw]
 ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
 ? dynamic_emit_prefix+0x350/0x350
 ? __dynamic_pr_debug+0xeb/0x130
 ? dynamic_emit_prefix+0x2c0/0x350
 ? vprintk_emit+0x1c6/0x310
 tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
 ? ss_skb_chop_head_tail+0x1f9/0x790 [tempesta_fw]
 ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
 ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
 ? ttls_send_alert+0x1d0/0x1d0 [tempesta_tls]
 ? rcu_nmi_exit+0x50/0xc0
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_h2_frame_process+0x3c1/0x1280 [tempesta_fw]
 ? tfw_h2_stream_id_close+0x380/0x380 [tempesta_fw]
 tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
 tfw_tls_connection_recv+0x52b/0xaa0 [tempesta_fw]
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
 tcp_rcv_established+0x1155/0x1d10
 ? tcp_data_queue+0x4cb0/0x4cb0
 tcp_v4_do_rcv+0x4fa/0x750
 tcp_v4_rcv+0x2588/0x3530
 ? tcp_v4_early_demux+0x790/0x790
 ip_protocol_deliver_rcu+0x6a/0x550
 ? nf_confirm+0x340/0x340 [nf_conntrack]
 ip_local_deliver_finish+0x1a4/0x250
 ip_local_deliver+0x246/0x2a0
 ? ip_local_deliver_finish+0x250/0x250
 ? ip_route_input_slow+0x32a0/0x32a0
 ? ip_protocol_deliver_rcu+0x550/0x550
 ip_sublist_rcv_finish+0x11f/0x1b0
 ip_sublist_rcv+0x3e5/0x630
 ? ip_rcv_core+0xa70/0xa70
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
 ? get_page_from_freelist+0x1b39/0x4da0
 ip_list_rcv+0x293/0x3b0
 ? ip_rcv+0x180/0x180
 __netif_receive_skb_list_core+0x61d/0x8b0
 ? page_to_skb+0x69b/0x9f0 [virtio_net]
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
 ? __zone_watermark_ok+0x420/0x420
 ? virtqueue_add_split+0x4f0/0x1120
 netif_receive_skb_list_internal+0x5d7/0xc10
 ? __netif_receive_skb_list_core+0x8b0/0x8b0
 ? virtqueue_add_inbuf_ctx+0xc2/0xf0
 ? virtqueue_add_sgs+0x120/0x120
 ? try_fill_recv+0xaac/0x1c30 [virtio_net]
 napi_complete_done+0x188/0x5d0
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]
 virtnet_poll+0x90c/0xc81 [virtio_net]
 ? receive_buf+0x57f0/0x57f0 [virtio_net]
 ? kmem_cache_free_bulk+0x39d/0x7f0
 net_rx_action+0x37c/0xbc0
 ? napi_complete_done+0x5d0/0x5d0
 ? sched_clock_cpu+0x18/0x160
 __do_softirq+0x1a6/0x5ab
 ? handle_edge_irq+0x35e/0xb60
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 do_softirq_own_stack+0x37/0x40
 irq_exit_rcu+0x1ad/0x260
 common_interrupt+0x71/0x110
 asm_common_interrupt+0x1e/0x40
RIP: 0010:native_safe_halt+0xe/0x10
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 0
0 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000
 ? __sched_text_end+0x7/0x7
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0
 default_idle+0xa/0x10
 default_idle_call+0x74/0x1f0
 do_idle+0x3bd/0x550
 ? arch_cpu_idle_exit+0x40/0x40
 cpu_startup_entry+0x19/0x20
 start_secondary+0x22b/0x2d0
 ? set_cpu_sibling_map+0x21e0/0x21e0
 ? start_cpu0+0xc/0xc
 secondary_startup_64_no_verify+0xc2/0xcb
---[ end trace 321d13b12076d19e ]---
  • ./h2spec http2/8.1.2.1/3 -h f35tfw.local -p 443 -t -k
[tempesta fw]       enter tfw_h2_stream_fsm: stream->state=2, stream->id=1, type=1, flags=0x1
------------[ cut here ]------------
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/fw/http_stream.c:80 tfw_h2_stream_fsm+0x4b7/0x770 [tempesta_fw]
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJEC
T nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_f
ilter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus
pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virt
io_blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:tfw_h2_stream_fsm+0x4b7/0x770 [tempesta_fw]
Code: ea 03 0f b6 14 02 4c 89 d0 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 65 02 00 00 c7 43 2c 02 00 00 00 45 31 ed e9 ac fd ff ff <0f> 0b e9 9f fd ff ff 83 e
5 05 45 31 ed 40 80 fd 04 0f 84 95 fd ff
RSP: 0018:ffffc900001f7928 EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff88807228c9c0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 00000000000000f6 RDI: fffff5200003eef2
RBP: 0000000000000001 R08: 000000000000005d R09: ffffffff84cdcf87
R10: ffff88807228c9ec R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffff88807228c9f0 R15: ffffc900001f79b0
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc68afc9018 CR3: 000000012c71e003 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <IRQ>
 ? _raw_write_lock_irqsave+0xc0/0xc0
 tfw_h2_stream_id_close+0x161/0x380 [tempesta_fw]
 ? tfw_h2_stream_id+0xc0/0xc0 [tempesta_fw]
 ? __zone_watermark_ok+0x420/0x420
 tfw_h2_prep_resp+0xfd/0x7a0 [tempesta_fw]
 ? __alloc_pages_nodemask+0x2bd/0x5e0
 ? tfw_h2_resp_status_write+0x2c0/0x2c0 [tempesta_fw]
 ? alloc_pages_current+0x132/0x210
 ? memset+0x20/0x40
 ? __tfw_http_msg_alloc+0x362/0x6e0 [tempesta_fw]
 tfw_h2_send_resp+0x7b/0x130 [tempesta_fw]
 tfw_h2_send_err_resp+0xcf/0xf0 [tempesta_fw]
 ? tfw_h2_send_resp+0x130/0x130 [tempesta_fw]
 ? tfw_http_cli_error_resp_and_log+0x287/0xb80 [tempesta_fw]
 tfw_http_cli_error_resp_and_log+0x87f/0xb80 [tempesta_fw]
 ? tfw_h1_send_err_resp+0xe0/0xe0 [tempesta_fw]
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_http_req_process+0x938/0x15a0 [tempesta_fw]
 ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
 ? dynamic_emit_prefix+0x350/0x350
 ? __dynamic_pr_debug+0xeb/0x130
 ? dynamic_emit_prefix+0x2c0/0x350
 ? vprintk_emit+0x1c6/0x310
 tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
 ? ss_skb_chop_head_tail+0x1f9/0x790 [tempesta_fw]
 ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
 ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
 ? ttls_send_alert+0x1d0/0x1d0 [tempesta_tls]
 ? rcu_nmi_exit+0x50/0xc0
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_h2_frame_process+0x3c1/0x1280 [tempesta_fw]
 ? tfw_h2_stream_id_close+0x380/0x380 [tempesta_fw]
 tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
 tfw_tls_connection_recv+0x52b/0xaa0 [tempesta_fw]
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
 tcp_rcv_established+0x1155/0x1d10
 ? tcp_data_queue+0x4cb0/0x4cb0
 tcp_v4_do_rcv+0x4fa/0x750
 tcp_v4_rcv+0x2588/0x3530
 ? tcp_v4_early_demux+0x790/0x790
 ip_protocol_deliver_rcu+0x6a/0x550
 ? nf_confirm+0x340/0x340 [nf_conntrack]
 ip_local_deliver_finish+0x1a4/0x250
 ip_local_deliver+0x246/0x2a0
 ? ip_local_deliver_finish+0x250/0x250
 ? ip_route_input_slow+0x32a0/0x32a0
 ? ip_protocol_deliver_rcu+0x550/0x550
 ip_sublist_rcv_finish+0x11f/0x1b0
 ip_sublist_rcv+0x3e5/0x630
 ? ip_rcv_core+0xa70/0xa70
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
 ? get_page_from_freelist+0x1b39/0x4da0
 ip_list_rcv+0x293/0x3b0
 ? ip_rcv+0x180/0x180
 __netif_receive_skb_list_core+0x61d/0x8b0
 ? page_to_skb+0x69b/0x9f0 [virtio_net]
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
 ? __zone_watermark_ok+0x420/0x420
 ? virtqueue_add_split+0x4f0/0x1120
 netif_receive_skb_list_internal+0x5d7/0xc10
 ? __netif_receive_skb_list_core+0x8b0/0x8b0
 ? virtqueue_add_inbuf_ctx+0xc2/0xf0
 ? virtqueue_add_sgs+0x120/0x120
 ? try_fill_recv+0xaac/0x1c30 [virtio_net]
 napi_complete_done+0x188/0x5d0
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]
 virtnet_poll+0x90c/0xc81 [virtio_net]
 ? receive_buf+0x57f0/0x57f0 [virtio_net]
 ? kmem_cache_free_bulk+0x39d/0x7f0
 net_rx_action+0x37c/0xbc0
 ? napi_complete_done+0x5d0/0x5d0
 ? sched_clock_cpu+0x18/0x160
 __do_softirq+0x1a6/0x5ab
 ? handle_edge_irq+0x35e/0xb60
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 do_softirq_own_stack+0x37/0x40
 irq_exit_rcu+0x1ad/0x260
 common_interrupt+0x71/0x110
 asm_common_interrupt+0x1e/0x40
RIP: 0010:native_safe_halt+0xe/0x10
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 0
0 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000
 ? __sched_text_end+0x7/0x7
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0
 default_idle+0xa/0x10
 default_idle_call+0x74/0x1f0
 do_idle+0x3bd/0x550
 ? arch_cpu_idle_exit+0x40/0x40
 cpu_startup_entry+0x19/0x20
 start_secondary+0x22b/0x2d0
 ? set_cpu_sibling_map+0x21e0/0x21e0
 ? start_cpu0+0xc/0xc
 secondary_startup_64_no_verify+0xc2/0xcb
---[ end trace 321d13b12076d19f ]---
[tempesta fw]       exit tfw_h2_stream_fsm: stream->state=2, res=3
[tempesta fw]   tfw_h2_send_resp: HTTP/2 response message transformation error: conn=[0000000067ae9791]

@krizhanovsky krizhanovsky mentioned this issue Oct 5, 2022
Closed
@s0nx
Copy link
Contributor

s0nx commented Oct 10, 2022

I can't reproduce this one, but with the same config i've caught another warning.

[tempesta tls] Warning: [::ffff:192.168.122.1] ClientHello: bad extension size 281 (remaining extensions size 285)                                             
------------[ cut here ]------------                                                                                                                                                                                                                                                                                           
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/tls/tls_srv.c:916 ttls_parse_client_hello.cold+0xeec/0x1012 [tempesta_tls]                                                                                                                                                                                                       
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virtio_
blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]                                                
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1                                                                                                                                                                                                                                                          
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015                                                                                      
RIP: 0010:ttls_parse_client_hello.cold+0xeec/0x1012 [tempesta_tls]                                                                                                                                                                                                                                                             
Code: 0f 0b e9 b7 f9 ff ff 89 14 24 e8 6a ea e8 c0 8b 14 24 e9 70 f8 ff ff 48 8d 75 38 44 89 e1 48 c7 c7 80 07 9a c0 e8 68 1a b0 c2 <0f> 0b e9 77 f1 ff ff 49 8d 74 24 38 48 c7 c7 20 26 9a c0 e8 50 1a
RSP: 0018:ffffc900001f7c40 EFLAGS: 00010286                                                                                                                    
RAX: 0000000000000072 RBX: 0000000000000119 RCX: 0000000000000000                                                                                              
RDX: 0000000000000027 RSI: 0000000000000004 RDI: fffff5200003ef7f                                                                                              
RBP: ffff88817eb60a00 R08: 0000000000000072 R09: ffff888149720a8b                                                                                              
R10: ffffed10292e4151 R11: 0000000000000001 R12: 000000000000011d                                                                                              
R13: ffff888037ad5bbf R14: 000000000000014c R15: ffff888128b014f0                                                                                              
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000                                                                                   
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                                                                                              
CR2: 00007fc687e84660 CR3: 000000012c71e006 CR4: 0000000000770ee0                                                                                                                                                                                                                                                              
PKRU: 55555554                                                                                                                                                                                                                                                                                                                 
Call Trace:                                                                                                                                                                                                                                                                                                                    
 <IRQ>                                                                                                                                                                                                                                                                                                                         
 ? rgb_background+0xe2/0x140                                                                                                                                                                                                                                                                                                   
 ? desc_read_finalized_seq+0x100/0x100                                                                                                                         
 ? atomic_notifier_call_chain+0xa8/0xe0                                                                                                                        
 ttls_handshake_server_step+0xc19/0x41d0 [tempesta_tls]                                                                                                        
 ? prb_read_valid+0x56/0x70                                                                                                                                                                                                                                                                                                    
 ? ttls_write_server_key_exchange+0xf70/0xf70 [tempesta_tls]                                                                                                   
 ? up+0x17/0xc0                                                                                                                                                
 ? vprintk_store+0x1d3/0x420                                                                                                                                                                                                                                                                                                   
 ? _raw_spin_lock+0x6a/0xb0                                                                                                                                    
 ? irq_work_claim+0x16/0x60                                                                                                                                    
 ? irq_work_queue+0xb/0x30                                                                                                                                     
 ? vprintk_emit+0x1c6/0x310                                                                                                                                    
 ? printk+0x96/0xb2                                                                                                                                            
 ? record_print_text.cold+0x11/0x11                                                                                                                            
 ? hex_dump_to_buffer+0xa90/0xa90                                                                                                                                                                                                                                                                                              
 ? __dynamic_pr_debug+0xeb/0x130                                                                                                                               
 ? __dynamic_pr_debug+0xeb/0x130                                                                                                                               
 ? dynamic_emit_prefix+0x350/0x350                                                                                                                             
 ? nowork+0xc/0xc [sha512_ssse3]                                                                                                                               
 ? memcpy+0x39/0x60                                                                                                                                            
 ? sha512_update+0xcc/0x1c0 [sha512_ssse3]                                                                                                                                                                                                                                                                                     
 ? ttls_update_checksum+0x1b6/0x430 [tempesta_tls]                                                                                                             
 ttls_recv+0xb08/0x1e40 [tempesta_tls]                                                                                                                         
 ss_skb_process+0x393/0x570 [tempesta_fw]                                                                                                                      
 ? ttls_handle_alert+0x1b0/0x1b0 [tempesta_tls]                                                                                                                                                                                                                                                                                
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]                                                                                                                                                                                                                                                                                   
 ? _raw_spin_lock+0x6a/0xb0                                                                                                                                    
 ? _raw_write_lock_irqsave+0xc0/0xc0                                                                                                                           
 ? _raw_write_unlock_bh+0x60/0x60                                                                                                                                                                                                                                                                                              
 tfw_tls_connection_recv+0x175/0xaa0 [tempesta_fw]                                                                                                                                                                                                                                                                             
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]                                                                                                             
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]                                                                                                                 
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]                                                                                                                                                                                                                                                                             
 ? tcp_grow_window+0xcb/0x700                                                                                                                                  
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]                                                                                                                   
 tcp_data_queue+0x1669/0x4cb0                                                                                                                                                                                                                                                                                                  
 ? tcp_data_ready+0x450/0x450                                                                                                                                                                                                                                                                                                  
 ? tcp_try_keep_open+0x2b0/0x2b0                                                                                                                               
 ? kvm_clock_get_cycles+0xd/0x10                                                                                                                               
 ? ktime_get+0x55/0xd0                                                                                                                                                                                                                                                                                                         
 ? tempesta_sock_tcp_rcv+0x8d/0xc0                                                                                                                                                                                                                                                                                             
 tcp_rcv_established+0x6ae/0x1d10                                                                                                                              
 ? tcp_data_queue+0x4cb0/0x4cb0                                                                                                                                
 tcp_v4_do_rcv+0x4fa/0x750                                                                                                                                     
 tcp_v4_rcv+0x2588/0x3530                                                                                                                                      
 ? nf_ct_deliver_cached_events+0xe4/0x410 [nf_conntrack]                                                                                                       
 ? tcp_v4_early_demux+0x790/0x790                                                                                                                                                                                                                                                                                              
 ip_protocol_deliver_rcu+0x6a/0x550                                                                                                                            
 ? nf_confirm+0x340/0x340 [nf_conntrack]                                                                                                                       
 ip_local_deliver_finish+0x1a4/0x250                                                                                                                           
 ip_local_deliver+0x246/0x2a0                                                                                                                                  
 ? ip_local_deliver_finish+0x250/0x250                                                                                                                                                                                                                                                                                         
 ? ip_protocol_deliver_rcu+0x550/0x550                                                                                                                         
 ip_sublist_rcv_finish+0x11f/0x1b0                                                                                                                                                                                                                                                                                             
 ip_sublist_rcv+0x3e5/0x630                                                                                                                                    
 ? ip_rcv_core+0xa70/0xa70                                                                                                                                                                                                                                                                                                     
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0                                                                                                                
 ip_list_rcv+0x293/0x3b0                                                                                                                                       
 ? ip_rcv+0x180/0x180                                                                                                                                          
 ? memset+0x20/0x40                                                                                                                                                                                                                                                                                                            
 __netif_receive_skb_list_core+0x61d/0x8b0                                                                                                                                                                                                                                                                                     
 ? page_to_skb+0x6c6/0x9f0 [virtio_net]                                                                                                                                                                                                                                                                                        
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
 ? gro_normal_one+0x56/0x220                                                                                                                                   
 ? napi_gro_receive+0x23c/0x2a0                                                                                                                                
 ? receive_buf+0xb68/0x57f0 [virtio_net]                                                                                                                       
 netif_receive_skb_list_internal+0x5d7/0xc10                                                                                                                   
 ? __netif_receive_skb_list_core+0x8b0/0x8b0                                                                                                                   
 ? page_to_skb+0x9f0/0x9f0 [virtio_net]                                                                                                                        
 napi_complete_done+0x188/0x5d0                                                                                                                                                                                                                                                                                                
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]                                                                                                                                                                                                                                                                                
 virtnet_poll+0x90c/0xc81 [virtio_net]                                                                                                                                                                                                                                                                                         
 ? receive_buf+0x57f0/0x57f0 [virtio_net]                                                                                                                      
 ? kvm_sched_clock_read+0xd/0x20                                                                                                                                                                                                                                                                                               
 ? sched_clock+0x5/0x10                                                                                                                                        
 ? sched_clock_cpu+0x18/0x160                                                                                                                                                                                                                                                                                                  
 net_rx_action+0x37c/0xbc0                                                                                                                                     
 ? rcu_accelerate_cbs+0x61/0x510                                                                                                                               
 ? napi_complete_done+0x5d0/0x5d0                                                                                                                              
 __do_softirq+0x1a6/0x5ab                                                                                                                                      
 asm_call_irq_on_stack+0xf/0x20                                                                                                                                
 </IRQ>                                                                                                                                                        
 do_softirq_own_stack+0x37/0x40                                                                                                                                
 irq_exit_rcu+0x1ad/0x260                                                                                                                                      
 sysvec_apic_timer_interrupt+0x33/0x80                                                                                                                         
 asm_sysvec_apic_timer_interrupt+0x12/0x20                                                                                                                                                                                                                                                                                     
RIP: 0010:native_safe_halt+0xe/0x10                                                                                                                                                                                                                                                                                            
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206                                                                                                                                                                                                                                                                                    
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d                                                                                                                                                                                                                                                              
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80                                                                                              
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83                                                                                              
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002                                                                                              
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000                                                                                                                                                                                                                                                              
 ? __sched_text_end+0x7/0x7                                                                                                                                    
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0                                                                                                                         
 default_idle+0xa/0x10                                                                                                                                                                                                                                                                                                         
 default_idle_call+0x74/0x1f0                                                                                                                                  
 do_idle+0x3bd/0x550                                                                                                                                           
 ? arch_cpu_idle_exit+0x40/0x40                                                                                                                                
 cpu_startup_entry+0x19/0x20                                                                                                                                   
 start_secondary+0x22b/0x2d0                                                                                                                                   
 ? set_cpu_sibling_map+0x21e0/0x21e0                                                                                                                           
 ? start_cpu0+0xc/0xc                                                                                                                                                                                                                                                                                                          
 secondary_startup_64_no_verify+0xc2/0xcb                                                                                                                      
---[ end trace 321d13b12076d1c7 ]---                                                                                                                           
[tempesta tls]   send alert level=2 message=50

@s0nx
Copy link
Contributor

s0nx commented Oct 10, 2022

Created the separate issue #1713 for it.

@s0nx
Copy link
Contributor

s0nx commented Oct 16, 2022 •
edited
Loading

  • Some strange crash on if-non-match parsing:
[  362.596229] [tempesta fw]   tfw_h2_parse_req_hdr: fin=1, len=15, data="6331593c-3cd1", req=[00000000529c3f7d]
[  362.596236] [tempesta fw]       enter FSM at state 0000000000000000
[  362.596245] [tempesta fw]       parser at Req_HdrIf_None_MatchV: c=0x22("), p_off=0
[  362.596256] [tempesta fw]       enter FSM at state 000000000884cb17
[  362.596263] [tempesta fw]       parser at I_GMT: c=0x20( ), p_off=-81543651
[  362.596278] BUG: kernel NULL pointer dereference, address: 000000000000000f
[  362.596288] #PF: supervisor instruction fetch in kernel mode
[  362.596293] #PF: error_code(0x0010) - not-present page
[  362.596297] PGD 0 P4D 0
[  362.596304] Oops: 0010 [#1] SMP KASAN NOPTI
[  362.596309] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           OE     5.10.35+ #1
[  362.596313] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[  362.596319] RIP: 0010:0xf
[  362.596325] Code: Unable to access opcode bytes at RIP 0xffffffffffffffe5.
[  362.596329] RSP: 0018:ffffc9000024fad0 EFLAGS: 00010282
[  362.596334] RAX: 00000000fffffffe RBX: 0000000000000022 RCX: 0000000000000000
[  362.596337] RDX: 0000000000000020 RSI: 0000000000000000 RDI: ffff88812a302798
[  362.596341] RBP: ffff88812a302778 R08: 0000000000000001 R09: ffff88812a302778
[  362.596344] R10: ffffed10254604f2 R11: 0000000000000001 R12: 0000000000000000
[  362.596348] R13: ffff88812fa3e203 R14: ffffffffc0bd8e5a R15: ffff88812ac7a020
[  362.596352] FS:  0000000000000000(0000) GS:ffff888149780000(0000) knlGS:0000000000000000
[  362.596357] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  362.596360] CR2: ffffffffffffffe5 CR3: 000000017f016004 CR4: 0000000000770ee0
[  362.596374] PKRU: 55555554
[  362.596377] Call Trace:
[  362.596380]  <IRQ>
[  362.596424]  ? tfw_hpack_decode+0x17f6/0x7850 [tempesta_fw]
[  362.596462]  ? tfw_h2_parse_req+0x448/0x8a0 [tempesta_fw]
[  362.596499]  ? ss_skb_process+0x393/0x570 [tempesta_fw]
[  362.596534]  ? tfw_http_parse_req+0x398a0/0x398a0 [tempesta_fw]
[  362.596542]  ? irq_work_queue+0x1a/0x30
[  362.596578]  ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
[  362.596612]  ? tfw_http_req_process+0x1da/0x15a0 [tempesta_fw]
[  362.596646]  ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
[  362.596652]  ? dynamic_emit_prefix+0x350/0x350
[  362.596657]  ? alloc_pages_current+0x132/0x210
[  362.596661]  ? __get_free_pages+0x11/0xa0
[  362.596693]  ? __tfw_pool_new+0x25/0x180 [tempesta_fw]
[  362.596725]  ? tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
[  362.596757]  ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
[  362.596790]  ? ss_skb_chop_head_tail+0x34e/0x790 [tempesta_fw]
[  362.596824]  ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
[  362.596857]  ? ss_skb_split+0x32f/0x480 [tempesta_fw]
[  362.596888]  ? tfw_h2_frame_process+0x3d1/0x12a0 [tempesta_fw]
[  362.596919]  ? tfw_h2_stream_id_close+0x3a0/0x3a0 [tempesta_fw]
[  362.596950]  ? tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
[  362.596983]  ? tfw_tls_connection_recv+0x4d9/0xa30 [tempesta_fw]
[  362.597015]  ? ss_skb_unroll+0x9bd/0xeb0 [tempesta_fw]
[  362.597048]  ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
[  362.597081]  ? ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
[  362.597087]  ? deref_stack_reg+0xcf/0x170
[  362.597119]  ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
[  362.597125]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597158]  ? ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
[  362.597164]  ? tcp_data_queue+0x1669/0x4cb0
[  362.597167]  ? tcp_reset+0x1d0/0x1d0
[  362.597171]  ? tcp_data_ready+0x450/0x450
[  362.597174]  ? tcp_try_keep_open+0x2b0/0x2b0
[  362.597178]  ? kvm_clock_get_cycles+0xd/0x10
[  362.597183]  ? ktime_get+0x55/0xd0
[  362.597187]  ? tempesta_sock_tcp_rcv+0x8d/0xc0
[  362.597191]  ? tcp_rcv_established+0x6ae/0x1d10
[  362.597196]  ? sk_filter_trim_cap+0x2cb/0x710
[  362.597200]  ? tcp_data_queue+0x4cb0/0x4cb0
[  362.597204]  ? tcp_v4_do_rcv+0x4fa/0x750
[  362.597207]  ? tcp_v4_rcv+0x2588/0x3530
[  362.597220]  ? nf_ct_deliver_cached_events+0xe4/0x410 [nf_conntrack]
[  362.597225]  ? __inet_lookup_established+0x27e/0x4d0
[  362.597229]  ? tcp_v4_early_demux+0x790/0x790
[  362.597233]  ? ip_protocol_deliver_rcu+0x6a/0x550
[  362.597242]  ? nf_confirm+0x340/0x340 [nf_conntrack]
[  362.597246]  ? ip_local_deliver_finish+0x1a4/0x250
[  362.597251]  ? ip_local_deliver+0x246/0x2a0
[  362.597254]  ? ip_local_deliver_finish+0x250/0x250
[  362.597258]  ? ip_rcv_finish_core.constprop.0+0x2e9/0x18a0
[  362.597262]  ? ip_protocol_deliver_rcu+0x550/0x550
[  362.597266]  ? ip_rcv+0x15a/0x180
[  362.597270]  ? ip_local_deliver+0x2a0/0x2a0
[  362.597273]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597277]  ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
[  362.597282]  ? get_stack_info+0x30/0x90
[  362.597287]  ? __netif_receive_skb_core.constprop.0+0xdd9/0x3050
[  362.597291]  ? deref_stack_reg+0xcf/0x170
[  362.597295]  ? generic_xdp_tx+0x410/0x410
[  362.597300]  ? update_sd_lb_stats.constprop.0+0x375/0x2cc0
[  362.597306]  ? inet_gro_receive+0x9fb/0x11d0
[  362.597310]  ? memset+0x20/0x40
[  362.597313]  ? __netif_receive_skb_list_core+0x2cf/0x8b0
[  362.597320]  ? page_to_skb+0x69b/0x9f0 [virtio_net]
[  362.597324]  ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
[  362.597328]  ? gro_normal_one+0x56/0x220
[  362.597332]  ? napi_gro_receive+0x23c/0x2a0
[  362.597336]  ? receive_buf+0xb68/0x57f0 [virtio_net]
[  362.597340]  ? netif_receive_skb_list_internal+0x5d7/0xc10
[  362.597344]  ? __netif_receive_skb_list_core+0x8b0/0x8b0
[  362.597349]  ? page_to_skb+0x9f0/0x9f0 [virtio_net]
[  362.597383]  ? tfw_apm_prcntl_tmfn+0xcc2/0x1640 [tempesta_fw]
[  362.597388]  ? napi_complete_done+0x188/0x5d0
[  362.597393]  ? virtqueue_napi_complete+0x2b/0x80 [virtio_net]
[  362.597398]  ? virtnet_poll+0x90c/0xc81 [virtio_net]
[  362.597403]  ? receive_buf+0x57f0/0x57f0 [virtio_net]
[  362.597408]  ? net_rx_action+0x37c/0xbc0
[  362.597412]  ? napi_complete_done+0x5d0/0x5d0
[  362.597416]  ? sched_clock_cpu+0x18/0x160
[  362.597421]  ? __do_softirq+0x1a6/0x5ab
[  362.597426]  ? handle_edge_irq+0x35e/0xb60
[  362.597431]  ? asm_call_irq_on_stack+0xf/0x20
[  362.597433]  </IRQ>
[  362.597437]  ? do_softirq_own_stack+0x37/0x40
[  362.597442]  ? irq_exit_rcu+0x1ad/0x260
[  362.597446]  ? common_interrupt+0x71/0x110
[  362.597450]  ? asm_common_interrupt+0x1e/0x40
[  362.597454]  ? __sched_text_end+0x7/0x7
[  362.597458]  ? rcu_eqs_enter.constprop.0+0x7d/0xa0
[  362.597462]  ? native_safe_halt+0xe/0x10
[  362.597466]  ? default_idle+0xa/0x10
[  362.597470]  ? default_idle_call+0x74/0x1f0
[  362.597474]  ? do_idle+0x3bd/0x550
[  362.597478]  ? arch_cpu_idle_exit+0x40/0x40
[  362.597481]  ? cpu_startup_entry+0x19/0x20
[  362.597485]  ? start_secondary+0x22b/0x2d0
[  362.597489]  ? set_cpu_sibling_map+0x21e0/0x21e0
[  362.597493]  ? start_cpu0+0xc/0xc
[  362.597496]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597500] Modules linked in: tempesta_fw(OE) tempesta_db(OE) sha256_ssse3 sha512_ssse3 tempesta_tls(OE) tempesta_lib(OE) xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 ip6tab
le_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel
 kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support irqbypass virtiofs input_leds lpc_ich pcspkr i2c_i801 led_class i2c_smbus virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_int
el virtio_net ghash_clmulni_intel serio_raw virtio_scsi net_failover failover virtio_console virtio_blk qemu_fw_cfg ipmi_devintf ipmi_msghandler
[  362.597590] CR2: 000000000000000f
[  362.597598] ---[ end trace d3e017451f69f5c6 ]---
[  362.597602] RIP: 0010:0xf
[  362.597607] Code: Unable to access opcode bytes at RIP 0xffffffffffffffe5.
[  362.597611] RSP: 0018:ffffc9000024fad0 EFLAGS: 00010282
[  362.597615] RAX: 00000000fffffffe RBX: 0000000000000022 RCX: 0000000000000000
[  362.597622] RDX: 0000000000000020 RSI: 0000000000000000 RDI: ffff88812a302798
[  362.597625] RBP: ffff88812a302778 R08: 0000000000000001 R09: ffff88812a302778
[  362.597628] R10: ffffed10254604f2 R11: 0000000000000001 R12: 0000000000000000
[  362.597632] R13: ffff88812fa3e203 R14: ffffffffc0bd8e5a R15: ffff88812ac7a020
[  362.597637] FS:  0000000000000000(0000) GS:ffff888149780000(0000) knlGS:0000000000000000
[  362.597640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  362.597643] CR2: ffffffffffffffe5 CR3: 000000017f016004 CR4: 0000000000770ee0
[  362.597657] PKRU: 55555554
[  362.597661] Kernel panic - not syncing: Fatal exception in interrupt
[  362.597771] Kernel Offset: disabled
[  362.597780] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
gef➤

@s0nx
Copy link
Contributor

s0nx commented Oct 17, 2022

Unfortunately, i wasn't able to reproduce the following issues:

@krizhanovsky krizhanovsky mentioned this issue Nov 7, 2022
Merged
@s0nx s0nx closed this as completed in #1705 Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s0nx s0nx

Labels
bug h2
Projects
None yet
Milestone
0.7 - HTTP/2
Development

Successfully merging a pull request may close this issue.

http2: Fix various warnings tempesta-tech/tempesta
5 participants
@krizhanovsky @i-rinat @vankoven @avbelov23 @s0nx