Segmentation fault coming from ngx_http_headers_more_headers_in

[mrgonza78]

core dump

(gdb) bt full
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {65536, 94556631326832, 140725098555056, 139957303924096, 4222451712, 94556631326832, 94556631326933, 94556631326832, 94556631326832, 94556631327069, 
            94556631327132, 94556631326832, 94556631327132, 0, 0, 0}}
        pid = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
#1  0x00007f4a595dd7f1 in __GI_abort () at abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x55ffae6be070, sa_sigaction = 0x55ffae6be070}, sa_mask = {__val = {0, 139957307477056, 0, 0, 140725098554728, 0, 0, 5, 140725098554560, 
              139957305180352, 139957305165504, 0, 6501681080503838208, 139957305150466, 0, 139957305165504}}, sa_flags = -1444510640, sa_restorer = 0x55ffa9e684a0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
        __cnt = <optimized out>
        __set = <optimized out>
        __cnt = <optimized out>
        __set = <optimized out>
#2  0x00007f4a595cd3fa in __assert_fail_base (fmt=0x7f4a597546c0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x55ffa9e684a0 "!(r->headers_in.headers.part.next == ((void *)0) && r->headers_in.headers.last != &r->headers_in.headers.part)", 
    file=file@entry=0x55ffa9e68450 "../headers-more-nginx-module-0.33/src/ngx_http_headers_more_headers_in.c", line=line@entry=277, 
    function=function@entry=0x55ffa9e68510 "ngx_http_set_header_helper") at assert.c:92
        str = 0x55ffae6be070 "\240\341k\256\377U"
        total = 4096
#3  0x00007f4a595cd472 in __GI___assert_fail (assertion=0x55ffa9e684a0 "!(r->headers_in.headers.part.next == ((void *)0) && r->headers_in.headers.last != &r->headers_in.headers.part)", 
    file=0x55ffa9e68450 "../headers-more-nginx-module-0.33/src/ngx_http_headers_more_headers_in.c", line=277, function=0x55ffa9e68510 "ngx_http_set_header_helper") at assert.c:101
No locals.
#4  0x000055ffa9e20168 in ?? ()
No symbol table info available.
#5  0x000055ffa9e20c17 in ngx_http_headers_more_exec_input_cmd ()
No symbol table info available.
#6  0x000055ffa9e1f262 in ?? ()
No symbol table info available.
#7  0x000055ffa9d5916c in ngx_http_core_rewrite_phase ()
No symbol table info available.
#8  0x000055ffa9d546b5 in ngx_http_core_run_phases ()
No symbol table info available.
#9  0x000055ffa9d5d5e0 in ngx_http_run_posted_requests ()
No symbol table info available.
#10 0x000055ffa9d5ffbc in ?? ()
No symbol table info available.
#11 0x000055ffa9d3d5e4 in ngx_event_accept ()
No symbol table info available.
#12 0x000055ffa9d46548 in ?? ()
No symbol table info available.
#13 0x000055ffa9d3cae6 in ngx_process_events_and_timers ()
No symbol table info available.
#14 0x000055ffa9d44968 in ?? ()
No symbol table info available.
#15 0x000055ffa9d42f02 in ngx_spawn_process ()
No symbol table info available.
#16 0x000055ffa9d44000 in ?? ()
No symbol table info available.
#17 0x000055ffa9d454b3 in ngx_master_process_cycle ()
No symbol table info available.
#18 0x000055ffa9d1baa7 in main ()
No symbol table info available.

error.log

../headers-more-nginx-module-0.33/src/ngx_http_headers_more_headers_in.c:277: ngx_http_set_header_helper: Assertion `!(r->headers_in.headers.part.next == ((void *)0) && r->headers_in.headers.last != &r->headers_in.headers.part)' failed.
2022/03/11 13:45:53 [info] 19631#19631: *216 proxy 172.30.188.163:40466 connected to 172.30.161.22:443
2022/03/11 13:45:53 [info] 19631#19631: *216 recv() failed (104: Connection reset by peer) while proxying and reading from client, client: unix:, server: unix:/var/run/nginx-gw/funnel_eventsink_dm.sock, upstream: "172.30.161.22:443", bytes from/to client:235/5379, bytes from/to upstream:5379/235
2022/03/11 13:45:53 [info] 19631#19631: *216 client disconnected, bytes from/to client:235/5379, bytes from/to upstream:5379/235
2022/03/11 13:45:53 [notice] 19630#19630: signal 17 (SIGCHLD) received from 19640
2022/03/11 13:45:53 [alert] 19630#19630: worker process 19640 exited on signal 6 (core dumped)
2022/03/11 13:45:53 [notice] 19630#19630: start worker process 19900
2022/03/11 13:45:53 [notice] 19630#19630: signal 29 (SIGIO) received

nginx/openresty version

nginx version: openresty/1.19.3.1
built with OpenSSL 1.1.1h  22 Sep 2020 (running with OpenSSL 1.1.1l  24 Aug 2021)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-stream --with-http_ssl_module

I can't paste the configuration as is, but basically the locations looks:

  location /event-sink/v1/time {
    mirror /event-sink-mirror;
    proxy_set_header Connection keep-alive;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass https://funnel_eventsink_dm;
  }

  location /event-sink-mirror {
    internal;
    rewrite /event-sink-mirror $request_uri; break;
    proxy_set_header Connection keep-alive;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass https://api_transforwarder;
  }

and the http config:

  more_set_headers 'Server-Version: e0cf91188bb713df7a480a127cd840dc0f95262d';
  more_set_headers 'X-Frame-Options: SAMEORIGIN';
  more_set_headers 'X-Content-Type-Options: nosniff';
  more_set_headers 'X-XSS-Protection: 1; mode=block';
  more_set_headers 'X-Request-ID: $request_id';
  more_set_headers 'x-upstream-status: $upstream_status';
  more_set_headers 'x-upstream-sent: $upstream_bytes_sent';
  more_set_headers 'x-upstream-received: $upstream_bytes_received';
  more_set_headers 'x-upstream-conn-time: $upstream_connect_time';
  more_set_headers 'x-upstream-resp-time: $upstream_response_time';

I think those are the relevant parts of the config.

The error is just reproducible with this simple curl

curl -ufoo:bar localhost/event-sink/v1/time

If we remove the mirror, the error doesn't happen
If we remove the "authorization" header, the error doesn't happen either

Any help, appreciate it

[zhuizhuhaomeng]

it is better to add an test in t/ that can reproduce this issue..