-
Notifications
You must be signed in to change notification settings - Fork 20
RQGSpec solaris backtrace improvements
philip-stoev edited this page Jul 17, 2012
·
1 revision
This page is the specification for the following Launchpad blueprint for the Random Query Generator: solaris-backtrace-improvements
See RandomQueryGeneratorSpecifications for other specifications.
Source code:
- Repository: lp:randgen (bzr)
- File: lib/GenTest/Reporter/Backtrace.pm
if (windows()) { ...
} else
push @commands, "gdb --batch --se=$binary --core=$core --command=backtrace.gdb";
push @commands, "gdb --batch --se=$binary --core=$core --command=backtrace-all.gdb";
}
if ($^O eq 'solaris') {
push @commands, "echo '::stack' | mdb $core | c++filt";
}
...
If code has been compiled with Sun Studio and a core file is generated due to a crash, gdb will try to analyze it but fail and produce another core dump which eventually will contain remnants of the stack trace we are insterested in. Example (taken from falcon_limit test):
Writing a core file # 18:54:25 datadir is /export/home/pb2/test/sb_1-894007-1256665811.97/mysql-6.0.12-alpha-solaris10-sparc-test/vardirs/master-data/ # 18:54:25 Core file appears to be /export/home/pb2/test/sb_1-894007-1256665811.97/mysql-6.0.12-alpha-solaris10-sparc-test/vardirs/master-data//core warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074 warning: Lowest section in /lib/libthread.so.1 is .dynamic at 00000074 warning: Lowest section in /lib/libdl.so.1 is .hash at 000000b4 # 18:54:41 Core was generated by `/export/home/pb2/test/sb_1-894007-1256665811.97/mysql-6.0.12-alpha-solaris10-sp'. # 18:54:41 Program terminated with signal 6, Aborted. # 18:54:41 #0 0xfee4c714 in _lwp_kill () from /lib/libc.so.1 # 18:54:41 #0 0xfee4c714 in _lwp_kill () from /lib/libc.so.1 # 18:54:41 #1 0x01155d2c in my_write_core (sig=6) # 18:54:41 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/mysys/stacktrace.c:309 # 18:54:41 ../../gdb-6.6/gdb/dwarf2read.c:2736: internal-error: process_die: Assertion `die->child == NULL' failed. # 18:54:41 A problem internal to GDB has been detected, # 18:54:41 further debugging may prove unreliable. # 18:54:41 Quit this debugging session? (y or n) [answered Y; input not from terminal] # 18:54:41 ../../gdb-6.6/gdb/dwarf2read.c:2736: internal-error: process_die: Assertion `die->child == NULL' failed. # 18:54:41 A problem internal to GDB has been detected, # 18:54:41 further debugging may prove unreliable. # 18:54:41 Create a core file of GDB? (y or n) [answered Y; input not from terminal] warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074 warning: Lowest section in /lib/libthread.so.1 is .dynamic at 00000074 warning: Lowest section in /lib/libdl.so.1 is .hash at 000000b4 Die: DW_TAG_<unknown> (abbrev = 61, offset = 7504957) has children: TRUE attributes: DW_AT_name (DW_FORM_string) string: "List" DW_AT_<unknown> (DW_FORM_string) string: "__1nEList3CTA__" DW_AT_decl_file (DW_FORM_data1) constant: 19 DW_AT_decl_line (DW_FORM_data2) constant: 383 DW_AT_sibling (DW_FORM_ref4) constant ref: 7504992 (adjusted) Dwarf Error: Cannot find type of die [in module /export/home/pb2/test/sb_1-894007-1256665811.97/mysql-6.0.12-alpha-solaris10-sparc-test/libexec/mysqld] # 18:54:59 Core was generated by `/export/home/pb2/test/sb_1-894007-1256665811.97/mysql-6.0.12-alpha-solaris10-sp'. # 18:54:59 Program terminated with signal 6, Aborted. # 18:54:59 #0 0xfee4c714 in _lwp_kill () from /lib/libc.so.1 # 18:54:59 # 18:54:59 Thread 35 (process 2179152 ): # 18:54:59 #0 0xfee48ac4 in __lwp_park () from /lib/libc.so.1 # 18:54:59 #1 0xfee42918 in cond_sleep_queue () from /lib/libc.so.1 # 18:54:59 #2 0xfee42adc in cond_wait_queue () from /lib/libc.so.1 # 18:54:59 #3 0xfee42f20 in cond_wait_common () from /lib/libc.so.1 # 18:54:59 #4 0xfee430b4 in _cond_timedwait () from /lib/libc.so.1 # 18:54:59 #5 0xfee431a8 in cond_timedwait () from /lib/libc.so.1 # 18:54:59 #6 0xfee431e8 in pthread_cond_timedwait () from /lib/libc.so.1 # 18:54:59 #7 0x0113e6c8 in safe_cond_timedwait (cond=0xfd62d748, mp=0x1ff8c80, # 18:54:59 abstime=0xfd62d6b0, file=0x133788b "item_func.cc", line=3539) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/mysys/thr_mutex.c:477 # 18:54:59 #8 0x00365b70 in interruptible_wait (thd=0xb798188, cond=0xfd62d748, # 18:54:59 lock=0x1ff8c80, time=10) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//item_func.cc:3539 # 18:54:59 #9 0x003673cc in val_int (this=0xb7ce290) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//item_func.cc:3858 # 18:54:59 #10 0x0032a720 in send (this=0xb7ce290, protocol=0xb798494, buffer=0xfd62d968) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//item.cc:5557 # 18:54:59 #11 0x0046c1c4 in send_result_set_row (this=0xb798494, row_items=0xb799538) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//protocol.cc:820 # 18:54:59 # 18:54:59 Thread 34 (process 1720400 ): # 18:54:59 #0 0xfee4c404 in _read () from /lib/libc.so.1 # 18:54:59 #1 0xfee3aaf4 in read () from /lib/libc.so.1 # 18:54:59 #2 0x0122e3f8 in vio_read (vio=0xb3a8998, buf=0xb426178 "\a", size=4) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/vio/viosocket.c:44 # 18:54:59 #3 0x00468e6c in my_real_read (net=0xb44a224, complen=0xfd671e64) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//net_serv.cc:837 # 18:54:59 #4 0x00469518 in my_net_read (net=0xb44a224) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//net_serv.cc:1020 # 18:54:59 #5 0x004904f4 in do_command (thd=0xb44a188) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//sql_parse.cc:692 # 18:54:59 #6 0x0048d120 in handle_one_connection (arg=0xb44a188) # 18:54:59 at /export/home/pb2/build/sb_1-893373-1256656394.86/mysql-6.0.12-alpha/sql//sql_connect.cc:1146 # 18:54:59 #7 0xfee48a28 in _lwp_start () from /lib/libc.so.1 # 18:54:59 #8 0xfee48a28 in _lwp_start () from /lib/libc.so.1 # 18:54:59 Backtrace stopped: previous frame identical to this frame (corrupt stack?) # 18:54:59 # 18:54:59 Thread 33 (process 1589328 ): # 18:54:59 #0 0xfee4b9c0 in ___sigtimedwait () from /lib/libc.so.1 # 18:54:59 #1 0xfee3c0bc in __sigtimedwait () from /lib/libc.so.1 # 18:54:59 #2 0xfee33ef8 in __posix_sigwait () from /lib/libc.so.1 # 18:54:59 ../../gdb-6.6/gdb/dwarf2read.c:2736: internal-error: process_die: Assertion `die->child == NULL' failed. # 18:54:59 A problem internal to GDB has been detected, # 18:54:59 further debugging may prove unreliable. # 18:54:59 Quit this debugging session? (y or n) [answered Y; input not from terminal] # 18:54:59 ../../gdb-6.6/gdb/dwarf2read.c:2736: internal-error: process_die: Assertion `die->child == NULL' failed. # 18:54:59 A problem internal to GDB has been detected, # 18:54:59 further debugging may prove unreliable. # 18:54:59 Create a core file of GDB? (y or n) [answered Y; input not from terminal] # 18:55:05 libc.so.1`_lwp_kill+8(6, 1366d37, 84464, fee1f73c, feeb4f58, 1ff8d28) # 18:55:05 mysqld`handle_segfault+0x398(f, 0, fd5ed540, feeb03a8, fd5ed560, 1150de4) # 18:55:05 libc.so.1`__sighndlr+0xc(6, 0, fd5ed540, 47c2f8, 0, 0) # 18:55:05 libc.so.1`call_user_handler+0x3b8(6, 0, 12, 0, febeca00, fd5ed540) # 18:55:05 libc.so.1`sigacthandler+0x4c(6, 0, fd5ed540, feeb4e3d, febeca00, 0) # 18:55:05 libc.so.1`_lwp_kill+8(6, 0, feeb5800, 4, febeca00, 6) # 18:55:05 mysqld`void Error::debugBreak()+4(13cf9b4, fd5ed924, fd5ed924, 90d68, fee1fe1c # 18:55:05 , 4d) # 18:55:05 mysqld`void Error::error(const char*,...)+0x94(13d3c1c, a05df90, 1, 0, 0, febecafc) # 18:55:05 mysqld`Record*Table::fetchForUpdate(Transaction*,Record*,bool)+0x334(24fc8d8, # 18:55:05 261fe88, a05df90, 1, fd5ee310, febecafc) # 18:55:05 mysqld`int StorageDatabase::nextIndexed(StorageTable*,void*,int,bool)+0x150(2579ed8, # 18:55:05 2c7cdd0, 25b0808, 333, 1, febecafc) # 18:55:05 mysqld`int StorageTable::nextIndexed(int,bool)+0x88(2c7cdd0, 333, 1, fd5edc64, # 18:55:05 b8049b8, febecafc) # 18:55:05 mysqld`int StorageInterface::index_next(unsigned char*)+0xe4(b897600, b8977f8, 13e7, # 18:55:05 fd5edce0, 13309f4, febecafc) # 18:55:05 mysqld`int handler::read_range_next()+0xbc(b897600, 1391f51, 10d7, fd5edd7c, # 18:55:05 1, febecafc) # 18:55:05 mysqld`int handler::multi_range_read_next(char**)+0x84(b897600, fd5ede90, 0, # 18:55:05 fd5eddd4, 4c, fd5ede7c) # 18:55:05 mysqld`int StorageInterface::multi_range_read_next(char**)+0x2c(b897600, # 18:55:05 fd5ede90, 2179, fd5ede80, 4c, fd5ede7c) # 18:55:05 mysqld`int QUICK_RANGE_SELECT::get_next()+0xa0(b45c4c0, b816228, b897d68, # 18:55:05 b583a00, 0, b897d65) # 18:55:05 mysqld`__1cNfind_all_keys6FpnNst_sort_param_pnKSQdDL_SELECT_ppCpnLst_io_cache_77 # 18:55:05 _X_+0x460(fd5ee1c8, b804c90, b816188, fd5ee220, fd5ee310, 0) # 18:55:05 mysqld`unsigned long long filesort(THD*,TABLE*,st_sort_field*,unsigned,SQL_SELECT*,unsigned long long,bool,unsigned long long*)+ # 18:55:05 0x6b8(b4501a0, b583a00, b804dc8, 1, b804c90, 0) # 18:55:05 mysqld`__1cMmysql_update6FpnDTHD_pnKTABLE_LIST_rnEList4nEItem___5pn0C_IpnIst_ord # 18:55:05 er_XnPenum_duplicates_b_i_+0x1288(0, 4, b451550, b451788, b8049b8, 1) # 18:55:05 mysqld`int mysql_execute_command(THD*)+0x425c(b451788, b8049b8, 0, 1, 0, # 18:55:05 4) # 18:55:05 mysqld`void mysql_parse(THD*,const char*,unsigned,const char**)+0x2c0(b4501a0, b804188, 65, fd5f1b18, # 18:55:05 5c8, 0) # 18:55:05 mysqld`bool dispatch_command(enum_server_command,THD*,char*,unsigned)+0xa24(3, # 18:55:05 b4501a0, b42e181, 65, 0, 0) # 18:55:05 mysqld`bool do_command(THD*)+0x2d0(b4501a0, 47ace6f7, 0, 0, febeca00, 1) # 18:55:05 mysqld`handle_one_connection+0x1c0(b4501a0, fd5f2000, 0, 0, 48cf58, 1) # 18:55:05 libc.so.1`_lwp_start(0, 0, 0, 0, 0, 0)
- Use the Sun Studio tool dbx if available
- dbx -c "where -v;threads;quit" $binary $core
- Otherwise, use gdb/mdb and/or use
- pstack $core
- pflags $core
Without checking if dbx is available and without using pstack, pflags:
if (windows()) {...
} elsif ($^O eq 'solaris') {
push @commands, 'dbx -c "where -v;threads;quit" '.$binary.' '.$core;
} else {
push @commands, "gdb --batch --se=$binary --core=$core --command=backtrace.gdb";
push @commands, "gdb --batch --se=$binary --core=$core --command=backtrace-all.gdb";
}
TODO:
- check if binary is compiled with Sun Studio or not
- check if dbx is available (a Sun Studio tool)
- run pstack and/or pflags (part of core Solaris) for extra information (e.g. in the case of dbx not being available)
Writing a core file # 13:19:31 datadir is /export/home/tmp/je159969/mysql-dev/bzr-repos/build-falcon-team-sunstudio/var-randgen/master-data/ # 13:19:31 Core file appears to be /export/home/tmp/je159969/mysql-dev/bzr-repos/build-falcon-team-sunstudio/var-randgen/master-data//core# 13:19:35 Reading mysqld # 13:19:35 core file header read successfully # 13:19:35 Reading ld.so.1 # 13:19:35 Reading libmtmalloc.so.1 # 13:19:35 Reading librt.so.1 # 13:19:35 Reading libpthread.so.1 # 13:19:35 Reading libthread.so.1 # 13:19:35 Reading libdl.so.1 # 13:19:35 Reading libsocket.so.1 # 13:19:35 Reading libnsl.so.1 # 13:19:35 Reading libm.so.2 # 13:19:35 Reading libCstd.so.1 # 13:19:35 Reading libCrun.so.1 # 13:19:35 Reading libc.so.1 # 13:19:35 Reading libaio.so.1 # 13:19:35 Reading libmd.so.1 # 13:19:35 t@41 (l@41) terminated by signal ABRT (Abort) # 13:19:35 0xfffffd7fff0fca4a: __lwp_kill+0x000a: jae __lwp_kill+0x18 [ 0xfffffd7fff0fca58, .+0xe ] # 13:19:35 Current function is my_write_core # 13:19:35 309 pthread_kill(pthread_self(), sig); # 13:19:35 current thread: t@41 # 13:19:35 [1] __lwp_kill(0x29, 0x6, 0xffffffff923ba720, 0x0, 0x0, 0x0), at 0xfffffd7fff0fca4a # 13:19:35 [2] _thr_kill(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0f5323 # 13:19:35 =>[3] my_write_core(sig = 6), line 309 in "stacktrace.c" # 13:19:35 [4] handle_segfault(sig = 6), line 2711 in "mysqld.cc" # 13:19:35 [5] __sighndlr(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0f7226 # 13:19:35 [6] call_user_handler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0ebad2 # 13:19:35 [7] sigacthandler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0ebcee # 13:19:35 ---- called from signal handler with signal 6 (SIGABRT) ------ # 13:19:35 [8] __lwp_kill(0x29, 0x6, 0xffffffff923ba720, 0x0, 0xfefefefefefefeff, 0x10000), at 0xfffffd7fff0fca4a # 13:19:35 [9] _thr_kill(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0f5323 # 13:19:35 [10] raise(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0a1ca9 # 13:19:35 [11] Error::debugBreak(), line 94 in "Error.cpp" # 13:19:35 [12] Error::error(string = 0x17f2de9 "Table::fetchForUpdate found a committed visible lock record.", ...), line 71 in "Error.cpp" # 13:19:35 [13] Table::fetchForUpdate(this = 0x29bdbf8, transaction = 0x2ab41c0, source = 0x4b9fdf8, usingIndex = true), line 3625 in "Table.cpp" # 13:19:35 [14] StorageDatabase::nextIndexed(this = 0x2a10ae8, storageTable = 0x34e6a58, recordBitmap = 0x2a29bb8, recordNumber = 10, lockForUpdate = true), line 447 in "StorageDatabase.cpp" # 13:19:35 [15] StorageTable::nextIndexed(this = 0x34e6a58, recordNumber = 9, lockForUpdate = true), line 190 in "StorageTable.cpp" # 13:19:35 [16] StorageInterface::index_next(this = 0xb3f49c8, buf = 0xb3f4cb0 "0ü^I"), line 1871 in "ha_falcon.cpp" # 13:19:35 [17] handler::read_range_next(this = 0xb3f49c8), line 5104 in "handler.cc" # 13:19:35 [18] handler::multi_range_read_next(this = 0xb3f49c8, range_info = 0xfffffd7ffbe9a950), line 4325 in "handler.cc" # 13:19:35 [19] StorageInterface::multi_range_read_next(this = 0xb3f49c8, rangeInfo = 0xfffffd7ffbe9a950), line 2000 in "ha_falcon.cpp" # 13:19:35 [20] QUICK_RANGE_SELECT::get_next(this = 0x2a3b170), line 8579 in "opt_range.cc" # 13:19:35 [21] find_all_keys(param = 0xfffffd7ffbe9ac38, select = 0xb4c5680, sort_keys = 0xb4d2970, buffpek_pointers = 0xfffffd7ffbe9acc0, tempfile = 0xfffffd7ffbe9ae30, indexfile = (nil)), line 569 in "filesort.cc" # 13:19:35 [22] filesort(thd = 0xb49c978, table = 0xafa09c0, sortorder = 0xb4c5840, s_length = 1U, select = 0xb4c5680, max_rows = 18446744073709551615U, sort_positions = true, examined_rows = 0xfffffd7ffbe9b0b8), line 245 in "filesort.cc" # 13:19:35 [23] mysql_delete(thd = 0xb49c978, table_list = 0xb4c4af0, conds = 0xb4c5288, order = 0xb49e870, limit = 3U, options = 0, reset_auto_increment = false), line 237 in "sql_delete.cc" # 13:19:35 [24] mysql_execute_command(thd = 0xb49c978), line 3385 in "sql_parse.cc" # 13:19:35 [25] mysql_parse(thd = 0xb49c978, inBuf = 0xb4c4998 "DELETE FROM E WHERE `int_key` < '2001-08-15 14:59:49' ORDER BY `datetime_key` LIMIT 3", length = 85U, found_semicolon = 0xfffffd7ffbe9ebe8), line 5979 in "sql_parse.cc" # 13:19:35 [26] dispatch_command(command = COM_QUERY, thd = 0xb49c978, packet = 0xb4ca969 "DELETE FROM E WHERE `int_key` < '2001-08-15 14:59:49' ORDER BY `datetime_key` LIMIT 3", packet_length = 85U), line 1064 in "sql_parse.cc" # 13:19:35 [27] do_command(thd = 0xb49c978), line 746 in "sql_parse.cc" # 13:19:35 [28] handle_one_connection(arg = 0xb49c978), line 1146 in "sql_connect.cc" # 13:19:35 [29] _thr_setup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0f6eeb # 13:19:35 [30] _lwp_start(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff0f7120 # 13:19:35 t@1 a l@1 ?() LWP suspended in __pollsys() # 13:19:35 t@2 a l@2 thread() sleep on 0x2a11af8 in __lwp_park() # 13:19:35 t@3 a l@3 thread() sleep on 0x2a11c10 in __lwp_park() # 13:19:35 t@4 a l@4 thread() sleep on 0x2a1b7b0 in __lwp_park() # 13:19:35 t@5 a l@5 thread() sleep on 0x2a1b8c8 in __lwp_park() # 13:19:35 t@6 a l@6 thread() sleep on 0x2a1b9e0 in __lwp_park() # 13:19:35 t@7 a l@7 thread() sleep on 0x2a1baf8 in __lwp_park() # 13:19:35 t@8 a l@8 thread() sleep on 0x2a1bc10 in __lwp_park() # 13:19:35 t@9 a l@9 thread() LWP suspended in releaseBlock() # 13:19:35 t@10 a l@10 thread() sleep on 0x2a1c480 in __lwp_park() # 13:19:35 t@11 a l@11 thread() sleep on 0x2a1c598 in __lwp_park() # 13:19:35 t@12 a l@12 thread() sleep on 0x2a1c6b0 in __lwp_park() # 13:19:35 t@13 a l@13 thread() sleep on 0x2a1c7c8 in __lwp_park() # 13:19:35 t@14 a l@14 thread() sleep on 0x2a4ac90 in __lwp_park() # 13:19:35 t@15 a l@15 thread() sleep on 0x2a53900 in __lwp_park() # 13:19:35 t@16 a l@16 io_handler_thread() sleep on 0x584f0c0 in __lwp_park() # 13:19:35 t@17 a l@17 io_handler_thread() sleep on 0x584f1c8 in __lwp_park() # 13:19:35 t@18 a l@18 io_handler_thread() sleep on 0x584f2d0 in __lwp_park() # 13:19:35 t@19 a l@19 io_handler_thread() sleep on 0x584f3d8 in __lwp_park() # 13:19:35 t@20 a l@20 srv_lock_timeout_and_monitor_thread() LWP suspended in __pollsys() # 13:19:35 t@21 a l@21 srv_error_monitor_thread() LWP suspended in __pollsys() # 13:19:35 t@22 a l@22 srv_master_thread() sleep on 0x273dd00 in __lwp_park() # 13:19:35 t@23 a l@23 ma_checkpoint_background() sleep on 0x2697a48 in __lwp_park() # 13:19:35 t@24 b l@24 signal_hand() LWP suspended in ___sigtimedwait() # 13:19:35 t@26 b l@26 handle_one_connection() LWP suspended in _read() # 13:19:35 t@33 b l@33 handle_one_connection() sleep on 0xfffffd7ffc0ce730 in __lwp_park() # 13:19:35 t@34 b l@34 handle_one_connection() sleep on 0x2a93ee8 in __lwp_park() # 13:19:35 t@35 b l@35 handle_one_connection() LWP suspended in _write() # 13:19:35 t@36 b l@36 handle_one_connection() LWP suspended in __lwp_park() # 13:19:35 t@37 b l@37 handle_one_connection() sleep on 0x2a9eb48 in __lwp_park() # 13:19:35 t@38 b l@38 handle_one_connection() sleep on 0x2a9f298 in __lwp_park() # 13:19:35 t@39 b l@39 handle_one_connection() sleep on 0x2aa2978 in __lwp_park() # 13:19:35 t@40 b l@40 handle_one_connection() LWP suspended in lwp_yield() # 13:19:35 o> t@41 b l@41 handle_one_connection() signal SIGABRT in __lwp_kill() # 13:19:35 t@43 b l@43 handle_one_connection() sleep on 0x2ab2670 in __lwp_park()
Example pstack output:
----------------- lwp# 1 / thread# 1 -------------------- fffffd7fff0fc58a __pollsys () + a fffffd7fff0a33a4 pselect () + 154 fffffd7fff0a3672 select () + 72 0000000000b2a2e7 __1cbAhandle_connections_sockets6F_v_ () + 2d7 0000000000b29480 main () + 620 00000000009c585c ???????? () ----------------- lwp# 2 / thread# 2 -------------------- (...) Lots of other threads... (...) ----------------- lwp# 42 / thread# 42 -------------------- fffffd7fff0fca4a _lwp_kill () + a 00000000016aa2c6 my_write_core () + 46 0000000000b248d1 handle_segfault () + 2e1 fffffd7fff0f7226 __sighndlr () + 6 fffffd7fff0ebad2 call_user_handler () + 252 fffffd7fff0ebcee sigacthandler (6, 0, fffffd7ffbe9a110) + de --- called from signal handler with signal 6 (SIGABRT) --- fffffd7fff0fca4a _lwp_kill () + a fffffd7fff0a1ca9 raise () + 19 0000000000ee60c3 __1cFErrorKdebugBreak6F_v_ () + 13 0000000000ee5feb __1cFErrorFerror6FpkcE_v_ () + 12b 0000000000f26e3a __1cFTableOfetchForUpdate6MpnLTransaction_pnGRecord_b_4_ () + 32a 0000000000f00117 __1cPStorageDatabaseHnextRow6MpnMStorageTable_ib_i_ () + 147 0000000000f0a8ff __1cMStorageTableEnext6Mib_i_ () + 3f 0000000000ee7f74 __1cQStorageInterfaceIrnd_next6MpC_i_ () + c4 0000000000cfb5f9 __1cNfind_all_keys6FpnNst_sort_param_pnKSQdDL_SELECT_ppCpnLst_io_cache_77_L_ () + 709 0000000000cf9d3d __1cIfilesort6FpnDTHD_pnFTABLE_pnNst_sort_field_IpnKSQdDL_SELECT_LbpL_L_ () + 85d 0000000000c617e0 __1cMmysql_delete6FpnDTHD_pnKTABLE_LIST_pnEItem_pnLst_sql_list_LXb_b_ () + ef0 0000000000b3f89a __1cVmysql_execute_command6FpnDTHD__i_ () + 4dfa 0000000000b49e2c __1cLmysql_parse6FpnDTHD_pkcIp3_v_ () + 38c 0000000000b383ef __1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ () + b5f 0000000000b375a8 __1cKdo_command6FpnDTHD__b_ () + 2f8 0000000000b358b7 handle_one_connection () + 267 fffffd7fff0f6eeb _thr_setup () + 5b fffffd7fff0f7120 _lwp_start () ----------------- lwp# 43 / thread# 43 -------------------- fffffd7fff0fc69a _read () + a 0000000001663a06 vio_read () + e6 0000000000b122e9 __1cMmy_real_read6FpnGst_net_pL_L_ () + 159 0000000000b12a6d my_net_read () + 3d 0000000000b3737d __1cKdo_command6FpnDTHD__b_ () + cd 0000000000b358b7 handle_one_connection () + 267 fffffd7fff0f6eeb _thr_setup () + 5b fffffd7fff0f7120 _lwp_start ()
Example pflags output:
data model = _LP64 flags = MSACCT|MSFORK
/1: flags = STOPPED pollsys(0xfffffd7fffdf7400,0x2,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/2: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/3: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/4: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffe940db0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/5: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/6: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/7: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/8: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffe142cc0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/9: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/10: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/11: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/12: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/13: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/14: flags = DETACH|STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/15: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffd34bd40,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/16: flags = STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/17: flags = STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/18: flags = STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/19: flags = STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/20: flags = STOPPED pollsys(0xfffffd7ffc950e50,0x0,0xfffffd7ffc950ee0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/21: flags = STOPPED pollsys(0xfffffd7ffc751e50,0x0,0xfffffd7ffc751ee0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/22: flags = STOPPED lwp_park(0x0,0x0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/23: flags = STOPPED lwp_park(0x0,0xfffffd7ffc353d80,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/24: flags = DETACH|STOPPED sigtimedwait(0xfffffd7ffc154d30,0xfffffd7ffc154d40,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/26: flags = DETACH|STOPPED read(0x29,0xaebe968,0x4)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/33: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffc0ce4d0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/34: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffc08d240,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/35: flags = DETACH|STOPPED
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/36: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffc00b3f0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/37: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffbfca1f0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/38: flags = DETACH|STOPPED
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/39: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffbf483f0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/41: flags = DETACH|STOPPED lwp_park(0x0,0xfffffd7ffbf073f0,0x0)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000
/42: flags = DETACH
sigmask = 0xfffffeff,0x0000ffff cursig = SIGABRT
/43: flags = DETACH|STOPPED read(0x33,0xb4f6970,0x4)
why = PR_SUSPENDED
sigmask = 0x00807007,0x00000000